vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide/mrs_01_0441.html
Yang, Tong 6182f91ba8 MRS component operation guide_normal 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-09 14:55:21 +00:00

275 lines
33 KiB
HTML
Raw Blame History

<a name="mrs_01_0441"></a><a name="mrs_01_0441"></a>
<h1 class="topictitle1">Synchronizing Binlog-based MySQL Data to the MRS Cluster</h1>
<div id="body1589421630465"><p id="mrs_01_0441__p88981612131616">This section describes how to use the Maxwell data synchronization tool to migrate offline binlog-based data to an MRS Kafka cluster.</p>
<p id="mrs_01_0441__p4320132817422">Maxwell is an open source application that reads MySQL binlogs, converts operations, such as addition, deletion, and modification, into a JSON format, and sends them to an output end, such as a console, a file, and Kafka. For details about Maxwell, visit <a href="https://maxwells-daemon.io" target="_blank" rel="noopener noreferrer">https://maxwells-daemon.io</a>. Maxwell can be deployed on a MySQL server or on other servers that can communicate with MySQL.</p>
<p id="mrs_01_0441__p175435133716">Maxwell runs on a Linux server, including EulerOS, Ubuntu, Debian, CentOS, and OpenSUSE. Java 1.8+ must be supported.</p>
<p id="mrs_01_0441__p477616042511">The following provides details about data synchronization.</p>
<ol id="mrs_01_0441__ol1916419139261"><li id="mrs_01_0441__li171641413192619"><a href="#mrs_01_0441__section17682011252">Configuring MySQL</a></li><li id="mrs_01_0441__li14452317192615"><a href="#mrs_01_0441__section935158131710">Installing Maxwell</a></li><li id="mrs_01_0441__li19727105083513"><a href="#mrs_01_0441__section897274173612">Configuring Maxwell</a></li><li id="mrs_01_0441__li2097615493512"><a href="#mrs_01_0441__section182543817211">Starting Maxwell</a></li><li id="mrs_01_0441__li2013711313613"><a href="#mrs_01_0441__section191261916641">Verifying Maxwell</a></li><li id="mrs_01_0441__li195515203610"><a href="#mrs_01_0441__section07395261236">Stopping Maxwell</a></li><li id="mrs_01_0441__li594020743612"><a href="#mrs_01_0441__section3609745132419">Format of the Maxwell Generated Data and Description of Common Fields</a></li></ol>
<div class="section" id="mrs_01_0441__section17682011252"><a name="mrs_01_0441__section17682011252"></a><a name="section17682011252"></a><h4 class="sectiontitle">Configuring MySQL</h4><ol id="mrs_01_0441__ol734963410257"><li id="mrs_01_0441__li173491934142511"><span>Start the binlog, open the <strong id="mrs_01_0441__b552314565292">my.cnf</strong> file in MySQL, and check whether <strong id="mrs_01_0441__b13998118133012">server_id</strong>, <strong id="mrs_01_0441__b157461128103018">log-bin</strong>, and <strong id="mrs_01_0441__b1396819328301">binlog_format</strong> are configured in the <strong id="mrs_01_0441__b1850611394303">[mysqld]</strong> block. If they are not configured, run the following command to add configuration items and restart MySQL. If they are configured, skip this step.</span><p><pre class="screen" id="mrs_01_0441__screen20330144611296">$ vi my.cnf
[mysqld]
server_id=1
log-bin=master
binlog_format=row</pre>
</p></li><li id="mrs_01_0441__li101111736164511"><a name="mrs_01_0441__li101111736164511"></a><a name="li101111736164511"></a><span>Maxwell needs to connect to MySQL, create a database named <strong id="mrs_01_0441__b1641081214325">maxwell</strong> for storing metadata, and access the database to be synchronized. Therefore, you are advised to create a MySQL user for Maxwell to use. Log in to MySQL as user <strong id="mrs_01_0441__b131911529153515">root</strong> and run the following commands to create a user named <strong id="mrs_01_0441__b11233142034018">maxwell</strong> (<em id="mrs_01_0441__i219312916359">XXXXXX</em> indicates the password and needs to be replaced with actual one).</span><p><ul id="mrs_01_0441__ul722113920473"><li id="mrs_01_0441__li1222123913470">If Maxwell is deployed on a non-MySQL server, the created user <strong id="mrs_01_0441__b1298014875110">maxwell</strong> must have a permission to remotely log in to the database. In this case, run the following command to create the user:<p id="mrs_01_0441__p16203192720174"><b><span class="cmdname" id="mrs_01_0441__cmdname1432623820172">mysql&gt; GRANT ALL on maxwell.* to 'maxwell'@'%' identified by 'XXXXXX';</span></b></p>
<p id="mrs_01_0441__p12203122717177"><b><span class="cmdname" id="mrs_01_0441__cmdname951018412170">mysql&gt; GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'%';</span></b></p>
</li><li id="mrs_01_0441__li14399226144811">If Maxwell is deployed on the MySQL server, the created user <strong id="mrs_01_0441__b112131755182520">maxwell</strong> can be configured to log in to the database only on the local host. In this case, run the following command:<p id="mrs_01_0441__p14421245181717"><b><span class="cmdname" id="mrs_01_0441__cmdname96734490175">mysql&gt; GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'localhost' identified by 'XXXXXX';</span></b></p>
<p id="mrs_01_0441__p164420459178"><b><span class="cmdname" id="mrs_01_0441__cmdname1615125331716">mysql&gt; GRANT ALL on maxwell.* to 'maxwell'@'localhost';</span></b></p>
</li></ul>
</p></li></ol>
</div>
<div class="section" id="mrs_01_0441__section935158131710"><a name="mrs_01_0441__section935158131710"></a><a name="section935158131710"></a><h4 class="sectiontitle">Installing Maxwell</h4><ol id="mrs_01_0441__ol870915192174"><li id="mrs_01_0441__li1770921910174"><span>Download the installation package at <a href="https://github.com/zendesk/maxwell/releases" target="_blank" rel="noopener noreferrer">https://github.com/zendesk/maxwell/releases</a> and select the <strong id="mrs_01_0441__b7399142712323">maxwell-XXX.tar.gz</strong> binary file for download. In the file name, <strong id="mrs_01_0441__b14736155433216">XXX</strong> indicates a version number.</span></li><li id="mrs_01_0441__li358210183114"><span>Upload the <strong id="mrs_01_0441__b1850252883316">tar.gz</strong> package to any directory (the <strong id="mrs_01_0441__b1275175963219">/opt</strong> directory of the Master node used as an example here).</span></li><li id="mrs_01_0441__li1884361593112"><span>Log in to the server where Maxwell is deployed and run the following command to go to the directory where the <strong id="mrs_01_0441__b1296478153712">tar.gz</strong> package is stored.</span><p><p id="mrs_01_0441__p9740201188"><b><span class="cmdname" id="mrs_01_0441__cmdname687664121817">cd /opt</span></b></p>
</p></li><li id="mrs_01_0441__li10264030103119"><span>Run the following commands to decompress the <strong id="mrs_01_0441__b1513633418385">maxwell-XXX.tar.gz</strong> package and go to the <strong id="mrs_01_0441__b5179445123818">maxwell-XXX</strong> directory:</span><p><p id="mrs_01_0441__p12177610111812"><b><span class="cmdname" id="mrs_01_0441__cmdname1616541461815">tar -zxvf maxwell-XXX.tar.gz</span></b></p>
<p id="mrs_01_0441__p161772010151816"><b><span class="cmdname" id="mrs_01_0441__cmdname644452081813">cd maxwell-XXX</span></b></p>
</p></li></ol>
</div>
<div class="section" id="mrs_01_0441__section897274173612"><a name="mrs_01_0441__section897274173612"></a><a name="section897274173612"></a><h4 class="sectiontitle">Configuring Maxwell</h4><p id="mrs_01_0441__p13503852173814">If the <strong id="mrs_01_0441__b5761195020523">conf</strong> directory exists in the <strong id="mrs_01_0441__b0762950165215">maxwell-XXX</strong> folder, configure the <strong id="mrs_01_0441__b9762750175216">config.properties</strong> file. For details about the configuration items, see <a href="#mrs_01_0441__table49411722133920">Table 1</a>. If the <strong id="mrs_01_0441__b2019931261812">conf</strong> directory does not exist, change <strong id="mrs_01_0441__b169031248182">config.properties.example</strong> in the <strong id="mrs_01_0441__b1393911283183">maxwell-XXX</strong> folder to <strong id="mrs_01_0441__b489718353187">config.properties</strong>.</p>
<div class="tablenoborder"><a name="mrs_01_0441__table49411722133920"></a><a name="table49411722133920"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0441__table49411722133920" width="100%" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Maxwell configuration item description</caption><thead align="left"><tr id="mrs_01_0441__row16941172216396"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.8.3.2.5.1.1"><p id="mrs_01_0441__p594142213399">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="12.690000000000001%" id="mcps1.3.8.3.2.5.1.2"><p id="mrs_01_0441__p794111220399">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="47.31%" id="mcps1.3.8.3.2.5.1.3"><p id="mrs_01_0441__p1294111221394">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.8.3.2.5.1.4"><p id="mrs_01_0441__p99413224399">Default Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_0441__row494172283910"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p1450445214116">user</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p85041352174117">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p1050445212418">Name of the user for connecting to MySQL, that is, the user created in <a href="#mrs_01_0441__li101111736164511">2</a>.</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p1350445220417">-</p>
</td>
</tr>
<tr id="mrs_01_0441__row1294110221394"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p350435204117">password</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p145041652144110">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p95042052164118">Password for connecting to MySQL</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p4504135217414">-</p>
</td>
</tr>
<tr id="mrs_01_0441__row169411822143913"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p165043525419">host</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p850419522412">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p3504105234114">MySQL address</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p14504125213411">localhost</p>
</td>
</tr>
<tr id="mrs_01_0441__row19941192211392"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p1850555220417">port</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p65052524413">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p19505105244110">MySQL port</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p250545244119">3306</p>
</td>
</tr>
<tr id="mrs_01_0441__row179411322133910"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p05050526411">log_level</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p1505115220418">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p1750513528414">Log print level. The options are as follows:</p>
<ul id="mrs_01_0441__ul3178205310440"><li id="mrs_01_0441__li14200185716449">debug</li><li id="mrs_01_0441__li854520313454">info</li><li id="mrs_01_0441__li1340548164517">warn</li><li id="mrs_01_0441__li10178165320447">error</li></ul>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p125051452164117">info</p>
</td>
</tr>
<tr id="mrs_01_0441__row209411222153919"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p195050523413">output_ddl</p>
<p id="mrs_01_0441__p1505852104115"></p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p250513523417">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p850535214111">Whether to send a DDL (modified based on definitions of the database and data table) event</p>
<ul id="mrs_01_0441__ul25311317194514"><li id="mrs_01_0441__li13101142316457"><strong id="mrs_01_0441__b1556262915508">true</strong>: Send DDL events.</li><li id="mrs_01_0441__li105328171453"><strong id="mrs_01_0441__b2027593819507">false</strong>: Do not send DDL events.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p7506115244113">false</p>
</td>
</tr>
<tr id="mrs_01_0441__row19941522143912"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p5506105212410">producer</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p1050615528414">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p45065521419">Producer type. Set this parameter to <strong id="mrs_01_0441__b3306395513">kafka</strong>.</p>
<ul id="mrs_01_0441__ul18903443184517"><li id="mrs_01_0441__li1728184812452"><strong id="mrs_01_0441__b6871037165620">stdout</strong>: Log the generated events.</li><li id="mrs_01_0441__li2090344344519"><strong id="mrs_01_0441__b1412810611545">kafka</strong>: Send the generated events to Kafka.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p15064523411">stdout</p>
</td>
</tr>
<tr id="mrs_01_0441__row1913185319468"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p213255324619">producer_partition_by</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p51323530466">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p13745572472">Partition policy used to ensure that data of the same type is written to the same partition of Kafka.</p>
<ul id="mrs_01_0441__ul108581437155118"><li id="mrs_01_0441__li116519442513"><strong id="mrs_01_0441__b1694452155714">database</strong>: Events of the same database are written to the same partition of Kafka.</li><li id="mrs_01_0441__li1858837125114"><strong id="mrs_01_0441__b6224134018579">table</strong>: Events of the same table are written to the same partition of Kafka.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p11374165714475">databa</p>
</td>
</tr>
<tr id="mrs_01_0441__row0596105394618"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p57665511487">ignore_producer_error</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p8766175174811">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p167666512489">Specifies whether to ignore the error that the producer fails to send data.</p>
<ul id="mrs_01_0441__ul720014302521"><li id="mrs_01_0441__li467393735219"><strong id="mrs_01_0441__b1932510361583">true</strong>: The error information is logged and the error data is skipped. The program continues to run.</li><li id="mrs_01_0441__li4201133095214"><strong id="mrs_01_0441__b11385101295">false</strong>: The error information is logged and the program is terminated.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p076717517482">true</p>
</td>
</tr>
<tr id="mrs_01_0441__row845205464613"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p1876795134818">metrics_slf4j_interval</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p14767165134817">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p1576775124819">Interval for outputting statistics on data successfully uploaded or failed to be uploaded to Kafka in logs. The unit is second.</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p167671255489">60</p>
</td>
</tr>
<tr id="mrs_01_0441__row17348195410461"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p91321216174816">kafka.bootstrap.servers</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p4132141612486">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p11132131684817">Address of the Kafka proxy node. The value is in the format of <strong id="mrs_01_0441__b1645619187206">HOST:PORT[,HOST:PORT]</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p81321816134817">-</p>
</td>
</tr>
<tr id="mrs_01_0441__row96461454154613"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p81322016124817">kafka_topic</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p141322016204813">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p12132516204814">Name of the topic that is written to Kafka</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p013221694818">maxwell</p>
</td>
</tr>
<tr id="mrs_01_0441__row208796549468"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p1132111611488">dead_letter_topic</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p413391613487">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p12133716154812">Kafka topic used to record the primary key of the error log record when an error occurs when the record is sent</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p11333167481">-</p>
</td>
</tr>
<tr id="mrs_01_0441__row819165584611"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p113314161485">kafka_version</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p2013321684811">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p61331616174814">Kafka producer version used by Maxwell, which cannot be configured in the <strong id="mrs_01_0441__b1496723583312">config.properties</strong> file. You need to use the <strong id="mrs_01_0441__b15189833173618">-- kafka_version xxx</strong> parameter to import the version number when starting the command.</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p31339166482">-</p>
</td>
</tr>
<tr id="mrs_01_0441__row2392155515467"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p1213391644820">kafka_partition_hash</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p121331216204811">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p11133101614482">Kafka topic partitioning algorithm. The value can be <strong id="mrs_01_0441__b58201132194014">default</strong> or <strong id="mrs_01_0441__b8767154154011">murmur3</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p41334168482">default</p>
</td>
</tr>
<tr id="mrs_01_0441__row1374917557462"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p10133131610482">kafka_key_format</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p2133516194814">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p2133161619482">Key generation method of the Kafka record. The value can be <strong id="mrs_01_0441__b14933795432">array</strong> or <strong id="mrs_01_0441__b115485139435">Hash</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p11133716154811">Hash</p>
</td>
</tr>
<tr id="mrs_01_0441__row1725595615462"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p1133316124817">ddl_kafka_topic</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p14133141664817">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p6133171613483">Topic that is written to the DDL operation when <strong id="mrs_01_0441__b5435205816436">output_ddl</strong> is set to <strong id="mrs_01_0441__b177564115444">true</strong></p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p013331624810">{kafka_topic}</p>
</td>
</tr>
<tr id="mrs_01_0441__row12559125616463"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.1 "><p id="mrs_01_0441__p16493202424812">filter</p>
</td>
<td class="cellrowborder" valign="top" width="12.690000000000001%" headers="mcps1.3.8.3.2.5.1.2 "><p id="mrs_01_0441__p8493924144814">No</p>
</td>
<td class="cellrowborder" valign="top" width="47.31%" headers="mcps1.3.8.3.2.5.1.3 "><p id="mrs_01_0441__p349332464814">Used to filter databases or tables.</p>
<ul id="mrs_01_0441__ul1439410583596"><li id="mrs_01_0441__li1611218511706">If only the <strong id="mrs_01_0441__b725516618451">mydatabase</strong> database needs to be collected, set this parameter to the following:<p id="mrs_01_0441__p15976175911016">exclude: *.*,include: mydatabase.*</p>
</li><li id="mrs_01_0441__li1810195419">If only the <strong id="mrs_01_0441__b2133152554510">mydatabase.mytable</strong> table needs to be collected, set this parameter to the following:<p id="mrs_01_0441__p76591078118">exclude: *.*,include: mydatabase.mytable</p>
</li><li id="mrs_01_0441__li142742142017">If only the <strong id="mrs_01_0441__b2096414014493">mytable</strong>, <strong id="mrs_01_0441__b4876106154919">mydate_123</strong>, and <strong id="mrs_01_0441__b626115122494">mydate_456</strong> tables in the <strong id="mrs_01_0441__b4271731134919">mydatabase</strong> database need to be collected, set this parameter to the following:<p id="mrs_01_0441__p112093169112">exclude: *.*,include: mydatabase.mytable, include: mydatabase./mydate_\\d*/</p>
</li></ul>
<p id="mrs_01_0441__p1949472434815"></p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.8.3.2.5.1.4 "><p id="mrs_01_0441__p5559356104618">-</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="mrs_01_0441__section182543817211"><a name="mrs_01_0441__section182543817211"></a><a name="section182543817211"></a><h4 class="sectiontitle">Starting Maxwell</h4><ol id="mrs_01_0441__ol17962162116216"><li id="mrs_01_0441__li11962152112216"><span>Log in to the server where Maxwell is deployed.</span></li><li id="mrs_01_0441__li9244241237"><span>Run the following command to go to the Maxwell installation directory:</span><p><p id="mrs_01_0441__p11995171014313"><b><span class="cmdname" id="mrs_01_0441__cmdname987817120171">cd /opt/maxwell-1.21.0/</span></b></p>
<div class="note" id="mrs_01_0441__note84435247313"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0441__p77311130835">For the first time to use Maxwell, you are advised to change <strong id="mrs_01_0441__b18715134825314">log_level</strong> in <strong id="mrs_01_0441__b16716114885320">conf/config.properties</strong> to <strong id="mrs_01_0441__b1871714484536">debug</strong> (debug level) so that you can check whether data can be obtained from MySQL and sent to Kafka after startup. After the entire process is debugged, change <strong id="mrs_01_0441__b471712489533">log_level</strong> to <strong id="mrs_01_0441__b16718154835318">info</strong>, and then restart Maxwell for the modification to take effect.</p>
<p id="mrs_01_0441__p973218301536"># log level [debug | info | warn | error]</p>
<p id="mrs_01_0441__p1873218301136">log_level=debug</p>
</div></div>
</p></li><li id="mrs_01_0441__li115142462310"><span>Run the following commands to start Maxwell:</span><p><p id="mrs_01_0441__p77212531317"><strong id="mrs_01_0441__b028144753919">source /opt/client/bigdata_env</strong></p>
<p id="mrs_01_0441__p67217533313"><b><span class="cmdname" id="mrs_01_0441__cmdname241531451914">bin/Maxwell</span></b></p>
<p id="mrs_01_0441__p1672175310314"><b><span class="cmdname" id="mrs_01_0441__cmdname186268589226">bin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \</span></b></p>
<p id="mrs_01_0441__p1472110531334"><b><span class="cmdname" id="mrs_01_0441__cmdname39201128237">--producer=kafka --kafka.bootstrap.servers=kafkahost:9092 --kafka_topic=Maxwell</span></b></p>
<p id="mrs_01_0441__p572115319315">In the preceding commands, <strong id="mrs_01_0441__b918213252561">user</strong>, <strong id="mrs_01_0441__b22736276562">password</strong>, and <strong id="mrs_01_0441__b129971231175613">host</strong> indicate the username, password, and IP address of MySQL, respectively. You can configure the three parameters by modifying configurations of the configuration items or using the preceding commands. <strong id="mrs_01_0441__b1384524914411">kafkahost</strong> indicates the IP address of the Core node in the streaming cluster.</p>
<p id="mrs_01_0441__p27211753233">If information similar to the following appears, Maxwell has started successfully:</p>
<pre class="screen" id="mrs_01_0441__screen2298717206">Success to start Maxwell [78092].</pre>
</p></li></ol>
</div>
<div class="section" id="mrs_01_0441__section191261916641"><a name="mrs_01_0441__section191261916641"></a><a name="section191261916641"></a><h4 class="sectiontitle">Verifying Maxwell</h4><ol id="mrs_01_0441__ol113488313413"><li id="mrs_01_0441__li334883115418"><span>Log in to the server where Maxwell is deployed.</span></li><li id="mrs_01_0441__li1129111441543"><span>View the logs. If the log file does not contain an ERROR log and the following information is displayed, the connection between Maxwell and MySQL is normal:</span><p><pre class="screen" id="mrs_01_0441__screen8939151065">BinlogConnectorLifecycleListener - Binlog connected.</pre>
</p></li><li id="mrs_01_0441__li19012541645"><span>Log in to the MySQL database and update, create, or delete test data. The following provides operation statement examples for your reference. </span><p><pre class="screen" id="mrs_01_0441__screen58251981452">--Creating a database
create database test;
--Creating a table
create table test.e (
id int(10) not null primary key auto_increment,
m double,
c timestamp(6),
comment varchar(255) charset 'latin1'
);
-- Adding a record
insert into test.e set m = 4.2341, c = now(3), comment = 'I am a creature of light.';
--Updating a record
update test.e set m = 5.444, c = now(3) where id = 1;
--Deleting a record
delete from test.e where id = 1;
--Modifying a table
alter table test.e add column torvalds bigint unsigned after m;
--Deleting a table
drop table test.e;
-- Deleting a database
drop database test;</pre>
</p></li><li id="mrs_01_0441__li1893172011203"><span>Check the Maxwell logs. If no WARN/ERROR is displayed, Maxwell is installed and configured properly.</span><p><div class="p" id="mrs_01_0441__p1462293414273">To check whether the data is successfully uploaded, set <strong id="mrs_01_0441__b1488161253411">log_level</strong> in the <strong id="mrs_01_0441__b1289201219343">config.properties</strong> file to <strong id="mrs_01_0441__b590171219342">debug</strong>. When the data is successfully uploaded, the following JSON data is printed immediately. For details about the fields, see <a href="#mrs_01_0441__section3609745132419">Format of the Maxwell Generated Data and Description of Common Fields</a>.<pre class="screen" id="mrs_01_0441__screen25291389112">{"database":"test","table":"e","type":"insert","ts":1541150929,"xid":60556,"commit":true,"data":{"id":1,"m":4.2341,"c":"2018-11-02 09:28:49.297000","comment":"I am a creature of light."}}
......</pre>
<div class="note" id="mrs_01_0441__note155971031152012"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0441__p15597173112011">After the entire process is debugged, you can change the value of <strong id="mrs_01_0441__b1355975973118">log_level</strong> in the <strong id="mrs_01_0441__b116168205323">config.properties</strong> file to <strong id="mrs_01_0441__b165605592312">info</strong> to reduce the number of logs to be printed and restart Maxwell for the modification to take effect.</p>
<pre class="screen" id="mrs_01_0441__screen1059717311202"># log level [debug | info | warn | error]
log_level=info</pre>
</div></div>
</div>
</p></li></ol>
</div>
<div class="section" id="mrs_01_0441__section07395261236"><a name="mrs_01_0441__section07395261236"></a><a name="section07395261236"></a><h4 class="sectiontitle">Stopping Maxwell</h4><ol id="mrs_01_0441__ol3769184072317"><li id="mrs_01_0441__li12767114132320"><span>Log in to the server where Maxwell is deployed.</span></li><li id="mrs_01_0441__li117671241192319"><span>Run the command to obtain the Maxwell process ID (PID). The second field in the command output is PID.</span><p><p id="mrs_01_0441__p99222722412"><b><span class="cmdname" id="mrs_01_0441__cmdname633651414241">ps -ef | grep Maxwell | grep -v grep</span></b></p>
</p></li><li id="mrs_01_0441__li2767174122311"><span>Run the following command to forcibly stop the Maxwell process:</span><p><p id="mrs_01_0441__p1463523662419"><b><span class="cmdname" id="mrs_01_0441__cmdname7260174015245">kill -9 PID</span></b></p>
</p></li></ol>
</div>
<div class="section" id="mrs_01_0441__section3609745132419"><a name="mrs_01_0441__section3609745132419"></a><a name="section3609745132419"></a><h4 class="sectiontitle">Format of the Maxwell Generated Data and Description of Common Fields</h4><p id="mrs_01_0441__p773431082514">The data generated by Maxwell is in JSON format. The common fields are described as follows:</p>
<ul id="mrs_01_0441__ul7759743142511"><li id="mrs_01_0441__li1153915481254"><strong id="mrs_01_0441__b1328243343710">type</strong>: operation type. The options are <strong id="mrs_01_0441__b4587953133712">database-create</strong>, <strong id="mrs_01_0441__b111081457113711">database-drop</strong>, <strong id="mrs_01_0441__b1273313218381">table-create</strong>, <strong id="mrs_01_0441__b154568718388">table-drop</strong>, <strong id="mrs_01_0441__b996851033812">table-alter</strong>, <strong id="mrs_01_0441__b684181493814">insert</strong>, <strong id="mrs_01_0441__b61971175388">update</strong>, and <strong id="mrs_01_0441__b1061021953811">delete</strong>.</li><li id="mrs_01_0441__li182201352192510"><strong id="mrs_01_0441__b134961727123813">database</strong>: name of the database to be operated</li><li id="mrs_01_0441__li452111557253"><strong id="mrs_01_0441__b1339126103918">ts</strong>: operation time, which is a 13-digit timestamp</li><li id="mrs_01_0441__li193378582254"><strong id="mrs_01_0441__b6297940183910">table</strong>: name of the table to be operated</li><li id="mrs_01_0441__li31914112269"><strong id="mrs_01_0441__b58435465397">data</strong>: content after data is added, deleted, or modified</li><li id="mrs_01_0441__li12405146265"><strong id="mrs_01_0441__b14332195753919">old</strong>: content before data is modified or schema definition before a table is modified</li><li id="mrs_01_0441__li1924116192619"><strong id="mrs_01_0441__b1991810164402">sql</strong>: SQL statement for DDL operations</li><li id="mrs_01_0441__li142414159268"><strong id="mrs_01_0441__b11492128134012">def</strong>: schema definition for table creation and modification</li><li id="mrs_01_0441__li165510183261"><strong id="mrs_01_0441__b1363831524115">xid</strong>: unique ID of an object</li><li id="mrs_01_0441__li14759134310252"><strong id="mrs_01_0441__b3806112784119">commit</strong>: check whether such operations as data addition, deletion, and modification have been submitted.</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0375.html">Using Kafka</a></div>
</div>
</div>
View Git Blame Copy Permalink