vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_24751.html
Yang, Tong 48706b7552 MRS COMP-LTS 320-lts.1 version 
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2024-04-12 12:51:10 +00:00

315 lines
31 KiB
HTML
Raw Blame History

<a name="mrs_01_24751"></a><a name="mrs_01_24751"></a>
<h1 class="topictitle1">Synchronizing Data from MySQL to Hudi</h1>
<div id="body0000001532950756"><div class="section" id="mrs_01_24751__section7722164042318"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24751__p664594413238">This section describes how to import data from MySQL to Hudi by using the CDLService web UI of a cluster with Kerberos authentication enabled.</p>
</div>
<div class="section" id="mrs_01_24751__section1438820494231"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24751__ul2041871362311"><li id="mrs_01_24751__li12418213122313">The CDL and Hudi services have been installed in a cluster and are running properly.</li><li id="mrs_01_24751__li362910581468">The prerequisites for the MySQL database have been met. For details, see <a href="mrs_01_24124.html#mrs_01_24124__li268123915168">Prerequisites for Enabling the MySQL Database</a>.</li><li id="mrs_01_24751__li1441819135232">You have created a human-machine user, for example, <strong id="mrs_01_24751__b1857556144610">cdluser</strong>, added the user to user groups <strong id="mrs_01_24751__b1864155620468">cdladmin</strong> (primary group), <strong id="mrs_01_24751__b086419566460">hadoop</strong>, <strong id="mrs_01_24751__b14864135616468">kafka</strong>, and <strong id="mrs_01_24751__b1886455614615">supergroup</strong>, and associated the user with the <strong id="mrs_01_24751__b1786455616467">System_administrator</strong> role on FusionInsight Manager.</li></ul>
</div>
<div class="section" id="mrs_01_24751__section19362472245"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24751__ol18871133225317"><li id="mrs_01_24751__li1365211683614"><span>Log in to FusionInsight Manager as user <strong id="mrs_01_24751__b101811954421">cdluser</strong> (change the password upon your first login), choose <strong id="mrs_01_24751__b133011943035944">Cluster</strong> &gt; <strong id="mrs_01_24751__b11102831535944">Services</strong> &gt; <strong id="mrs_01_24751__b160695716635944">CDL</strong>, and click the link next to <strong id="mrs_01_24751__b109518290035944">CDLService UI</strong> to go to the CDLService web UI.</span></li><li id="mrs_01_24751__li717018430356"><span>Choose <strong id="mrs_01_24751__b196599826435944">Driver Management</strong> and click <strong id="mrs_01_24751__b145705566135944">Upload Driver</strong> to upload the driver file <strong id="mrs_01_24751__b473212011498">mysql-connector-java-8.0.24.jar</strong> of MySQL. For details, see <a href="mrs_01_24237.html">Uploading a Driver File</a>.</span></li><li id="mrs_01_24751__li3618121145616"><span>Choose <strong id="mrs_01_24751__b83702024835944">Link Management</strong> and click <strong id="mrs_01_24751__b172077121635944">Add Link</strong>. In the displayed dialog box, set parameters for adding the <strong id="mrs_01_24751__b150569027535944">mysql</strong> and <strong id="mrs_01_24751__b147976007235944">hudi</strong> links by referring to the following tables.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24751__table134120364291" frame="border" border="1" rules="all"><caption><b>Table 1 </b>MySQL data link parameters</caption><thead align="left"><tr id="mrs_01_24751__row2041836162918"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.3.2.1.2.3.1.1"><p id="mrs_01_24751__p4411136172915">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.3.2.1.2.3.1.2"><p id="mrs_01_24751__p19411736142919">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24751__row10411236152920"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24751__p154119368298">Link Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24751__p154153632911">mysql</p>
</td>
</tr>
<tr id="mrs_01_24751__row16426365294"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24751__p44220365290">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24751__p104253620293">mysqllink</p>
</td>
</tr>
<tr id="mrs_01_24751__row134233616299"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24751__p1142436122912">DB driver</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24751__p1242163612296">mysql-connector-java-8.0.24.jar</p>
</td>
</tr>
<tr id="mrs_01_24751__row442636152915"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24751__p9427368295">Host</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24751__p17303172811553">10.10.10.10</p>
</td>
</tr>
<tr id="mrs_01_24751__row11423363293"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24751__p10421364293">Port</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24751__p17303122811553">3306</p>
</td>
</tr>
<tr id="mrs_01_24751__row164263622915"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24751__p1442636162910">User</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24751__p4916845152114">dbuser</p>
</td>
</tr>
<tr id="mrs_01_24751__row104317366296"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24751__p194314367297">Password</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24751__p391712458215"><em id="mrs_01_24751__i125741521235944">Password of the dbuser user</em></p>
</td>
</tr>
<tr id="mrs_01_24751__row124313614299"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24751__p74353613299">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24751__p184383615296">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24751__table1437691715220" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Hudi data link parameters</caption><thead align="left"><tr id="mrs_01_24751__row1437618171029"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.3.2.2.2.3.1.1"><p id="mrs_01_24751__p137651713218">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.3.2.2.2.3.1.2"><p id="mrs_01_24751__p1237720171128">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24751__row33815178212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24751__p153814179219">Link Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24751__p938112176210">hudi</p>
</td>
</tr>
<tr id="mrs_01_24751__row193811117822"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24751__p138161715211">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24751__p1738113173219">hudilink</p>
</td>
</tr>
<tr id="mrs_01_24751__row83811417425"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24751__p125527208312">Storage Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24751__p35331203318">hdfs</p>
</td>
</tr>
<tr id="mrs_01_24751__row736874119319"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24751__p1814611463913">Auth KeytabFile</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24751__p1214617466915">/opt/Bigdata/third_lib/CDL/user_libs/cdluser.keytab</p>
</td>
</tr>
<tr id="mrs_01_24751__row936915411130"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24751__p14146546699">Principal</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24751__p1414619465918">cdluser</p>
</td>
</tr>
<tr id="mrs_01_24751__row938313177210"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24751__p1938310171727">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24751__p5383161716218">-</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="mrs_01_24751__li1829512373488"><span>After the parameters are configured, click <strong id="mrs_01_24751__b82477175013">Test</strong> to check whether the data link is normal.</span><p><p id="mrs_01_24751__p13530140134817">After the test is successful, click <strong id="mrs_01_24751__b364014695010">OK</strong>.</p>
</p></li><li id="mrs_01_24751__li976974910717"><span>(Optional) Choose <strong id="mrs_01_24751__en-us_topic_0000001446993949_b45961936164618">ENV Management</strong> and click <strong id="mrs_01_24751__en-us_topic_0000001446993949_b03551454164613">Add Env</strong>. In the displayed dialog box, configure the parameters based on the following table.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24751__en-us_topic_0000001446993949_table54201432181218" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Parameters for adding an ENV</caption><thead align="left"><tr id="mrs_01_24751__en-us_topic_0000001446993949_row10418153291216"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.2.3.1.1"><p id="mrs_01_24751__en-us_topic_0000001446993949_p18418032161213">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.2.3.1.2"><p id="mrs_01_24751__en-us_topic_0000001446993949_p441810320123">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24751__en-us_topic_0000001446993949_row7419193251216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p1141815324123">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p14419183241213">test-env</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row04191332161212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p18419133214122">Driver Memory</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p341923215124">1 GB</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row1468718181894"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p156889181495">Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p106881618096">spark</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row2419632151212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p941918323125">Executor Memory</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p1419832121219">1 GB</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row20419143211122"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p041933213124">Executor Cores</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p1341983217125">1</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row341915321123"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p194191032181217">Number Executors</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p841933251210">1</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row77610527328"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p1525831203317">Queue</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p17675211325">-</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row34191032171216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p94191032181210">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p5419193231211">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24751__en-us_topic_0000001446993949_p145511354104314">Click <strong id="mrs_01_24751__en-us_topic_0000001446993949_b1436468274744">OK</strong>.</p>
</p></li><li id="mrs_01_24751__li8419191320242"><span>Choose <strong id="mrs_01_24751__b1687071320504">Job Management</strong> &gt; <strong id="mrs_01_24751__b891312228525">Data synchronization task</strong> and click <strong id="mrs_01_24751__b164004175219">Add Job</strong>. In the displayed dialog box, set parameters and click <strong id="mrs_01_24751__b169228285835944">Next</strong>.</span><p><div class="p" id="mrs_01_24751__p1442915568513">Parameters are as follows.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24751__table1379713112014" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_24751__row13797011905"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.1.3.1.1"><p id="mrs_01_24751__p17797411309">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.1.3.1.2"><p id="mrs_01_24751__p137971011309">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24751__row1679841118018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.1.3.1.1 "><p id="mrs_01_24751__p391613171005">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.1.3.1.2 "><p id="mrs_01_24751__p149141171404">job_mysqltohudi</p>
</td>
</tr>
<tr id="mrs_01_24751__row779916111602"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.1.3.1.1 "><p id="mrs_01_24751__p83069155152">Desc</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.1.3.1.2 "><p id="mrs_01_24751__p14306191511512">-</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</p></li><li id="mrs_01_24751__li2737114320203"><span>Configure MySQL job parameters.</span><p><ol type="a" id="mrs_01_24751__ol14971759152111"><li id="mrs_01_24751__li637518545212">On the <strong id="mrs_01_24751__b100668771235944">Job Management</strong> page, drag the <strong id="mrs_01_24751__b49234883735944">mysql</strong> icon on the left to the editing area on the right and double-click the icon to go to the MySQL job configuration page. Configure parameters based on the following table.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24751__table10411111415287" frame="border" border="1" rules="all"><caption><b>Table 4 </b>MySQL job parameters</caption><thead align="left"><tr id="mrs_01_24751__row141212145288"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.7.2.1.1.3.2.3.1.1"><p id="mrs_01_24751__p24121142285">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.7.2.1.1.3.2.3.1.2"><p id="mrs_01_24751__p1141210147286">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24751__row124120140281"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__p8412214142810">Link</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__p19412101412820">mysqllink</p>
</td>
</tr>
<tr id="mrs_01_24751__row7412114142812"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__p641291419288">Tasks Max</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__p34123144287">1</p>
</td>
</tr>
<tr id="mrs_01_24751__row6412114202815"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__p16824101662217">Mode</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__p124120149289">insert, update, and delete</p>
</td>
</tr>
<tr id="mrs_01_24751__row1241241482819"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__p1335218238574">DB Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__p154132014142817">MYSQLDBA</p>
</td>
</tr>
<tr id="mrs_01_24751__row641318147280"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__p18536745175718">Schema Auto Create</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__p24131314202812">Yes</p>
</td>
</tr>
<tr id="mrs_01_24751__row111312068574"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__p2028410825714">Connect With Hudi</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__p1928415815573">Yes</p>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="mrs_01_24751__li44410384820">Click the plus sign (+) to display more parameters.<p id="mrs_01_24751__p4691370565"><a name="mrs_01_24751__li44410384820"></a><a name="li44410384820"></a><span><img id="mrs_01_24751__image6789124010381" src="en-us_image_0000001532632200.png"></span></p>
<div class="note" id="mrs_01_24751__note196144369536"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24751__ul1896191665411"><li id="mrs_01_24751__li1141153105918"><strong id="mrs_01_24751__b677148175711">PosFromBeginning</strong>: whether to capture CDC events from the start position</li><li id="mrs_01_24751__li46945471316"><strong id="mrs_01_24751__b1345214414578">DBZ Snapshot Locking Mode</strong>: lock mode when the connector is performing a snapshot. <strong id="mrs_01_24751__b11801121246">none</strong> indicates that no lock is held during the snapshot.</li><li id="mrs_01_24751__li38637129111"><strong id="mrs_01_24751__b87716524435944">WhiteList</strong>: Enter the database table to receive data, for example, <strong id="mrs_01_24751__b46434904235944">myclass</strong>.</li><li id="mrs_01_24751__li16889142116713"><strong id="mrs_01_24751__b113013351210">Blacklist</strong>: Enter the database table that does not need to capture data.</li><li id="mrs_01_24751__li195085453816"><strong id="mrs_01_24751__b103731138181214">Multi Partition</strong>: whether to enable Kafka partitioning.</li><li id="mrs_01_24751__li1213925553112"><strong id="mrs_01_24751__b87326978135944">Topic Table Mapping</strong><ul id="mrs_01_24751__ul6402209193312"><li id="mrs_01_24751__li1669122883219">This parameter is mandatory if <strong id="mrs_01_24751__b159179906935944">Connect With Hudi</strong> is set to <strong id="mrs_01_24751__b13226619435944">Yes</strong>.</li><li id="mrs_01_24751__li4127819155416">Enter the table name in the first text box, for example, <strong id="mrs_01_24751__b3298193521317">test</strong>. Enter a topic name in the second text box, for example, <strong id="mrs_01_24751__b18425144917136">test_topic</strong>. The topic name must match the table name in the first text box.</li></ul>
</li></ul>
</div></div>
</li><li id="mrs_01_24751__li5602185288">Click <strong id="mrs_01_24751__b121637969135944">OK</strong>. The MySQL job parameters are configured.</li></ol>
</p></li><li id="mrs_01_24751__li1736682513364"><span>Configure Hudi job parameters.</span><p><ol type="a" id="mrs_01_24751__en-us_topic_0000001446993949_ol5126919103712"><li id="mrs_01_24751__en-us_topic_0000001446993949_li1838281915919">On the <strong id="mrs_01_24751__en-us_topic_0000001446993949_b03141951155111">Job Management</strong> page, drag the <strong id="mrs_01_24751__en-us_topic_0000001446993949_b1231619511511">hudi</strong> icon in the Sink area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Configure parameters based on the following table:
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24751__en-us_topic_0000001446993949_table8874114418226" frame="border" border="1" rules="all"><caption><b>Table 5 </b>Sink Hudi job parameters</caption><thead align="left"><tr id="mrs_01_24751__en-us_topic_0000001446993949_row987511446222"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.8.2.1.1.3.2.3.1.1"><p id="mrs_01_24751__en-us_topic_0000001446993949_p58757449224">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.8.2.1.1.3.2.3.1.2"><p id="mrs_01_24751__en-us_topic_0000001446993949_p1887504472218">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24751__en-us_topic_0000001446993949_row1587674415221"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p12876104442216">Link</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p7876194411224">hudilink</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row35558544298"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p2555155414292">Path</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p6555854162917">/cdl/test</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row12712192213018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p147126226302">Interval</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p13712152214307">10</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row20793951153013"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p379455115308">Max Rate Per Partition</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p11794851153010">0</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row42214819316"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p162211283311">Parallelism</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p722168113117">10</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row676792412313"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p3767182412318">Target Hive Database</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p476712249313">default</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row1361784793113"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p161734715319">Configuring Hudi Table Attributes</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p1061712477312">Visual View</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row2067113416323"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p146712034183218">Global Configuration of Hudi Table Attributes</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p106771234196">-</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row42681465338"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p142688618336">Configuring the Attributes of the Hudi Table: Table Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p72681867333">test</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row1910916203342"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p8109122010343">Configuring the Attributes of the Hudi Table: Table Type Opt Key</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p1710942033418">COPY_ON_WRITE</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row19141834163410"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p1790210503342">Configuring the Attributes of the Hudi Table: Hudi TableName Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p9914163483419">-</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row647111417340"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p777795153416">Configuring the Attributes of the Hudi Table: Hive TableName Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p247215414341">-</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row17458174623411"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p195211652153416">Configuring the Attributes of the Hudi Table: Table Primarykey Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p045954613341">id</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row122962024153511"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p112969244352">Configuring the Attributes of the Hudi Table: Table Hudi Partition Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p1629615242355">-</p>
</td>
</tr>
<tr id="mrs_01_24751__en-us_topic_0000001446993949_row745311201363"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.1 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p15906132116367">Configuring the Attributes of the Hudi Table: Custom Config</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.8.2.1.1.3.2.3.1.2 "><p id="mrs_01_24751__en-us_topic_0000001446993949_p124542206369">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24751__en-us_topic_0000001446993949_p4903131211378"><span><img id="mrs_01_24751__en-us_topic_0000001446993949_image691514311374" src="en-us_image_0000001537076386.png"></span></p>
</li><li id="mrs_01_24751__en-us_topic_0000001446993949_li1287784482210">(Optional) Click the plus sign (+) to display the <strong id="mrs_01_24751__en-us_topic_0000001446993949_b37240125495">Execution Env</strong> parameter. Select a created environment for it. The default value is <strong id="mrs_01_24751__en-us_topic_0000001446993949_b8790104014502">defaultEnv</strong>.<p id="mrs_01_24751__en-us_topic_0000001446993949_p88771944172217"><span><img id="mrs_01_24751__en-us_topic_0000001446993949_image138771844162215" src="en-us_image_0000001587875989.png"></span></p>
</li><li id="mrs_01_24751__en-us_topic_0000001446993949_li17384171782510">Click <strong id="mrs_01_24751__en-us_topic_0000001446993949_b12926539144311">OK</strong>.</li></ol>
</p></li><li id="mrs_01_24751__li032595183911"><span>Drag the two icons to associate the job parameters and click <strong id="mrs_01_24751__b153130588335944">Save</strong>. The job configuration is complete.</span><p><p id="mrs_01_24751__p38715177149"><span><img id="mrs_01_24751__image146935173144" src="en-us_image_0000001532472728.png"></span></p>
</p></li><li id="mrs_01_24751__li7968152124413"><span>In the job list on the <strong id="mrs_01_24751__b51128986935944">Job Management</strong> page, locate the created job, click <strong id="mrs_01_24751__b82355133035944">Start</strong> in the <strong id="mrs_01_24751__b102317454635944">Operation</strong> column, and wait until the job is started.</span><p><p id="mrs_01_24751__p29726217442">Check whether the data transmission takes effect, for example, insert data into the table in the MySQL database and view the content of the file imported to Hudi.</p>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24240.html">Common CDL Jobs</a></div>
</div>
</div>
View Git Blame Copy Permalink