vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_24752.html
Yang, Tong 48706b7552 MRS COMP-LTS 320-lts.1 version 
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2024-04-12 12:51:10 +00:00

340 lines
32 KiB
HTML
Raw Blame History

<a name="mrs_01_24752"></a><a name="mrs_01_24752"></a>
<h1 class="topictitle1">Synchronizing Data from PgSQL to Hudi</h1>
<div id="body0000001583271049"><div class="section" id="mrs_01_24752__section7722164042318"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24752__p664594413238">This section describes how to import data from PgSQL to Hudi by using the CDLService web UI of a cluster with Kerberos authentication enabled.</p>
</div>
<div class="section" id="mrs_01_24752__section1438820494231"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24752__ul2041871362311"><li id="mrs_01_24752__li12418213122313">The CDL and Hudi services have been installed in a cluster and are running properly.</li><li id="mrs_01_24752__li362910581468">The prerequisites for the PgSQL database have been met. For details, see <a href="mrs_01_24124.html#mrs_01_24124__li1868193914169">Policy for Modifying Write-Ahead Logs in PostgreSQL Databases</a>.</li><li id="mrs_01_24752__li1441819135232">You have created a human-machine user, for example, <strong id="mrs_01_24752__b1569255182919">cdluser</strong>, added the user to user groups <strong id="mrs_01_24752__b4698125122913">cdladmin</strong> (primary group), <strong id="mrs_01_24752__b17698251122911">hadoop</strong>, <strong id="mrs_01_24752__b56992511295">kafka</strong>, and <strong id="mrs_01_24752__b116995512298">supergroup</strong>, and associated the user with the <strong id="mrs_01_24752__b16699351152915">System_administrator</strong> role on FusionInsight Manager.</li></ul>
</div>
<div class="section" id="mrs_01_24752__section19362472245"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24752__ol18871133225317"><li id="mrs_01_24752__li1365211683614"><span>Log in to FusionInsight Manager as user <strong id="mrs_01_24752__b335617196198">cdluser</strong> (change the password upon your first login), choose <strong id="mrs_01_24752__b1935661951916">Cluster</strong> &gt; <strong id="mrs_01_24752__b63566197199">Services</strong> &gt; <strong id="mrs_01_24752__b63571819151913">CDL</strong>, and click the link next to <strong id="mrs_01_24752__b9357161910192">CDLService UI</strong> to go to the CDLService web UI.</span></li><li id="mrs_01_24752__li3618121145616"><span>Choose <strong id="mrs_01_24752__b187715596196">Link Management</strong> and click <strong id="mrs_01_24752__b1883165915194">Add Link</strong>. In the displayed dialog box, set parameters for adding the <strong id="mrs_01_24752__b1283195916194">pgsql</strong> and <strong id="mrs_01_24752__b1584259141913">hudi</strong> links by referring to the following tables.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24752__table134120364291" frame="border" border="1" rules="all"><caption><b>Table 1 </b>PgSQL data link parameters</caption><thead align="left"><tr id="mrs_01_24752__row2041836162918"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.1.2.3.1.1"><p id="mrs_01_24752__p4411136172915">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.1.2.3.1.2"><p id="mrs_01_24752__p19411736142919">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24752__row10411236152920"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24752__p154119368298">Link Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24752__p154153632911">pgsql</p>
</td>
</tr>
<tr id="mrs_01_24752__row16426365294"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24752__p44220365290">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24752__p104253620293">pgsqllink</p>
</td>
</tr>
<tr id="mrs_01_24752__row442636152915"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24752__p9427368295">Host</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24752__p17303172811553">10.10.10.10</p>
</td>
</tr>
<tr id="mrs_01_24752__row11423363293"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24752__p10421364293">Port</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24752__p17303122811553">5432</p>
</td>
</tr>
<tr id="mrs_01_24752__row3895752111910"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24752__p14895155281911">DB Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24752__p15895175221913">testDB</p>
</td>
</tr>
<tr id="mrs_01_24752__row164263622915"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24752__p1442636162910">User</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24752__p4916845152114">user</p>
</td>
</tr>
<tr id="mrs_01_24752__row104317366296"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24752__p194314367297">Password</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24752__p391712458215"><em id="mrs_01_24752__i26742183234153">Password of the <strong id="mrs_01_24752__b2084242310267">user</strong> user</em></p>
</td>
</tr>
<tr id="mrs_01_24752__row124313614299"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24752__p74353613299">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24752__p184383615296">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24752__table1437691715220" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Hudi data link parameters</caption><thead align="left"><tr id="mrs_01_24752__row1437618171029"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.2.2.3.1.1"><p id="mrs_01_24752__p137651713218">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.2.2.3.1.2"><p id="mrs_01_24752__p1237720171128">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24752__row33815178212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24752__p153814179219">Link Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24752__p938112176210">hudi</p>
</td>
</tr>
<tr id="mrs_01_24752__row193811117822"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24752__p138161715211">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24752__p1738113173219">hudilink</p>
</td>
</tr>
<tr id="mrs_01_24752__row83811417425"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24752__p125527208312">Storage Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24752__p35331203318">hdfs</p>
</td>
</tr>
<tr id="mrs_01_24752__row736874119319"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24752__p1814611463913">Auth KeytabFile</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24752__p1214617466915">/opt/Bigdata/third_lib/CDL/user_libs/cdluser.keytab</p>
</td>
</tr>
<tr id="mrs_01_24752__row936915411130"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24752__p14146546699">Principal</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24752__p1414619465918">cdluser</p>
</td>
</tr>
<tr id="mrs_01_24752__row938313177210"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24752__p1938310171727">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24752__p5383161716218">xxx</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="mrs_01_24752__li1829512373488"><span>After the parameters are configured, click <strong id="mrs_01_24752__b185001741202612">Test</strong> to check whether the data link is normal.</span><p><p id="mrs_01_24752__p13530140134817">After the test is successful, click <strong id="mrs_01_24752__b10351943142618">OK</strong>.</p>
</p></li><li id="mrs_01_24752__li976974910717"><span>(Optional) Choose <strong id="mrs_01_24752__en-us_topic_0000001446993949_b45961936164618">ENV Management</strong> and click <strong id="mrs_01_24752__en-us_topic_0000001446993949_b03551454164613">Add Env</strong>. In the displayed dialog box, configure the parameters based on the following table.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24752__en-us_topic_0000001446993949_table54201432181218" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Parameters for adding an ENV</caption><thead align="left"><tr id="mrs_01_24752__en-us_topic_0000001446993949_row10418153291216"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.1.2.3.1.1"><p id="mrs_01_24752__en-us_topic_0000001446993949_p18418032161213">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.1.2.3.1.2"><p id="mrs_01_24752__en-us_topic_0000001446993949_p441810320123">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24752__en-us_topic_0000001446993949_row7419193251216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p1141815324123">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p14419183241213">test-env</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row04191332161212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p18419133214122">Driver Memory</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p341923215124">1 GB</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row1468718181894"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p156889181495">Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p106881618096">spark</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row2419632151212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p941918323125">Executor Memory</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p1419832121219">1 GB</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row20419143211122"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p041933213124">Executor Cores</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p1341983217125">1</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row341915321123"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p194191032181217">Number Executors</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p841933251210">1</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row77610527328"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p1525831203317">Queue</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p17675211325">-</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row34191032171216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p94191032181210">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p5419193231211">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24752__en-us_topic_0000001446993949_p145511354104314">Click <strong id="mrs_01_24752__en-us_topic_0000001446993949_b1436468274744">OK</strong>.</p>
</p></li><li id="mrs_01_24752__li8419191320242"><span>Choose <strong id="mrs_01_24752__b09311565271">Job Management</strong> &gt; <strong id="mrs_01_24752__b189387618272">Data synchronization task</strong> and click <strong id="mrs_01_24752__b17938176142717">Add Job</strong>. In the displayed dialog box, set parameters and click <strong id="mrs_01_24752__b189381762273">Next</strong>.</span><p><div class="p" id="mrs_01_24752__p1442915568513">Parameters are as follows.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24752__table1379713112014" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_24752__row13797011905"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.1.1.3.1.1"><p id="mrs_01_24752__p17797411309">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.1.1.3.1.2"><p id="mrs_01_24752__p137971011309">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24752__row1679841118018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.1 "><p id="mrs_01_24752__p391613171005">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.2 "><p id="mrs_01_24752__p149141171404">job_pgtohudi</p>
</td>
</tr>
<tr id="mrs_01_24752__row779916111602"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.1 "><p id="mrs_01_24752__p83069155152">Desc</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.2 "><p id="mrs_01_24752__p14306191511512">-</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</p></li><li id="mrs_01_24752__li2737114320203"><span>Configure PgSQL job parameters.</span><p><ol type="a" id="mrs_01_24752__ol14971759152111"><li id="mrs_01_24752__li637518545212">On the <strong id="mrs_01_24752__b203177219278">Job Management</strong> page, drag the <strong id="mrs_01_24752__b132392119271">pgsql</strong> icon on the left to the editing area on the right and double-click the icon to go to the PgSQL job configuration page. Configure parameters based on the following table.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24752__table10411111415287" frame="border" border="1" rules="all"><caption><b>Table 4 </b>PgSQL job parameters</caption><thead align="left"><tr id="mrs_01_24752__row141212145288"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.3.2.3.1.1"><p id="mrs_01_24752__p24121142285">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.3.2.3.1.2"><p id="mrs_01_24752__p1141210147286">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24752__row124120140281"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p8412214142810">Link</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p19412101412820">pgsqllink</p>
</td>
</tr>
<tr id="mrs_01_24752__row7412114142812"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p641291419288">Tasks Max</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p34123144287">1</p>
</td>
</tr>
<tr id="mrs_01_24752__row6412114202815"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p16824101662217">Mode</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p124120149289">insert, update, and delete</p>
</td>
</tr>
<tr id="mrs_01_24752__row1492515218238"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p1192617529232">dbName Alias</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p1292655202313">pgsqldb</p>
</td>
</tr>
<tr id="mrs_01_24752__row1241241482819"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p541361452815">Schema</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p154132014142817">pgschema</p>
</td>
</tr>
<tr id="mrs_01_24752__row641318147280"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p441331418283">Slot Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p24131314202812">pg_slot</p>
</td>
</tr>
<tr id="mrs_01_24752__row452704772510"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p1852764762516">Enable FailOver Slot</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p65278478255">No</p>
</td>
</tr>
<tr id="mrs_01_24752__row1153218113268"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p353319113267">Slot Drop</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p653312162613">No</p>
</td>
</tr>
<tr id="mrs_01_24752__row111312068574"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p2028410825714">Connect With Hudi</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p1928415815573">Yes</p>
</td>
</tr>
<tr id="mrs_01_24752__row1534114388263"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p8341193842612">Use Exist Publication</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p1341138162618">No</p>
</td>
</tr>
<tr id="mrs_01_24752__row223718182710"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__p72373818278">Publication Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__p10237788274">publicationtest</p>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="mrs_01_24752__li44410384820">Click the plus sign (+) to display more parameters.<p id="mrs_01_24752__p4691370565"><a name="mrs_01_24752__li44410384820"></a><a name="li44410384820"></a><span><img id="mrs_01_24752__image15525055162718" src="en-us_image_0000001532791948.png"></span></p>
<div class="note" id="mrs_01_24752__note196144369536"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24752__ul1896191665411"><li id="mrs_01_24752__li209631617544"><strong id="mrs_01_24752__b3288111615285">Start Time</strong>: indicates the start time of table synchronization.</li><li id="mrs_01_24752__li16560114132818"><strong id="mrs_01_24752__b43161032102810">WhiteList</strong>: Enter a table in the database.</li><li id="mrs_01_24752__li1163193182915"><strong id="mrs_01_24752__b16532133305912">Blacklist</strong>: Enter the database table that does not need to capture data.</li><li id="mrs_01_24752__li1213925553112"><strong id="mrs_01_24752__b89734589834153">Topic Table Mapping</strong><ul id="mrs_01_24752__ul6402209193312"><li id="mrs_01_24752__li1669122883219">This parameter is mandatory if <strong id="mrs_01_24752__b127108366336">Connect With Hudi</strong> is set to <strong id="mrs_01_24752__b207101936193311">Yes</strong>.</li><li id="mrs_01_24752__li4127819155416">Enter the table name in the first text box, for example, <strong id="mrs_01_24752__b1884253813339">test</strong>. Enter a topic name in the second text box, for example, <strong id="mrs_01_24752__b12916184211336">test_topic</strong>. The topic name must match the table name in the first text box.</li></ul>
</li></ul>
</div></div>
</li><li id="mrs_01_24752__li5602185288">Click <strong id="mrs_01_24752__b81982917634153">OK</strong>. The PgSQL job parameters are configured.</li></ol>
</p></li><li id="mrs_01_24752__li1736682513364"><span>Configure Hudi job parameters.</span><p><ol type="a" id="mrs_01_24752__en-us_topic_0000001446993949_ol5126919103712"><li id="mrs_01_24752__en-us_topic_0000001446993949_li1838281915919">On the <strong id="mrs_01_24752__en-us_topic_0000001446993949_b03141951155111">Job Management</strong> page, drag the <strong id="mrs_01_24752__en-us_topic_0000001446993949_b1231619511511">hudi</strong> icon in the Sink area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Configure parameters based on the following table:
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24752__en-us_topic_0000001446993949_table8874114418226" frame="border" border="1" rules="all"><caption><b>Table 5 </b>Sink Hudi job parameters</caption><thead align="left"><tr id="mrs_01_24752__en-us_topic_0000001446993949_row987511446222"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.7.2.1.1.3.2.3.1.1"><p id="mrs_01_24752__en-us_topic_0000001446993949_p58757449224">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.7.2.1.1.3.2.3.1.2"><p id="mrs_01_24752__en-us_topic_0000001446993949_p1887504472218">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24752__en-us_topic_0000001446993949_row1587674415221"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p12876104442216">Link</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p7876194411224">hudilink</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row35558544298"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p2555155414292">Path</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p6555854162917">/cdl/test</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row12712192213018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p147126226302">Interval</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p13712152214307">10</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row20793951153013"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p379455115308">Max Rate Per Partition</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p11794851153010">0</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row42214819316"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p162211283311">Parallelism</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p722168113117">10</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row676792412313"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p3767182412318">Target Hive Database</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p476712249313">default</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row1361784793113"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p161734715319">Configuring Hudi Table Attributes</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p1061712477312">Visual View</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row2067113416323"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p146712034183218">Global Configuration of Hudi Table Attributes</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p106771234196">-</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row42681465338"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p142688618336">Configuring the Attributes of the Hudi Table: Table Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p72681867333">test</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row1910916203342"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p8109122010343">Configuring the Attributes of the Hudi Table: Table Type Opt Key</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p1710942033418">COPY_ON_WRITE</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row19141834163410"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p1790210503342">Configuring the Attributes of the Hudi Table: Hudi TableName Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p9914163483419">-</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row647111417340"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p777795153416">Configuring the Attributes of the Hudi Table: Hive TableName Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p247215414341">-</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row17458174623411"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p195211652153416">Configuring the Attributes of the Hudi Table: Table Primarykey Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p045954613341">id</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row122962024153511"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p112969244352">Configuring the Attributes of the Hudi Table: Table Hudi Partition Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p1629615242355">-</p>
</td>
</tr>
<tr id="mrs_01_24752__en-us_topic_0000001446993949_row745311201363"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p15906132116367">Configuring the Attributes of the Hudi Table: Custom Config</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24752__en-us_topic_0000001446993949_p124542206369">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24752__en-us_topic_0000001446993949_p4903131211378"><span><img id="mrs_01_24752__en-us_topic_0000001446993949_image691514311374" src="en-us_image_0000001446755301.png"></span></p>
</li><li id="mrs_01_24752__en-us_topic_0000001446993949_li1287784482210">(Optional) Click the plus sign (+) to display the <strong id="mrs_01_24752__en-us_topic_0000001446993949_b37240125495">Execution Env</strong> parameter. Select a created environment for it. The default value is <strong id="mrs_01_24752__en-us_topic_0000001446993949_b8790104014502">defaultEnv</strong>.<p id="mrs_01_24752__en-us_topic_0000001446993949_p88771944172217"><span><img id="mrs_01_24752__en-us_topic_0000001446993949_image138771844162215" src="en-us_image_0000001446835121.png"></span></p>
</li><li id="mrs_01_24752__en-us_topic_0000001446993949_li17384171782510">Click <strong id="mrs_01_24752__en-us_topic_0000001446993949_b12926539144311">OK</strong>.</li></ol>
</p></li><li id="mrs_01_24752__li032595183911"><span>Drag the two icons to associate the job parameters and click <strong id="mrs_01_24752__b949412718346">Save</strong>. The job configuration is complete.</span><p><p id="mrs_01_24752__p1142733891316"><span><img id="mrs_01_24752__image1962193841313" src="en-us_image_0000001532951884.png"></span></p>
</p></li><li id="mrs_01_24752__li7968152124413"><span>In the job list on the <strong id="mrs_01_24752__b1866915193416">Job Management</strong> page, locate the created job, click <strong id="mrs_01_24752__b972115163416">Start</strong> in the <strong id="mrs_01_24752__b137381517348">Operation</strong> column, and wait until the job is started.</span><p><p id="mrs_01_24752__p29726217442">Check whether the data transmission takes effect, for example, insert data into the table in the PgSQL database and view the content of the file imported to Hudi.</p>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24240.html">Common CDL Jobs</a></div>
</div>
</div>
View Git Blame Copy Permalink