vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_24763.html
Yang, Tong 48706b7552 MRS COMP-LTS 320-lts.1 version 
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2024-04-12 12:51:10 +00:00

372 lines
34 KiB
HTML
Raw Blame History

<a name="mrs_01_24763"></a><a name="mrs_01_24763"></a>
<h1 class="topictitle1">Synchronizing Data from ThirdKafka to Hudi</h1>
<div id="body0000001532471616"><div class="section" id="mrs_01_24763__section7722164042318"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24763__p664594413238">This section describes how to import data from ThirdKafka to Hudi by using the CDLService web UI of a cluster with Kerberos authentication enabled.</p>
</div>
<div class="section" id="mrs_01_24763__section1438820494231"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24763__ul2041871362311"><li id="mrs_01_24763__li12418213122313">The CDL and Hudi services have been installed in a cluster and are running properly.</li><li id="mrs_01_24763__li362910581468">Topics of the ThirdKafka database can be consumed by the MRS cluster. For details, see <a href="mrs_01_24124.html#mrs_01_24124__li347115587209">PrerequisitesforThirdPartyKafka</a>.</li><li id="mrs_01_24763__li1441819135232">You have created a human-machine user, for example, <strong id="mrs_01_24763__b270516018300">cdluser</strong>, added the user to user groups <strong id="mrs_01_24763__b127114016305">cdladmin</strong> (primary group), <strong id="mrs_01_24763__b19711190133013">hadoop</strong>, <strong id="mrs_01_24763__b671215012307">kafka</strong>, and <strong id="mrs_01_24763__b17123093017">supergroup</strong>, and associated the user with the <strong id="mrs_01_24763__b1771310014309">System_administrator</strong> role on FusionInsight Manager.</li></ul>
</div>
<div class="section" id="mrs_01_24763__section19362472245"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24763__ol18871133225317"><li id="mrs_01_24763__li1365211683614"><span>Log in to FusionInsight Manager as user <strong id="mrs_01_24763__b53871418133716">cdluser</strong> (change the password upon your first login), choose <strong id="mrs_01_24763__b538761810377">Cluster</strong> &gt; <strong id="mrs_01_24763__b538815182370">Services</strong> &gt; <strong id="mrs_01_24763__b15388141815371">CDL</strong>, and click the link next to <strong id="mrs_01_24763__b4388121893711">CDLService UI</strong> to go to the CDLService web UI.</span></li><li id="mrs_01_24763__li3618121145616"><span>Choose <strong id="mrs_01_24763__b1999018277389">Link Management</strong> and click <strong id="mrs_01_24763__b1799616272382">Add Link</strong>. In the displayed dialog box, set parameters for adding the <strong id="mrs_01_24763__b19996327183816">thirdparty-kafka</strong> and <strong id="mrs_01_24763__b1199752773816">hudi</strong> links by referring to the following tables.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24763__table134120364291" frame="border" border="1" rules="all"><caption><b>Table 1 </b>thirdparty-kafka data link parameters</caption><thead align="left"><tr id="mrs_01_24763__row2041836162918"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.1.2.3.1.1"><p id="mrs_01_24763__p4411136172915">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.1.2.3.1.2"><p id="mrs_01_24763__p19411736142919">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24763__row152710381074"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p4272153820713">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p1827243817711">opengausslink</p>
</td>
</tr>
<tr id="mrs_01_24763__row15621741186"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p55621641180">Link Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p17562134114812">thirdparty-kafka</p>
</td>
</tr>
<tr id="mrs_01_24763__row57691582916"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p776998695">Bootstrap Servers</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p1777058692">10.10.10.10:9093</p>
</td>
</tr>
<tr id="mrs_01_24763__row155168121018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p253831016">Security Protocol</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p16568131011">SASL_SSL</p>
</td>
</tr>
<tr id="mrs_01_24763__row18608193591018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p960873551019">Username</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p6608123516101">testuser</p>
</td>
</tr>
<tr id="mrs_01_24763__row63735317117"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p53740311117">Password</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p20374331101115"><em id="mrs_01_24763__i4204206143920">Password of the <strong id="mrs_01_24763__b46844914618">testuser</strong> user</em></p>
</td>
</tr>
<tr id="mrs_01_24763__row34061455201118"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p14071550115">SSL Truststore Location</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p1940775520113">Click <strong id="mrs_01_24763__b147957241135">Upload</strong> to upload the authentication file.</p>
</td>
</tr>
<tr id="mrs_01_24763__row174818711317"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p04818720137">SSL Truststore Password</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p124817791310">-</p>
</td>
</tr>
<tr id="mrs_01_24763__row733315614148"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p73335671419">Datastore Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p173338661420">opengauss</p>
</td>
</tr>
<tr id="mrs_01_24763__row8783134101410"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p12784934151414">Host</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p878443412148">11.11.xxx.xxx,12.12.xxx.xxx</p>
</td>
</tr>
<tr id="mrs_01_24763__row4401315191511"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p164161512159">Port</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p841141520158">8000</p>
</td>
</tr>
<tr id="mrs_01_24763__row45605459157"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p956174561510">DB Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p656154514150">opengaussdb</p>
</td>
</tr>
<tr id="mrs_01_24763__row081915125167"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p78191512101618">User</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p28191312121619">opengaussuser</p>
</td>
</tr>
<tr id="mrs_01_24763__row14117193313168"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p17117193315161">DB Password</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p1926716379148"><em id="mrs_01_24763__i186584511315">Password of the <strong id="mrs_01_24763__b427817491538">opengaussuser</strong> user</em></p>
</td>
</tr>
<tr id="mrs_01_24763__row124313614299"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24763__p74353613299">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24763__p184383615296">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="mrs_01_24763__note6582124419562"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24763__p137611717226">MRS Kafka can also be used as the source of thirdparty-kafka. If the username and password are used for login authentication, log in to FusionInsight Manager, choose <strong id="mrs_01_24763__b9939125912914">Cluster</strong> &gt; <strong id="mrs_01_24763__b536843103019">Services</strong> &gt; <strong id="mrs_01_24763__b114433716303">Kafka</strong>, click <strong id="mrs_01_24763__b681682112309">Configuration</strong>, search for the <strong id="mrs_01_24763__b12831130203015">sasl.enabled.mechanisms</strong> parameter in the search box, add <strong id="mrs_01_24763__b10709173083513">PLAIN</strong> as the parameter value, click <strong id="mrs_01_24763__b199741146380">Save</strong> to save the configuration, and restart the Kafka service for the configuration to take effect.</p>
<p id="mrs_01_24763__p18858325162214"><span><img id="mrs_01_24763__image1577531117162" src="en-us_image_0000001583151865.png"></span></p>
<p id="mrs_01_24763__p115831944155617">On the CDL web UI, configure the thirdparty-kafka link that uses MRS Kafka as the source. For example, the data link configuration is as follows:</p>
<p id="mrs_01_24763__p59661756105714"><span><img id="mrs_01_24763__image378722515586" src="en-us_image_0000001583391861.png"></span></p>
</div></div>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24763__table1437691715220" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Hudi data link parameters</caption><thead align="left"><tr id="mrs_01_24763__row1437618171029"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.3.2.3.1.1"><p id="mrs_01_24763__p137651713218">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.3.2.3.1.2"><p id="mrs_01_24763__p1237720171128">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24763__row33815178212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.1 "><p id="mrs_01_24763__p153814179219">Link Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.2 "><p id="mrs_01_24763__p938112176210">hudi</p>
</td>
</tr>
<tr id="mrs_01_24763__row193811117822"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.1 "><p id="mrs_01_24763__p138161715211">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.2 "><p id="mrs_01_24763__p1738113173219">hudilink</p>
</td>
</tr>
<tr id="mrs_01_24763__row83811417425"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.1 "><p id="mrs_01_24763__p125527208312">Storage Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.2 "><p id="mrs_01_24763__p35331203318">hdfs</p>
</td>
</tr>
<tr id="mrs_01_24763__row736874119319"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.1 "><p id="mrs_01_24763__p1814611463913">Auth KeytabFile</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.2 "><p id="mrs_01_24763__p1214617466915">/opt/Bigdata/third_lib/CDL/user_libs/cdluser.keytab</p>
</td>
</tr>
<tr id="mrs_01_24763__row936915411130"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.1 "><p id="mrs_01_24763__p14146546699">Principal</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.2 "><p id="mrs_01_24763__p1414619465918">cdluser</p>
</td>
</tr>
<tr id="mrs_01_24763__row938313177210"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.1 "><p id="mrs_01_24763__p1938310171727">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.3.2.3.1.2 "><p id="mrs_01_24763__p5383161716218">-</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="mrs_01_24763__li1829512373488"><span>After the parameters are configured, click <strong id="mrs_01_24763__b17334591137">Test</strong> to check whether the data link is normal.</span><p><p id="mrs_01_24763__p13530140134817">After the test is successful, click <strong id="mrs_01_24763__b1172060443">OK</strong>.</p>
</p></li><li id="mrs_01_24763__li976974910717"><span>(Optional) Choose <strong id="mrs_01_24763__en-us_topic_0000001446993949_b45961936164618">ENV Management</strong> and click <strong id="mrs_01_24763__en-us_topic_0000001446993949_b03551454164613">Add Env</strong>. In the displayed dialog box, configure the parameters based on the following table.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24763__en-us_topic_0000001446993949_table54201432181218" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Parameters for adding an ENV</caption><thead align="left"><tr id="mrs_01_24763__en-us_topic_0000001446993949_row10418153291216"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.1.2.3.1.1"><p id="mrs_01_24763__en-us_topic_0000001446993949_p18418032161213">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.1.2.3.1.2"><p id="mrs_01_24763__en-us_topic_0000001446993949_p441810320123">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24763__en-us_topic_0000001446993949_row7419193251216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p1141815324123">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p14419183241213">test-env</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row04191332161212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p18419133214122">Driver Memory</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p341923215124">1 GB</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row1468718181894"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p156889181495">Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p106881618096">spark</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row2419632151212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p941918323125">Executor Memory</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p1419832121219">1 GB</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row20419143211122"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p041933213124">Executor Cores</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p1341983217125">1</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row341915321123"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p194191032181217">Number Executors</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p841933251210">1</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row77610527328"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p1525831203317">Queue</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p17675211325">-</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row34191032171216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p94191032181210">Description</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p5419193231211">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24763__en-us_topic_0000001446993949_p145511354104314">Click <strong id="mrs_01_24763__en-us_topic_0000001446993949_b1436468274744">OK</strong>.</p>
</p></li><li id="mrs_01_24763__li8419191320242"><span>Choose <strong id="mrs_01_24763__b1058671512412">Job Management</strong> &gt; <strong id="mrs_01_24763__b159211519412">Data synchronization task</strong> and click <strong id="mrs_01_24763__b19592915547">Add Job</strong>. In the displayed dialog box, set parameters and click <strong id="mrs_01_24763__b1759281515411">Next</strong>.</span><p><div class="p" id="mrs_01_24763__p1442915568513">Parameters are as follows.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24763__table1379713112014" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_24763__row13797011905"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.1.1.3.1.1"><p id="mrs_01_24763__p17797411309">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.1.1.3.1.2"><p id="mrs_01_24763__p137971011309">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24763__row1679841118018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.1 "><p id="mrs_01_24763__p391613171005">Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.2 "><p id="mrs_01_24763__p149141171404">job_opengausstohudi</p>
</td>
</tr>
<tr id="mrs_01_24763__row779916111602"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.1 "><p id="mrs_01_24763__p83069155152">Desc</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.1.3.1.2 "><p id="mrs_01_24763__p14306191511512">New CDL Job</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</p></li><li id="mrs_01_24763__li2737114320203"><span>Configure ThirdKafka job parameters.</span><p><ol type="a" id="mrs_01_24763__ol14971759152111"><li id="mrs_01_24763__li637518545212">On the <strong id="mrs_01_24763__b16390709544010">Job Management</strong> page, drag the <strong id="mrs_01_24763__b2909924574010">thirdparty-kafka</strong> icon on the left to the editing area on the right and double-click the icon to go to the ThirdpartyKafka job configuration page. Configure parameters based on the following table.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24763__table10411111415287" frame="border" border="1" rules="all"><caption><b>Table 4 </b>thirdparty-kafka job parameters</caption><thead align="left"><tr id="mrs_01_24763__row141212145288"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.3.2.3.1.1"><p id="mrs_01_24763__p24121142285">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.3.2.3.1.2"><p id="mrs_01_24763__p1141210147286">Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24763__row124120140281"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p8412214142810">Link</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p19412101412820">opengausslink</p>
</td>
</tr>
<tr id="mrs_01_24763__row7815453217"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p881605162111">DB Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p128162512210">opengaussdb</p>
</td>
</tr>
<tr id="mrs_01_24763__row11927529132218"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p169270299228">Schema</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p5927162972220">opengaussschema</p>
</td>
</tr>
<tr id="mrs_01_24763__row45108105231"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p45111110172310">Datastore Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p18511181010235">opengauss</p>
</td>
</tr>
<tr id="mrs_01_24763__row183443813234"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p283413811230">Source Topics</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p2083417387232">source_topic</p>
</td>
</tr>
<tr id="mrs_01_24763__row19381447192414"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p163894715241">Tasks Max</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p53894716245">1</p>
</td>
</tr>
<tr id="mrs_01_24763__row11823188162520"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p182318818251">Tolerance</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p118231817254">none</p>
</td>
</tr>
<tr id="mrs_01_24763__row1492716474261"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p49272473266">Start Time</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p99271471265">-</p>
</td>
</tr>
<tr id="mrs_01_24763__row19752100152717"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p1075217072711">Multi Partition</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p1075219015272">No</p>
</td>
</tr>
<tr id="mrs_01_24763__row10314192615275"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__p15314112616272">Topic Table Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__p1731422692717">test/hudi_topic</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24763__p26158357280"><span><img id="mrs_01_24763__image193551453122817" src="en-us_image_0000001583272169.png"></span></p>
</li><li id="mrs_01_24763__li5602185288">Click <strong id="mrs_01_24763__b16292031434010">OK</strong>. The ThirdpartyKafka job parameters are configured.</li></ol>
</p></li><li id="mrs_01_24763__li1736682513364"><span>Configure Hudi job parameters.</span><p><ol type="a" id="mrs_01_24763__en-us_topic_0000001446993949_ol5126919103712"><li id="mrs_01_24763__en-us_topic_0000001446993949_li1838281915919">On the <strong id="mrs_01_24763__en-us_topic_0000001446993949_b03141951155111">Job Management</strong> page, drag the <strong id="mrs_01_24763__en-us_topic_0000001446993949_b1231619511511">hudi</strong> icon in the Sink area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Configure parameters based on the following table:
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24763__en-us_topic_0000001446993949_table8874114418226" frame="border" border="1" rules="all"><caption><b>Table 5 </b>Sink Hudi job parameters</caption><thead align="left"><tr id="mrs_01_24763__en-us_topic_0000001446993949_row987511446222"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.7.2.1.1.3.2.3.1.1"><p id="mrs_01_24763__en-us_topic_0000001446993949_p58757449224">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.7.2.1.1.3.2.3.1.2"><p id="mrs_01_24763__en-us_topic_0000001446993949_p1887504472218">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24763__en-us_topic_0000001446993949_row1587674415221"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p12876104442216">Link</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p7876194411224">hudilink</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row35558544298"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p2555155414292">Path</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p6555854162917">/cdl/test</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row12712192213018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p147126226302">Interval</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p13712152214307">10</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row20793951153013"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p379455115308">Max Rate Per Partition</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p11794851153010">0</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row42214819316"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p162211283311">Parallelism</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p722168113117">10</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row676792412313"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p3767182412318">Target Hive Database</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p476712249313">default</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row1361784793113"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p161734715319">Configuring Hudi Table Attributes</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p1061712477312">Visual View</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row2067113416323"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p146712034183218">Global Configuration of Hudi Table Attributes</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p106771234196">-</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row42681465338"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p142688618336">Configuring the Attributes of the Hudi Table: Table Name</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p72681867333">test</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row1910916203342"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p8109122010343">Configuring the Attributes of the Hudi Table: Table Type Opt Key</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p1710942033418">COPY_ON_WRITE</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row19141834163410"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p1790210503342">Configuring the Attributes of the Hudi Table: Hudi TableName Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p9914163483419">-</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row647111417340"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p777795153416">Configuring the Attributes of the Hudi Table: Hive TableName Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p247215414341">-</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row17458174623411"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p195211652153416">Configuring the Attributes of the Hudi Table: Table Primarykey Mapping</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p045954613341">id</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row122962024153511"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p112969244352">Configuring the Attributes of the Hudi Table: Table Hudi Partition Type</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p1629615242355">-</p>
</td>
</tr>
<tr id="mrs_01_24763__en-us_topic_0000001446993949_row745311201363"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.1 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p15906132116367">Configuring the Attributes of the Hudi Table: Custom Config</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.7.2.1.1.3.2.3.1.2 "><p id="mrs_01_24763__en-us_topic_0000001446993949_p124542206369">-</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="mrs_01_24763__en-us_topic_0000001446993949_p4903131211378"><span><img id="mrs_01_24763__en-us_topic_0000001446993949_image691514311374" src="en-us_image_0000001587755985.png"></span></p>
</li><li id="mrs_01_24763__en-us_topic_0000001446993949_li1287784482210">(Optional) Click the plus sign (+) to display the <strong id="mrs_01_24763__en-us_topic_0000001446993949_b37240125495">Execution Env</strong> parameter. Select a created environment for it. The default value is <strong id="mrs_01_24763__en-us_topic_0000001446993949_b8790104014502">defaultEnv</strong>.<p id="mrs_01_24763__en-us_topic_0000001446993949_p88771944172217"><span><img id="mrs_01_24763__en-us_topic_0000001446993949_image138771844162215" src="en-us_image_0000001536916934.png"></span></p>
</li><li id="mrs_01_24763__en-us_topic_0000001446993949_li17384171782510">Click <strong id="mrs_01_24763__en-us_topic_0000001446993949_b12926539144311">OK</strong>.</li></ol>
</p></li><li id="mrs_01_24763__li032595183911"><span>Drag the two icons to associate the job parameters and click <strong id="mrs_01_24763__b54105191150">Save</strong>. The job configuration is complete.</span><p><p id="mrs_01_24763__p51802192443"><span><img id="mrs_01_24763__image451820444390" src="en-us_image_0000001582952097.png"></span></p>
</p></li><li id="mrs_01_24763__li7968152124413"><span>In the job list on the <strong id="mrs_01_24763__b165721931954">Job Management</strong> page, locate the created job, click <strong id="mrs_01_24763__b13572183118515">Start</strong> in the <strong id="mrs_01_24763__b857215311057">Operation</strong> column, and wait until the job is started.</span><p><p id="mrs_01_24763__p29726217442">Check whether the data transmission takes effect, for example, insert data into the table in the openGauss database and view the content of the file imported to Hudi.</p>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24240.html">Common CDL Jobs</a></div>
</div>
</div>
View Git Blame Copy Permalink