forked from docs/doc-exports
Yang, Tong
48706b7552
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
195 lines
18 KiB
HTML
195 lines
18 KiB
HTML
<a name="mrs_01_24232"></a><a name="mrs_01_24232"></a>
|
|
|
|
<h1 class="topictitle1">Using CDL from Scratch</h1>
|
|
<div id="body0000001532471608"><p id="mrs_01_24232__p2071919498443">CDL supports data synchronization or comparison tasks in multiple scenarios. This section describes how to import data from PgSQL to Kafka on the CDLService WebUI of a cluster with Kerberos authentication enabled.</p>
|
|
<div class="section" id="mrs_01_24232__section5590131211512"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24232__ul2041871362311"><li id="mrs_01_24232__li12418213122313">The CDL and Kafka services have been installed in a cluster and are running properly.</li><li id="mrs_01_24232__li362910581468">Write-ahead logging is enabled for the PostgreSQL database. For details, see <a href="mrs_01_24124.html#mrs_01_24124__li1868193914169">Policy for Modifying Write-Ahead Logs in PostgreSQL Databases</a>.</li><li id="mrs_01_24232__li1441819135232">You have created a human-machine user, for example, <strong id="mrs_01_24232__b28578215432">cdluser</strong>, added the user to user groups <strong id="mrs_01_24232__b787218244310">cdladmin</strong> (primary group), <strong id="mrs_01_24232__b888019234318">hadoop</strong>, and <strong id="mrs_01_24232__b088942104316">kafka</strong>, and associated the user with the <strong id="mrs_01_24232__b38961920432">System_administrator</strong> role on FusionInsight Manager.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_24232__section01876484714"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24232__ol18871133225317"><li id="mrs_01_24232__li1365211683614"><span>Log in to FusionInsight Manager as user <strong id="mrs_01_24232__b14163101144313">cdluser</strong> (change the password upon the first login) and choose <strong id="mrs_01_24232__b317716117437">Cluster</strong> > <strong id="mrs_01_24232__b618431117439">Services</strong> > <strong id="mrs_01_24232__b131921611144310">CDL</strong>. On the <strong id="mrs_01_24232__b13200511144312">Dashboard</strong> page, click the hyperlink next to <strong id="mrs_01_24232__b1520811112439">CDLService UI</strong> to go to the native CDL page.</span></li><li id="mrs_01_24232__li3618121145616"><span>Choose <strong id="mrs_01_24232__b1676271651710">Link Management</strong> and click <strong id="mrs_01_24232__b77681216181719">Add Link</strong>. On the displayed dialog box, set parameters for adding the <strong id="mrs_01_24232__b1769116101713">pgsql</strong> and <strong id="mrs_01_24232__b137691316171710">kafka</strong> links by referring to the following tables.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24232__table4349111692811" frame="border" border="1" rules="all"><caption><b>Table 1 </b>PgSQL data link parameters</caption><thead align="left"><tr id="mrs_01_24232__row334941612285"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.1.2.3.1.1"><p id="mrs_01_24232__p634921615285">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.1.2.3.1.2"><p id="mrs_01_24232__p534911162289">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24232__row11349716112813"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24232__p14349171682812">Link Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24232__p934941662810">pgsql</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row1134915163284"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24232__p83499164282">Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24232__p13350181611281">pgsqllink</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row83501316202810"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24232__p16350101622817">Host</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24232__p71531626165116">10.10.10.10</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row735051613281"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24232__p1350151615281">Port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24232__p91531026175115">5432</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row235071682815"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24232__p17350101692817">DB Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24232__p2035051620289">testDB</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row93501716132816"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24232__p83502168282">User</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24232__p892121124220">user</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row11350131602818"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24232__p153505167283">Password</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24232__p3922191194210"><em id="mrs_01_24232__i911813539369">Password of the user user</em></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row135121613286"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="mrs_01_24232__p203511916152810">Description</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="mrs_01_24232__p73511165283">-</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24232__table131852223913" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Kafka data link parameters</caption><thead align="left"><tr id="mrs_01_24232__row718517214396"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.2.2.3.1.1"><p id="mrs_01_24232__p91857220390">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.2.2.2.2.3.1.2"><p id="mrs_01_24232__p131851028395">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24232__row1818572103920"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24232__p13185132163910">Link Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24232__p151863273915">kafka</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row111862218391"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24232__p141861283916">Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24232__p1618642153919">kafkalink</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row1718620215395"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.1 "><p id="mrs_01_24232__p4186182123910">Description</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.2.2.2.2.3.1.2 "><p id="mrs_01_24232__p131870223912">-</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_24232__li912416105513"><span>After the parameters are configured, click <strong id="mrs_01_24232__b815219453259">Test</strong> to check whether the data link is normal.</span><p><p id="mrs_01_24232__p172041852499">After the test is successful, click <strong id="mrs_01_24232__b1918784772520">OK</strong>.</p>
|
|
</p></li><li id="mrs_01_24232__li8419191320242"><a name="mrs_01_24232__li8419191320242"></a><a name="li8419191320242"></a><span>On the <strong id="mrs_01_24232__b171293495255">Job Management</strong> page, click <strong id="mrs_01_24232__b13129184916255">Add Job</strong>. In the displayed dialog box, configure the parameters and click <strong id="mrs_01_24232__b184411803264">Next</strong>.</span><p><div class="p" id="mrs_01_24232__p1442915568513">Specifically:
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24232__table1379713112014" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_24232__row13797011905"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.1.1.1.3.1.1"><p id="mrs_01_24232__p17797411309">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.1.1.1.3.1.2"><p id="mrs_01_24232__p137971011309">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24232__row1679841118018"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.1.1.3.1.1 "><p id="mrs_01_24232__p391613171005">Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.1.1.3.1.2 "><p id="mrs_01_24232__p149141171404">job_pgsqltokafka</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row779916111602"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.1.1.3.1.1 "><p id="mrs_01_24232__p83069155152">Desc</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.1.1.1.3.1.2 "><p id="mrs_01_24232__p14306191511512">xxx</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</p></li><li id="mrs_01_24232__li1072816508257"><span>Configure PgSQL job parameters.</span><p><ol type="a" id="mrs_01_24232__ol2821057152517"><li id="mrs_01_24232__li288593182611">On the <strong id="mrs_01_24232__b599186192711">Job Management</strong> page, drag the <strong id="mrs_01_24232__b1210686102713">pgsql</strong> icon on the left to the editing area on the right and double-click the icon to go to the PgSQL job configuration page.
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24232__table1814971517167" frame="border" border="1" rules="all"><caption><b>Table 3 </b>PgSQL job parameters</caption><thead align="left"><tr id="mrs_01_24232__row1415041510166"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.1.3.2.3.1.1"><p id="mrs_01_24232__p4150915161614">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.5.2.1.1.3.2.3.1.2"><p id="mrs_01_24232__p215051514168">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24232__row3150915191617"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p2015091561617">Link</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p171507153167">pgsqllink</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row5150151516169"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p915061521613">Tasks Max</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p815017159166">1</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row15150815201614"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p23892122226">Mode</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p112792033162019">insert, update, and delete</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row1015017153166"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p15317111191811">Schema</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p16312101131814">public</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row17151191512169"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p123284921814">dbName Alias</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p10323109111811">cdc</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row17151315131617"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p13169919184">Slot Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p1231269121813">a4545sad</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row8798291295"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p579729892">Slot Drop</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p07914292097">No</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row19637192192819"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p5575915181716">Connect With Hudi</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p195752015171713">No</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row109981571692"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p499819573914">Use Exist Publication</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p19981575919">Yes</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24232__row13532122114101"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.1 "><p id="mrs_01_24232__p9533421121019">Publication Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.5.2.1.1.3.2.3.1.2 "><p id="mrs_01_24232__p175336213109">test</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_24232__li44410384820">Click the plus sign (+) to display more parameters.<p id="mrs_01_24232__p04061753155413"><a name="mrs_01_24232__li44410384820"></a><a name="li44410384820"></a><span><img id="mrs_01_24232__image16406195355414" src="en-us_image_0000001532951860.png"></span></p>
|
|
<div class="note" id="mrs_01_24232__note195817112269"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24232__ul17589192611"><li id="mrs_01_24232__li115815118268"><strong id="mrs_01_24232__b67281120972">WhiteList</strong>: Enter the name of the table in the database, for example, <strong id="mrs_01_24232__b157282201273">myclass</strong>.</li><li id="mrs_01_24232__li558715267"><strong id="mrs_01_24232__b1061584016711">Topic Table Mapping</strong>: In the first text box, enter a topic name (the value must be different from that of <strong id="mrs_01_24232__b982423220205">Name </strong>in <a href="#mrs_01_24232__li8419191320242">4</a>), for example, <strong id="mrs_01_24232__b82540261374">myclass_topic</strong>. In the second text box, enter a table name, for example, <strong id="mrs_01_24232__b58112465716">myclass</strong>. The value must be in one-to-one relationship with the topic name entered in the first text box.)</li></ul>
|
|
</div></div>
|
|
</li><li id="mrs_01_24232__li1539154131113">Click <strong id="mrs_01_24232__b14012073520">OK</strong>. The PgSQL job parameters are configured.</li></ol>
|
|
</p></li><li id="mrs_01_24232__li153641326297"><span>Configure Kafka job parameters.</span><p><ol type="a" id="mrs_01_24232__ol43821019691"><li id="mrs_01_24232__li1838281915919">On the <strong id="mrs_01_24232__b191231726103516">Job Management</strong> page, drag the <strong id="mrs_01_24232__b1312516265351">kafka</strong> icon on the left to the editing area on the right and double-click the icon to go to the Kafka job configuration page. Configure parameters based on <a href="#mrs_01_24232__table8128935153416">Table 4</a>.
|
|
<div class="tablenoborder"><a name="mrs_01_24232__table8128935153416"></a><a name="table8128935153416"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24232__table8128935153416" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Kafka job parameter</caption><thead align="left"><tr id="mrs_01_24232__row5128173510348"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.4.2.3.1.1"><p id="mrs_01_24232__p912817350347">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.6.2.1.1.4.2.3.1.2"><p id="mrs_01_24232__p11128635103416">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24232__row1012803517346"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.4.2.3.1.1 "><p id="mrs_01_24232__p31281035143413">Link</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.6.2.1.1.4.2.3.1.2 "><p id="mrs_01_24232__p712933513348">kafkalink</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_24232__li1638310191491">Click <strong id="mrs_01_24232__b151658180205222">OK</strong>.</li></ol>
|
|
</p></li><li id="mrs_01_24232__li111998510582"><span>After the job parameters are configured, drag the two icons to associate the job parameters and click <strong id="mrs_01_24232__b4443756183514">Save</strong>. The job configuration is complete.</span><p><p id="mrs_01_24232__p62045284122"><span><img id="mrs_01_24232__image61454503137" src="en-us_image_0000001583151841.png"></span></p>
|
|
</p></li><li id="mrs_01_24232__li1997814344117"><span>In the job list on the <strong id="mrs_01_24232__b85654513611">Job Management</strong> page, locate the created jobs, click <strong id="mrs_01_24232__b357445173618">Start</strong> in the <strong id="mrs_01_24232__b357714511363">Operation</strong> column, and wait until the jobs are started.</span><p><p id="mrs_01_24232__p7927192012121">Check whether the data transmission takes effect. For example, insert data into the table in the PgSQL database, go to the Kafka UI to check whether data is generated in the Kafka topic by referring to <a href="mrs_01_24138.html">Managing Topics on Kafka UI</a>.</p>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24123.html">Using CDL</a></div>
|
|
</div>
|
|
</div>
|
|
|