forked from docs/doc-exports
Yang, Tong
48706b7552
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
121 lines
14 KiB
HTML
121 lines
14 KiB
HTML
<a name="mrs_01_24775"></a><a name="mrs_01_24775"></a>
|
|
|
|
<h1 class="topictitle1">Creating a CDL Data Comparison Job</h1>
|
|
<div id="body0000001583390737"><div class="section" id="mrs_01_24775__section7722164042318"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24775__p890415519911">Data comparison checks the consistency between data in the source database and that in the target Hive. If the data is inconsistent, CDL can attempt to repair the inconsistent data.</p>
|
|
<p id="mrs_01_24775__p0904251194">The current data comparison task supports manual full comparison. The data comparison task runs in On Yarn mode, and the comparison result is uploaded to HDFS directories.</p>
|
|
<div class="note" id="mrs_01_24775__note1144327165812"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24775__ul852023755813"><li id="mrs_01_24775__li1852023755810">Currently, only basic data types can be compared. Special data types such as date, timestamp, decimal, numeric, and JSON cannot be compared.</li><li id="mrs_01_24775__li77892117597">Data cannot be compared for tables whose field names contain database keywords.</li><li id="mrs_01_24775__li1396418185911">A data comparison task for a single table supports comparison of a maximum of 100 fields. If a table contains more than 100 fields, you can specify two whitelists of different comparison fields for data comparison.</li><li id="mrs_01_24775__li51491023161619">Currently, only the data captured from PgSQL to Hudi can be compared. If the comparison result is inconsistent, a report address is generated only when there are no more than 2000 inconsistent data records. If there are more than 2000 inconsistent data records, no report address is generated and data cannot be repaired.</li><li id="mrs_01_24775__li1884955193215">If the Kafka lag of the CDL task involved in the comparison is not 0, the comparison result is inconsistent.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_24775__section1438820494231"><h4 class="sectiontitle">Prerequisites</h4><ol id="mrs_01_24775__ol66281122183"><li id="mrs_01_24775__li20629152210815">You have prepared the Hive UDF JAR package, copied <strong id="mrs_01_24775__b187431043191012">${BIGDATA_HOME}/FusionInsight_CDL_*/install/FusionInsight-CDL-*/cdl/hive-checksum/cdl-dc-hive-checksum-*.jar</strong> from the CDL installation directory to the <strong id="mrs_01_24775__b930885461011">${BIGDATA_HOME}/third_lib/Hive</strong> directory of Hive, and set the permission on the JAR package to <strong id="mrs_01_24775__b165055262114">750</strong> or higher.</li><li id="mrs_01_24775__li1477593912417">A user with the CDL management permission has been created for the cluster with Kerberos authentication enabled.. If Ranger authentication is enabled for the current cluster, grant the Hive administrator permission and UDF operation permission to the user by referring to <a href="mrs_01_1858.html">Adding a Ranger Access Permission Policy for Hive</a>.</li><li id="mrs_01_24775__li1570105994418">You have created a global UDF algorithm on the Hive client as a user with the Hive administrator permission.<p id="mrs_01_24775__p1511332518"><a name="mrs_01_24775__li1570105994418"></a><a name="li1570105994418"></a>Run the following command to create the <strong id="mrs_01_24775__b15691415111314">CheckSum</strong> function in the default database:</p>
|
|
<p id="mrs_01_24775__p178406415129"><strong id="mrs_01_24775__b17813135714453">create function checksum_aggregate as 'com.<span id="mrs_01_24775__ph13183181862214">xxx</span>.hive.checksum.ChecksumUdaf'</strong></p>
|
|
</li><li id="mrs_01_24775__li197981626135212">A CDL synchronization task exists. The comparison task determines the data to be compared based on the synchronization task status and data synchronization status.</li><li id="mrs_01_24775__li1876160201716">The database user in the data synchronization task associated with data comparison task must have the <strong id="mrs_01_24775__b1639519581474">create function</strong> permission on the current schema.</li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_24775__section9635165762318"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24775__ol62211517131719"><li id="mrs_01_24775__li2138162063820"><span>Log in to the CDLService web UI as the created user user (for the cluster where Kerberos authentication is not enabled). For details, see <a href="mrs_01_24236.html">Logging In to the CDLService WebUI</a>.</span></li><li id="mrs_01_24775__li1713802043818"><span>Choose <strong id="mrs_01_24775__b170759181413">Job Management</strong> > <strong id="mrs_01_24775__b1470710912143">Data comparison task</strong> and click <strong id="mrs_01_24775__b1770711913144">Add Job</strong>. In the displayed dialog box, set related job parameters and click <strong id="mrs_01_24775__b07084910148">Next</strong>.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24775__table1379713112014" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_24775__row13797011905"><th align="left" class="cellrowborder" valign="top" width="24.82%" id="mcps1.3.3.2.2.2.1.1.4.1.1"><p id="mrs_01_24775__p17797411309">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="28.939999999999998%" id="mcps1.3.3.2.2.2.1.1.4.1.2"><p id="mrs_01_24775__p350373011818">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="46.239999999999995%" id="mcps1.3.3.2.2.2.1.1.4.1.3"><p id="mrs_01_24775__p137971011309">Example</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24775__row1679841118018"><td class="cellrowborder" valign="top" width="24.82%" headers="mcps1.3.3.2.2.2.1.1.4.1.1 "><p id="mrs_01_24775__p391613171005">Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.939999999999998%" headers="mcps1.3.3.2.2.2.1.1.4.1.2 "><p id="mrs_01_24775__p050423017813">Name of the data comparison task.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%" headers="mcps1.3.3.2.2.2.1.1.4.1.3 "><p id="mrs_01_24775__p149141171404">job_dc_test</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24775__row41951220101816"><td class="cellrowborder" valign="top" width="24.82%" headers="mcps1.3.3.2.2.2.1.1.4.1.1 "><p id="mrs_01_24775__p819514204186">CDL Job Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.939999999999998%" headers="mcps1.3.3.2.2.2.1.1.4.1.2 "><p id="mrs_01_24775__p1150415301781">Name of the associated synchronization task. (Note: The user who runs the comparison task is the user of the Hudi Link in the associated synchronization task.)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%" headers="mcps1.3.3.2.2.2.1.1.4.1.3 "><p id="mrs_01_24775__p41951520181813">pg2hudi_test</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24775__row3485325142018"><td class="cellrowborder" valign="top" width="24.82%" headers="mcps1.3.3.2.2.2.1.1.4.1.1 "><p id="mrs_01_24775__p154866250203">Execution Env</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.939999999999998%" headers="mcps1.3.3.2.2.2.1.1.4.1.2 "><p id="mrs_01_24775__p95048301788">Environment variable required for running Spark tasks. If no ENV is available, create one by referring to <a href="mrs_01_24255.html">Managing ENV</a>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%" headers="mcps1.3.3.2.2.2.1.1.4.1.3 "><p id="mrs_01_24775__p1948612513205">dc_env</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24775__row779916111602"><td class="cellrowborder" valign="top" width="24.82%" headers="mcps1.3.3.2.2.2.1.1.4.1.1 "><p id="mrs_01_24775__p83069155152">Desc</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.939999999999998%" headers="mcps1.3.3.2.2.2.1.1.4.1.2 "><p id="mrs_01_24775__p750418301818">Description of the task.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%" headers="mcps1.3.3.2.2.2.1.1.4.1.3 "><p id="mrs_01_24775__p14306191511512">-</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_24775__li17516164719316"><span>On the <strong id="mrs_01_24775__b0491154513179">Create Compare-Pair</strong> page, set related parameters and click <strong id="mrs_01_24775__b134961310141816">Create</strong>.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24775__table99465135234" frame="border" border="1" rules="all"><tbody><tr id="mrs_01_24775__row8591417141119"><td class="cellrowborder" valign="top" width="31.03%"><p id="mrs_01_24775__p15913170112">Parameter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.73%"><p id="mrs_01_24775__p5595176113">Description</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%"><p id="mrs_01_24775__p6591917171114">Example</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24775__row8989213122310"><td class="cellrowborder" valign="top" width="31.03%"><p id="mrs_01_24775__p1898914134230">Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.73%"><p id="mrs_01_24775__p1347012584100">Name of the current comparison task.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%"><p id="mrs_01_24775__p17989201312319">test</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24775__row798941316238"><td class="cellrowborder" valign="top" width="31.03%"><p id="mrs_01_24775__p3989171313235">Source Table</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.73%"><p id="mrs_01_24775__p1447015586107">Source table name.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%"><p id="mrs_01_24775__p798951392310">tabletest</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24775__row1698921362314"><td class="cellrowborder" valign="top" width="31.03%"><p id="mrs_01_24775__p398931320234">Target Table</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.73%"><p id="mrs_01_24775__p7470125818104">Target table name.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%"><p id="mrs_01_24775__p2989713102314">tabletest</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24775__row393972664711"><td class="cellrowborder" valign="top" width="31.03%"><p id="mrs_01_24775__p15940192634710">WhiteList Columns</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.73%"><p id="mrs_01_24775__p5940926194715">Column family involved in data comparison.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%"><p id="mrs_01_24775__p1694072610475">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24775__row1517164715478"><td class="cellrowborder" valign="top" width="31.03%"><p id="mrs_01_24775__p1917144716478">BlackList Columns</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.73%"><p id="mrs_01_24775__p1817114774719">Column family not involved in data comparison.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%"><p id="mrs_01_24775__p4171147134719">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24775__row1452317533471"><td class="cellrowborder" valign="top" width="31.03%"><p id="mrs_01_24775__p1852315531478">Where Condition</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.73%"><p id="mrs_01_24775__p6523153184716">User-defined comparison conditions.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="46.239999999999995%"><p id="mrs_01_24775__p16523253174710">-</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_24775__p17305145121613">To compare multiple tables, click <strong id="mrs_01_24775__b125975592010">Add</strong>.</p>
|
|
</p></li><li id="mrs_01_24775__li315131261316"><span>In the data comparison task list, click <strong id="mrs_01_24775__b710211202715">Start</strong> in the row of the task to start data comparison.</span></li><li id="mrs_01_24775__li340861241314"><span>After the execution is complete, view the result in the <strong id="mrs_01_24775__b1012192912282">Comparing Result</strong> column.</span><p><p id="mrs_01_24775__p7419745416"><span><img id="mrs_01_24775__image16127851131418" src="en-us_image_0000001583391853.png"></span></p>
|
|
</p></li><li id="mrs_01_24775__li5527175421410"><span>If the result is <strong id="mrs_01_24775__b859217421296">Inconsistent</strong>, click <strong id="mrs_01_24775__b9165160193010">More</strong> and select <strong id="mrs_01_24775__b1859181514306">view records</strong>.</span><p><p id="mrs_01_24775__p79123810571"><span><img id="mrs_01_24775__image13488391586" src="en-us_image_0000001583272157.png"></span></p>
|
|
</p></li><li id="mrs_01_24775__li1333313651716"><span>In the <strong id="mrs_01_24775__b11537184412304">Task Run Log</strong> window, locate the target task and click <strong id="mrs_01_24775__b18758105613016">View Results</strong> in the <strong id="mrs_01_24775__b142491003319">Operation</strong> column.</span><p><p id="mrs_01_24775__p113731110571"><span><img id="mrs_01_24775__image7834623125710" src="en-us_image_0000001582952089.png"></span></p>
|
|
</p></li><li id="mrs_01_24775__li1910518193190"><span>Click <strong id="mrs_01_24775__b1274810110377">Repair</strong> to repair the data.</span><p><p id="mrs_01_24775__p6658123125810"><span><img id="mrs_01_24775__image1713184915815" src="en-us_image_0000001532632196.png"></span></p>
|
|
</p></li><li id="mrs_01_24775__li0650551142012"><span>After the repair is complete, check whether the value of <strong id="mrs_01_24775__b121690506348">Comparing Result</strong> is <strong id="mrs_01_24775__b485295315345">Consistent</strong>. If yes, data repair is successful. If not, the repair fails. In this case, obtain the report from the corresponding HDFS directory based on the value of <strong id="mrs_01_24775__b848162693611">Report Path</strong> and manually repair the data.</span><p><p id="mrs_01_24775__p1878161016415"><span><img id="mrs_01_24775__image142692271645" src="en-us_image_0000001532472724.png"></span></p>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24774.html">Creating a CDL Job</a></div>
|
|
</div>
|
|
</div>
|
|
|