forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
43 lines
9.2 KiB
HTML
43 lines
9.2 KiB
HTML
<a name="mrs_01_24063"></a><a name="mrs_01_24063"></a>
|
|
|
|
<h1 class="topictitle1">Operating a Hudi Table Using hudi-cli.sh</h1>
|
|
<div id="body0000001145430239"><div class="section" id="mrs_01_24063__section15304113923815"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24063__ul8657019554"><li id="mrs_01_24063__li12659014559">For a cluster with Kerberos authentication enabled, a user has been created on FusionInsight Manager of the cluster and associated with user groups <strong id="mrs_01_24063__b48271158112720">hadoop</strong> and <strong id="mrs_01_24063__b37438113283">hive</strong>.</li><li id="mrs_01_24063__li7656015555">The Hudi cluster client has been downloaded and installed.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_24063__section1879324535818"><h4 class="sectiontitle">Basic Operations</h4><ol id="mrs_01_24063__ol4590165662011"><li id="mrs_01_24063__li1959085618207">Log in to the cluster client as user <strong id="mrs_01_24063__b95211943558">root</strong> and run the following commands:<p id="mrs_01_24063__p17706123119196"><strong id="mrs_01_24063__b6136141435516">cd </strong> <em id="mrs_01_24063__i141427148554">Client installation directory</em></p>
|
|
<p id="mrs_01_24063__p577172482012"><strong id="mrs_01_24063__b11771924182013">source bigdata_env</strong></p>
|
|
<p id="mrs_01_24063__p112020268202"><strong id="mrs_01_24063__b17120126152013">source Hudi/component_env</strong></p>
|
|
<p id="mrs_01_24063__p1013271391515"><strong id="mrs_01_24063__b114197478565">kinit</strong> <em id="mrs_01_24063__i2425747185618">Created user</em></p>
|
|
</li><li id="mrs_01_24063__li8821195932115">Run the <strong id="mrs_01_24063__b76732050195717">hudi-cli.sh</strong> command to access the Hudi client.<p id="mrs_01_24063__p101560163229"><strong id="mrs_01_24063__b1693234815589">cd</strong> <em id="mrs_01_24063__i89389488589">Client installation directory</em><strong id="mrs_01_24063__b1940154816583">Hudi</strong></p>
|
|
<p id="mrs_01_24063__p1155013152227"><strong id="mrs_01_24063__b3180111311223">./hudi-cli.sh</strong></p>
|
|
<p id="mrs_01_24063__p299618185288"><span><img id="mrs_01_24063__image1399611812817" src="en-us_image_0000001349090349.png"></span></p>
|
|
</li><li id="mrs_01_24063__li187445132119">Run the following example commands as required. For details about all commands, visit the <a href="https://hudi.apache.org/docs/quick-start-guide/" target="_blank" rel="noopener noreferrer">Hudi official website</a>.<ul id="mrs_01_24063__ul584419520516"><li id="mrs_01_24063__li17804162854611">Viewing help information<p id="mrs_01_24063__p63931347134615"><a name="mrs_01_24063__li17804162854611"></a><a name="li17804162854611"></a><strong id="mrs_01_24063__b1918815955011">help</strong> // View all Hudi CLI commands.</p>
|
|
<p id="mrs_01_24063__p450654464719"><strong id="mrs_01_24063__b34422121504">help 'command'</strong> // View the help information and parameter list of a certain command.</p>
|
|
</li><li id="mrs_01_24063__li584418518515">Connecting to a table<p id="mrs_01_24063__p66917361075"><a name="mrs_01_24063__li584418518515"></a><a name="li584418518515"></a><strong id="mrs_01_24063__b20194195492411">connect --path '</strong><em id="mrs_01_24063__i2747454162411">/tmp/huditest/</em><em id="mrs_01_24063__i1316014599715">test_table</em>'</p>
|
|
</li><li id="mrs_01_24063__li9310010552">Viewing table information<p id="mrs_01_24063__p2090117431870"><a name="mrs_01_24063__li9310010552"></a><a name="li9310010552"></a><strong id="mrs_01_24063__b141994471373">desc</strong></p>
|
|
</li><li id="mrs_01_24063__li6859152516">Viewing compaction plans<p id="mrs_01_24063__p5676157284"><a name="mrs_01_24063__li6859152516"></a><a name="li6859152516"></a><strong id="mrs_01_24063__b16975191011816">compactions show all</strong></p>
|
|
</li><li id="mrs_01_24063__li195683441851">Viewing cleaning plans<p id="mrs_01_24063__p6387626686"><a name="mrs_01_24063__li195683441851"></a><a name="li195683441851"></a><strong id="mrs_01_24063__b1956993113814">cleans show</strong></p>
|
|
</li><li id="mrs_01_24063__li136878512511">Performing the cleaning operation<p id="mrs_01_24063__p139934377814"><a name="mrs_01_24063__li136878512511"></a><a name="li136878512511"></a><strong id="mrs_01_24063__b51254433815">cleans run</strong></p>
|
|
</li><li id="mrs_01_24063__li15812155515513">Viewing commit information<p id="mrs_01_24063__p1416584711810"><a name="mrs_01_24063__li15812155515513"></a><a name="li15812155515513"></a><strong id="mrs_01_24063__b1316125015810">commits show</strong></p>
|
|
</li><li id="mrs_01_24063__li3460191062">Viewing the partition where the commit is written to<p id="mrs_01_24063__p195021857987"><a name="mrs_01_24063__li3460191062"></a><a name="li3460191062"></a><strong id="mrs_01_24063__b1053771342415">commit showpartitions --commit </strong><em id="mrs_01_24063__i2541171362410">20210127153356</em></p>
|
|
<div class="note" id="mrs_01_24063__note1343517262012"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24063__p154361426406"><em id="mrs_01_24063__i69413165720">20210127153356</em> indicates the commit timestamp.</p>
|
|
</div></div>
|
|
</li><li id="mrs_01_24063__li143309810614">Viewing the file where the commit is written to<p id="mrs_01_24063__p4665758915"><a name="mrs_01_24063__li143309810614"></a><a name="li143309810614"></a><strong id="mrs_01_24063__b179279161249">commit showfiles --commit </strong><em id="mrs_01_24063__i1454571742411">20210127153356</em></p>
|
|
</li><li id="mrs_01_24063__li7736613765">Comparing the commit information of two tables<p id="mrs_01_24063__p206427131996"><a name="mrs_01_24063__li7736613765"></a><a name="li7736613765"></a><strong id="mrs_01_24063__b14168163462419">commits compare --path </strong><em id="mrs_01_24063__i15707143417244">/tmp/hudimor/</em><em id="mrs_01_24063__i2377922202418">mytest100</em></p>
|
|
</li><li id="mrs_01_24063__li157020193616">Rolling back a commit (Only the last commit can be rolled back.)<p id="mrs_01_24063__p17619122110915"><a name="mrs_01_24063__li157020193616"></a><a name="li157020193616"></a><strong id="mrs_01_24063__b16851122613245">commit rollback --commit </strong><em id="mrs_01_24063__i2588102714244">2021012716490</em><em id="mrs_01_24063__i314724212415">5</em></p>
|
|
</li><li id="mrs_01_24063__li62325511404">Scheduling a compaction<p id="mrs_01_24063__p623217514018"><a name="mrs_01_24063__li62325511404"></a><a name="li62325511404"></a><strong id="mrs_01_24063__b169023472413">compaction schedule --</strong><em id="mrs_01_24063__i19174103732412">hoodieConfigs 'hoodie.compaction.strategy=org.apache.hudi.table.action.compact.strategy.BoundedIOCompactionStrategy,hoodie.compaction.target.io=1,hoodie.compact.inline.max.delta.commits=1'</em></p>
|
|
</li><li id="mrs_01_24063__li10849163555016">Performing a compaction<p id="mrs_01_24063__p185133549598"><a name="mrs_01_24063__li10849163555016"></a><a name="li10849163555016"></a><strong id="mrs_01_24063__b182915437246">compaction run --</strong><em id="mrs_01_24063__i1655024932416">parallelism 100 --sparkMemory 1g --retry 1 --compactionInstant 20210602101315 --hoodieConfigs 'hoodie.compaction.strategy=org.apache.hudi.table.action.compact.strategy.BoundedIOCompactionStrategy,hoodie.compaction.target.io=1,hoodie.compact.inline.max.delta.commits=1' --propsFilePath hdfs://hacluster/tmp/default/tb_test_mor/.hoodie/hoodie.properties --schemaFilePath /tmp/default/tb_test_mor/.hoodie/compact_tb_base.json</em></p>
|
|
</li><li id="mrs_01_24063__li13858395528">Creating a savepoint<p id="mrs_01_24063__p117711959185415"><a name="mrs_01_24063__li13858395528"></a><a name="li13858395528"></a><strong id="mrs_01_24063__b317525145415">savepoint create</strong> <strong id="mrs_01_24063__b82301547165811">--commit </strong><em id="mrs_01_24063__i5595134211582">2</em><em id="mrs_01_24063__i119163185817">0210318155750</em></p>
|
|
</li><li id="mrs_01_24063__li8930151325314">Rolling back a specified savepoint<p id="mrs_01_24063__p1187185817545"><a name="mrs_01_24063__li8930151325314"></a><a name="li8930151325314"></a><strong id="mrs_01_24063__b175128544548">savepoint rollback</strong><strong id="mrs_01_24063__b83662051195818"> --savepoint </strong><em id="mrs_01_24063__i18556135665812">20210318155750</em></p>
|
|
<div class="caution" id="mrs_01_24063__note7192171013419"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><ol type="a" id="mrs_01_24063__ol27051113164118"><li id="mrs_01_24063__li1070515131415">If the commit operation causes metadata conflicts, you can run the <strong id="mrs_01_24063__b4961144152516">commit rollback</strong> and <strong id="mrs_01_24063__b1782010478258">savepoint rollback</strong> commands to roll back data, but the Hive metadata cannot be rolled back. In this case, you can delete the Hive table and manually synchronize data.</li><li id="mrs_01_24063__li270561384113">The <strong id="mrs_01_24063__b1421104122813">commit rollback</strong> command rolls back only the latest commit, and the <strong id="mrs_01_24063__b2757161814281">savepoint rollback</strong> command rolls back only the latest savepoint. You cannot specify a commit or savepoint to roll back.</li></ol>
|
|
</div></div>
|
|
</li></ul>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24100.html">Using the Hudi Client</a></div>
|
|
</div>
|
|
</div>
|
|
|