forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
44 lines
13 KiB
HTML
44 lines
13 KiB
HTML
<a name="mrs_01_24063"></a><a name="mrs_01_24063"></a>
|
|
|
|
<h1 class="topictitle1">Operating a Hudi Table Using hudi-cli.sh</h1>
|
|
<div id="body32001227"><div class="section" id="mrs_01_24063__en-us_topic_0000001173789698_section15304113923815"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24063__en-us_topic_0000001173789698_ul8657019554"><li id="mrs_01_24063__en-us_topic_0000001173789698_li12659014559">You have created a user and added the user to user groups <strong id="mrs_01_24063__en-us_topic_0000001173789698_b1680018381352">hadoop</strong> and <strong id="mrs_01_24063__en-us_topic_0000001173789698_b19806338173518">hive</strong> on Manager.</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li7656015555">The Hudi client has been downloaded and installed.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_24063__en-us_topic_0000001173789698_section1879324535818"><h4 class="sectiontitle">Basic Operations</h4><ol id="mrs_01_24063__en-us_topic_0000001173789698_ol4590165662011"><li id="mrs_01_24063__en-us_topic_0000001173789698_li1959085618207">Log in to the client node as user <strong id="mrs_01_24063__en-us_topic_0000001173789698_b95211943558">root</strong> and run the following commands:<p id="mrs_01_24063__en-us_topic_0000001173789698_p17706123119196"><strong id="mrs_01_24063__en-us_topic_0000001173789698_b6136141435516">cd </strong> <em id="mrs_01_24063__en-us_topic_0000001173789698_i141427148554">Client installation directory</em></p>
|
|
<p id="mrs_01_24063__en-us_topic_0000001173789698_p577172482012"><strong id="mrs_01_24063__en-us_topic_0000001173789698_b11771924182013">source bigdata_env</strong></p>
|
|
<p id="mrs_01_24063__en-us_topic_0000001173789698_p138145283206"><strong id="mrs_01_24063__en-us_topic_0000001173789698_b121421439165618">cd</strong> <em id="mrs_01_24063__en-us_topic_0000001173789698_i31494397563">Client installation directory</em><strong id="mrs_01_24063__en-us_topic_0000001173789698_b111619446204">Hudi</strong></p>
|
|
<p id="mrs_01_24063__en-us_topic_0000001173789698_p112020268202"><strong id="mrs_01_24063__en-us_topic_0000001173789698_b17120126152013">source component_env</strong></p>
|
|
<p id="mrs_01_24063__en-us_topic_0000001173789698_p1013271391515"><strong id="mrs_01_24063__en-us_topic_0000001173789698_b114197478565">kinit</strong> <em id="mrs_01_24063__en-us_topic_0000001173789698_i2425747185618">Created user</em></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li8821195932115">Run the <strong id="mrs_01_24063__en-us_topic_0000001173789698_b76732050195717">hudi-cli.sh</strong> command to access the Hudi client.<p id="mrs_01_24063__en-us_topic_0000001173789698_p101560163229"><strong id="mrs_01_24063__en-us_topic_0000001173789698_b1693234815589">cd</strong> <em id="mrs_01_24063__en-us_topic_0000001173789698_i89389488589">Client installation directory</em><strong id="mrs_01_24063__en-us_topic_0000001173789698_b1940154816583">Hudi</strong></p>
|
|
<p id="mrs_01_24063__en-us_topic_0000001173789698_p1155013152227"><strong id="mrs_01_24063__en-us_topic_0000001173789698_b3180111311223">hudi-cli.sh</strong></p>
|
|
<p id="mrs_01_24063__en-us_topic_0000001173789698_p299618185288"><span><img id="mrs_01_24063__en-us_topic_0000001173789698_image1399611812817" src="en-us_image_0000001348739865.png"></span></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li187445132119">Run the following example commands as required. For details about all commands, visit the <a href="https://hudi.apache.org/docs/quick-start-guide/" target="_blank" rel="noopener noreferrer">Hudi official website</a>.<ul id="mrs_01_24063__en-us_topic_0000001173789698_ul584419520516"><li id="mrs_01_24063__en-us_topic_0000001173789698_li17804162854611">Viewing help information<p id="mrs_01_24063__en-us_topic_0000001173789698_p63931347134615"><a name="mrs_01_24063__en-us_topic_0000001173789698_li17804162854611"></a><a name="en-us_topic_0000001173789698_li17804162854611"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b1918815955011">help</strong> // View all Hudi CLI commands.</p>
|
|
<p id="mrs_01_24063__en-us_topic_0000001173789698_p450654464719"><strong id="mrs_01_24063__en-us_topic_0000001173789698_b34422121504">help 'command'</strong> // View the help information and parameter list of a certain command.</p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li584418518515">Connecting to a table<p id="mrs_01_24063__en-us_topic_0000001173789698_p66917361075"><a name="mrs_01_24063__en-us_topic_0000001173789698_li584418518515"></a><a name="en-us_topic_0000001173789698_li584418518515"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b20194195492411">connect --path '</strong><em id="mrs_01_24063__en-us_topic_0000001173789698_i2747454162411">/tmp/huditest/</em><em id="mrs_01_24063__en-us_topic_0000001173789698_i1316014599715">test_table</em>'</p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li9310010552">Viewing table information<p id="mrs_01_24063__en-us_topic_0000001173789698_p2090117431870"><a name="mrs_01_24063__en-us_topic_0000001173789698_li9310010552"></a><a name="en-us_topic_0000001173789698_li9310010552"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b141994471373">desc</strong></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li6859152516">Viewing compaction plans<p id="mrs_01_24063__en-us_topic_0000001173789698_p5676157284"><a name="mrs_01_24063__en-us_topic_0000001173789698_li6859152516"></a><a name="en-us_topic_0000001173789698_li6859152516"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b16975191011816">compactions show all</strong></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li195683441851">Viewing cleaning plans<p id="mrs_01_24063__en-us_topic_0000001173789698_p6387626686"><a name="mrs_01_24063__en-us_topic_0000001173789698_li195683441851"></a><a name="en-us_topic_0000001173789698_li195683441851"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b1956993113814">cleans show</strong></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li136878512511">Performing the cleaning operation<p id="mrs_01_24063__en-us_topic_0000001173789698_p139934377814"><a name="mrs_01_24063__en-us_topic_0000001173789698_li136878512511"></a><a name="en-us_topic_0000001173789698_li136878512511"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b51254433815">cleans run</strong></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li15812155515513">Viewing commit information<p id="mrs_01_24063__en-us_topic_0000001173789698_p1416584711810"><a name="mrs_01_24063__en-us_topic_0000001173789698_li15812155515513"></a><a name="en-us_topic_0000001173789698_li15812155515513"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b1316125015810">commits show</strong></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li3460191062">Viewing the partition where the commit is written to<p id="mrs_01_24063__en-us_topic_0000001173789698_p195021857987"><a name="mrs_01_24063__en-us_topic_0000001173789698_li3460191062"></a><a name="en-us_topic_0000001173789698_li3460191062"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b1053771342415">commit showpartitions --commit </strong><em id="mrs_01_24063__en-us_topic_0000001173789698_i2541171362410">20210127153356</em></p>
|
|
<div class="note" id="mrs_01_24063__en-us_topic_0000001173789698_note1343517262012"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24063__en-us_topic_0000001173789698_p154361426406"><em id="mrs_01_24063__en-us_topic_0000001173789698_i69413165720">20210127153356</em> indicates the commit timestamp.</p>
|
|
</div></div>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li143309810614">Viewing the file where the commit is written to<p id="mrs_01_24063__en-us_topic_0000001173789698_p4665758915"><a name="mrs_01_24063__en-us_topic_0000001173789698_li143309810614"></a><a name="en-us_topic_0000001173789698_li143309810614"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b179279161249">commit showfiles --commit </strong><em id="mrs_01_24063__en-us_topic_0000001173789698_i1454571742411">20210127153356</em></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li7736613765">Comparing the commit information of two tables<p id="mrs_01_24063__en-us_topic_0000001173789698_p206427131996"><a name="mrs_01_24063__en-us_topic_0000001173789698_li7736613765"></a><a name="en-us_topic_0000001173789698_li7736613765"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b14168163462419">commits compare --path </strong><em id="mrs_01_24063__en-us_topic_0000001173789698_i15707143417244">/tmp/hudimor/</em><em id="mrs_01_24063__en-us_topic_0000001173789698_i2377922202418">mytest100</em></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li157020193616">Rolling back a commit (Only the last commit can be rolled back.)<p id="mrs_01_24063__en-us_topic_0000001173789698_p17619122110915"><a name="mrs_01_24063__en-us_topic_0000001173789698_li157020193616"></a><a name="en-us_topic_0000001173789698_li157020193616"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b16851122613245">commit rollback --commit </strong><em id="mrs_01_24063__en-us_topic_0000001173789698_i2588102714244">2021012716490</em><em id="mrs_01_24063__en-us_topic_0000001173789698_i314724212415">5</em></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li62325511404">Scheduling a compaction<p id="mrs_01_24063__en-us_topic_0000001173789698_p623217514018"><a name="mrs_01_24063__en-us_topic_0000001173789698_li62325511404"></a><a name="en-us_topic_0000001173789698_li62325511404"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b169023472413">compaction schedule --</strong><em id="mrs_01_24063__en-us_topic_0000001173789698_i19174103732412">hoodieConfigs 'hoodie.compaction.strategy=org.apache.hudi.table.action.compact.strategy.BoundedIOCompactionStrategy,hoodie.compaction.target.io=1,hoodie.compact.inline.max.delta.commits=1'</em></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li10849163555016">Performing a compaction<p id="mrs_01_24063__en-us_topic_0000001173789698_p185133549598"><a name="mrs_01_24063__en-us_topic_0000001173789698_li10849163555016"></a><a name="en-us_topic_0000001173789698_li10849163555016"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b182915437246">compaction run --</strong><em id="mrs_01_24063__en-us_topic_0000001173789698_i1655024932416">parallelism 100 --sparkMemory 1g --retry 1 --compactionInstant 20210602101315 --hoodieConfigs 'hoodie.compaction.strategy=org.apache.hudi.table.action.compact.strategy.BoundedIOCompactionStrategy,hoodie.compaction.target.io=1,hoodie.compact.inline.max.delta.commits=1' --propsFilePath hdfs://hacluster/tmp/default/tb_test_mor/.hoodie/hoodie.properties --schemaFilePath /tmp/default/tb_test_mor/.hoodie/compact_tb_base.json</em></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li13858395528">Creating a savepoint<p id="mrs_01_24063__en-us_topic_0000001173789698_p117711959185415"><a name="mrs_01_24063__en-us_topic_0000001173789698_li13858395528"></a><a name="en-us_topic_0000001173789698_li13858395528"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b317525145415">savepoint create</strong> <strong id="mrs_01_24063__en-us_topic_0000001173789698_b82301547165811">--commit </strong><em id="mrs_01_24063__en-us_topic_0000001173789698_i5595134211582">2</em><em id="mrs_01_24063__en-us_topic_0000001173789698_i119163185817">0210318155750</em></p>
|
|
</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li8930151325314">Rolling back a specified savepoint<p id="mrs_01_24063__en-us_topic_0000001173789698_p1187185817545"><a name="mrs_01_24063__en-us_topic_0000001173789698_li8930151325314"></a><a name="en-us_topic_0000001173789698_li8930151325314"></a><strong id="mrs_01_24063__en-us_topic_0000001173789698_b175128544548">savepoint rollback</strong><strong id="mrs_01_24063__en-us_topic_0000001173789698_b83662051195818"> --savepoint </strong><em id="mrs_01_24063__en-us_topic_0000001173789698_i18556135665812">20210318155750</em></p>
|
|
<div class="caution" id="mrs_01_24063__en-us_topic_0000001173789698_note7192171013419"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><ol type="a" id="mrs_01_24063__en-us_topic_0000001173789698_ol27051113164118"><li id="mrs_01_24063__en-us_topic_0000001173789698_li1070515131415">If the commit operation causes metadata conflicts, you can run the <strong id="mrs_01_24063__en-us_topic_0000001173789698_b4961144152516">commit rollback</strong> and <strong id="mrs_01_24063__en-us_topic_0000001173789698_b1782010478258">savepoint rollback</strong> commands to roll back data, but the Hive metadata cannot be rolled back. In this case, you can delete the Hive table and manually synchronize data.</li><li id="mrs_01_24063__en-us_topic_0000001173789698_li270561384113">The <strong id="mrs_01_24063__en-us_topic_0000001173789698_b1421104122813">commit rollback</strong> command rolls back only the latest commit, and the <strong id="mrs_01_24063__en-us_topic_0000001173789698_b2757161814281">savepoint rollback</strong> command rolls back only the latest savepoint. You cannot specify a commit or savepoint to roll back.</li></ol>
|
|
</div></div>
|
|
</li></ul>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24100.html">Using the Hudi Client</a></div>
|
|
</div>
|
|
</div>
|
|
|