forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
41 lines
8.1 KiB
HTML
41 lines
8.1 KiB
HTML
<a name="mrs_01_24088"></a><a name="mrs_01_24088"></a>
|
|
|
|
<h1 class="topictitle1">Clustering</h1>
|
|
<div id="body0000001104648222"><div class="section" id="mrs_01_24088__section1690119912318"><h4 class="sectiontitle">Introduction</h4><p id="mrs_01_24088__p4107023137">Clustering reorganizes data layout to improve query performance without affecting the ingestion speed.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24088__section112535238316"><h4 class="sectiontitle">Architecture</h4><p id="mrs_01_24088__p8060118">Hudi provides different operations, such as <strong id="mrs_01_24088__b13573122416428">insert</strong>, <strong id="mrs_01_24088__b13436426164217">upsert</strong>, and <strong id="mrs_01_24088__b281843024219">bulk_insert</strong>, through its write client API to write data to a Hudi table. To weight between file size and speed of importing data into the data lake, Hudi provides <strong id="mrs_01_24088__b2026442844515">hoodie.parquet.small.file.limit</strong> to configure the minimum file size. You can set it to <strong id="mrs_01_24088__b12917131414110">0</strong> to force new data to be written to new file groups, or to a higher value to ensure that new data is "padded" to existing small file groups until it reaches the specified size, but this increases ingestion latency.</p>
|
|
<p id="mrs_01_24088__p185849514293">To support fast ingestion without affecting query performance, the clustering service is introduced to rewrite data to optimize the layout of Hudi data lake files.</p>
|
|
<p id="mrs_01_24088__p165841059293">The clustering service can run asynchronously or synchronously. It adds a new operation type called <strong id="mrs_01_24088__b127695436515">REPLACE</strong>, which will mark the clustering operation in the Hudi metadata timeline.</p>
|
|
<p id="mrs_01_24088__p167027202054">Clustering service is based on the MVCC design of Hudi to allow new data to be inserted. Clustering operations run in the background to reformat data layout, ensuring snapshot isolation between concurrent readers and writers.</p>
|
|
<p id="mrs_01_24088__p021911349711"><span><img id="mrs_01_24088__image167225524618" src="en-us_image_0000001296090428.png"></span></p>
|
|
<p id="mrs_01_24088__p95841258293">Clustering is divided into two parts:</p>
|
|
<ul id="mrs_01_24088__ul6714413142913"><li id="mrs_01_24088__li0714613112913">Scheduling clustering: Create a clustering plan using a pluggable clustering strategy.<ol id="mrs_01_24088__ol87251635143211"><li id="mrs_01_24088__li5725133520321">Identify files that are eligible for clustering: Depending on the selected clustering strategy, the scheduling logic will identify the files eligible for clustering.</li><li id="mrs_01_24088__li1072553512322">Group files that are eligible for clustering based on specific criteria. The data size of each group must be a multiple of <strong id="mrs_01_24088__b113287191737">targetFileSize</strong>. Grouping is a part of the strategy defined in the plan. Additionally, there is an option to control group size to improve parallelism and avoid shuffling large volumes of data.</li><li id="mrs_01_24088__li2725735103217">Save the clustering plan to the timeline in Avro metadata format.</li></ol>
|
|
</li><li id="mrs_01_24088__li1121371682911">Execute clustering: Process the plan using an execution strategy to create new files and replace old files.<ol id="mrs_01_24088__ol165755487351"><li id="mrs_01_24088__li1257504873512">Read the clustering plan and get <strong id="mrs_01_24088__b1490010511278">clusteringGroups</strong> that marks the file groups to be clustered.</li><li id="mrs_01_24088__li8575164823517">Instantiate appropriate strategy class for each group using <strong id="mrs_01_24088__b19594744798">strategyParams</strong> (for example, <strong id="mrs_01_24088__b6267171021020">sortColumns</strong>) and apply the strategy to rewrite data.</li><li id="mrs_01_24088__li1575104819358">Create a <strong id="mrs_01_24088__b19924748111018">REPLACE</strong> commit and update the metadata in HoodieReplaceCommitMetadata.</li></ol>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_24088__section1143818436375"><h4 class="sectiontitle">How to Execute Clustering</h4><ol id="mrs_01_24088__ol148176202381"><li id="mrs_01_24088__li78171020163810">Executing clustering synchronously<p id="mrs_01_24088__p756679113717"><a name="mrs_01_24088__li78171020163810"></a><a name="li78171020163810"></a>Add the following configuration parameters when the data write operation is performed:</p>
|
|
<p id="mrs_01_24088__p205668933711"><strong id="mrs_01_24088__b1524661810405">option("hoodie.clustering.inline", "true").</strong></p>
|
|
<p id="mrs_01_24088__p1856619143715"><strong id="mrs_01_24088__b1724891854017">option("hoodie.clustering.inline.max.commits", "4").</strong></p>
|
|
<p id="mrs_01_24088__p18566893374"><strong id="mrs_01_24088__b9249618174018">option("hoodie.clustering.plan.strategy.target.file.max.bytes", "1073741824").</strong></p>
|
|
<p id="mrs_01_24088__p9566796370"><strong id="mrs_01_24088__b125091854012">option("hoodie.clustering.plan.strategy.small.file.limit", "629145600").</strong></p>
|
|
<p id="mrs_01_24088__p19566798379"><strong id="mrs_01_24088__b1525171819407">option("hoodie.clustering.plan.strategy.sort.columns", "column1,column2").</strong></p>
|
|
</li><li id="mrs_01_24088__li1277945983816">Executing clustering asynchronously<p id="mrs_01_24088__p12566169133712"><a name="mrs_01_24088__li1277945983816"></a><a name="li1277945983816"></a><strong id="mrs_01_24088__b18605122234018">spark-submit --master yarn --class org.apache.hudi.utilities.HoodieClusteringJob /opt/client/Hudi/hudi/lib/hudi-utilities*.jar --schedule --base-path <table_path> --table-name <table_name> --props /tmp/clusteringjob.properties --spark-memory 1g</strong></p>
|
|
<p id="mrs_01_24088__p1566209173712"><strong id="mrs_01_24088__b11606112254012">spark-submit --master yarn --driver-memory 16G --executor-memory 12G --executor-cores 4 --num-executors 4 --class org.apache.hudi.utilities.HoodieClusteringJob /opt/client/Hudi/hudi/lib/hudi-utilities*.jar --base-path <table_path> --instant-time 20210605112954 --table-name <table_name> --props /tmp/clusteringjob.properties --spark-memory 12g</strong></p>
|
|
<div class="note" id="mrs_01_24088__note16992611183915"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24088__p91501658183916"><strong id="mrs_01_24088__b17845102971519">clusteringjob.properties</strong> contains custom clustering configurations.</p>
|
|
<p id="mrs_01_24088__p21089546390">Example:</p>
|
|
<p id="mrs_01_24088__p966931613396">hoodie.clustering.plan.strategy.target.file.max.bytes=1073741824</p>
|
|
<p id="mrs_01_24088__p3669116133911">hoodie.clustering.inline.max.commits=4</p>
|
|
</div></div>
|
|
</li></ol>
|
|
<p id="mrs_01_24088__p145671991378">For details, see <a href="mrs_01_24032.html">Configuration Reference</a>.</p>
|
|
<div class="caution" id="mrs_01_24088__note10688193423917"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><ol id="mrs_01_24088__ol934414385397"><li id="mrs_01_24088__li1634443873911">By default, only the two partitions with the largest size are clustered. The clustering of other partitions depends on the custom strategy.</li><li id="mrs_01_24088__li534493814394">The sorting column of clustering cannot be null. This is restricted by Spark RDD.</li><li id="mrs_01_24088__li134414387392">If the value of <strong id="mrs_01_24088__b5885183214115">target.file.max.bytes</strong> is large, increase the value of <strong id="mrs_01_24088__b198742104114">--spark-memory</strong> to execute clustering. Otherwise, the executor memory overflow occurs.</li><li id="mrs_01_24088__li9344123893913">Currently, the clean mechanism cannot be used to delete junk files generated after the clustering fails.</li><li id="mrs_01_24088__li11344123812396">After the clustering, sizes of new files may be different, causing data skew.</li><li id="mrs_01_24088__li976583812347">Clustering and upsert operations cannot be performed at the same time.</li></ol>
|
|
</div></div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24038.html">Data Management and Maintenance</a></div>
|
|
</div>
|
|
</div>
|
|
|