forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
18 lines
2.9 KiB
HTML
18 lines
2.9 KiB
HTML
<a name="mrs_01_1459"></a><a name="mrs_01_1459"></a>
|
|
|
|
<h1 class="topictitle1">How to Avoid Minor Compaction for Historical Data?</h1>
|
|
<div id="body1595920215742"><div class="section" id="mrs_01_1459__s9df0afd73efa4c01a3e40c7d37367c57"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1459__adcb615019eb44b7da1823b4c3975e1f5">How to avoid minor compaction for historical data?</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1459__s7cfcd03c950f4008a527198d89dbbfe2"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1459__a903afa51fdca4ba195173b46de95a468">If you want to load historical data first and then the incremental data, perform following steps to avoid minor compaction of historical data:</p>
|
|
</div>
|
|
<ol id="mrs_01_1459__ob342d62888174281828595344715aa43"><li id="mrs_01_1459__lef25862779f843fa8630487968d49dd2">Load all historical data.</li><li id="mrs_01_1459__l577ff77a9d534cce97401cff495de38c">Configure the major compaction size to a value smaller than the segment size of historical data.</li><li id="mrs_01_1459__l51bbfee56db7427b8b7d17ae2227d18b">Run the major compaction once on historical data so that these segments will not be considered later for minor compaction.</li><li id="mrs_01_1459__l52abd3714b3d4f72b232504258973bbf">Load the incremental data.</li><li id="mrs_01_1459__lf25bc5bfd900406ca90ce0b5d98f0f0e">You can configure the minor compaction threshold as required.</li></ol>
|
|
<p id="mrs_01_1459__aa946550ce8554f1892d1c4695c851c42">For example:</p>
|
|
<ol id="mrs_01_1459__ob7572de3d3334794866e3ee57ebbeadb"><li id="mrs_01_1459__l116092aa5cf444c19b8a9954aa5f2cd4">Assume that you have loaded all historical data to CarbonData and the size of each segment is 500 GB.</li><li id="mrs_01_1459__l43943a5149ff4e5493a2710b63c3bf6c">Set the threshold of major compaction property to <span class="parmname" id="mrs_01_1459__p45b64d0ae0ab4570b341fc17175a3099"><b>carbon.major.compaction.size</b></span> = <span class="parmvalue" id="mrs_01_1459__pbd2a88e119c94f9db3d02a6acba80920"><b>491520</b></span> (480 GB x 1024).</li><li id="mrs_01_1459__lb94a97f4e2e0458e8427930932b5be8c">Run major compaction. All segments will be compacted because the size of each segment is more than configured size.</li><li id="mrs_01_1459__l60aef81df08548d2a175b190cb8a2fd9">Perform incremental loading.</li><li id="mrs_01_1459__l8ae1413f7aed401e9039f4d4a655b9e6">Configure the minor compaction threshold to <span class="parmname" id="mrs_01_1459__pcfa76501bfa04cf3ad03c8fbb1ff65b4"><b>carbon.compaction.level.threshold</b></span> = <span class="parmvalue" id="mrs_01_1459__pe2bc39b003114569844a349532ccf0f4"><b>6,6</b></span>.</li><li id="mrs_01_1459__l9c45e73a13994660a526fe93c17bc79a">Run minor compaction. As a result, only incremental data is compacted.</li></ol>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1457.html">CarbonData FAQ</a></div>
|
|
</div>
|
|
</div>
|
|
|