forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
164 lines
16 KiB
HTML
164 lines
16 KiB
HTML
<a name="mrs_01_24097"></a><a name="mrs_01_24097"></a>
|
|
|
|
<h1 class="topictitle1">Compaction and Cleaning Configurations</h1>
|
|
<div id="body0000001151303635">
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24097__table12264191413319" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_24097__row73071514830"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.1.1.4.1.1"><p id="mrs_01_24097__p73073148310">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="40%" id="mcps1.3.1.1.4.1.2"><p id="mrs_01_24097__p123071714832">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.1.1.4.1.3"><p id="mrs_01_24097__p2307191416317">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24097__row7357132775414"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p115401619297">hoodie.clean.automatic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p1335802714543">Specifies whether to perform automatic cleanup.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p83581327175414">true</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row13071614938"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p3307914332">hoodie.cleaner.policy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p93071514236">Specifies the cleaning policy to be used. Hudi will delete the Parquet file of an old version to reclaim space. Any query or computation referring to this version of the file will fail. You are advised to ensure that the data retention time exceeds the maximum query execution time.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p143072141536">KEEP_LATEST_COMMITS</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row330718141636"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p18307514132">hoodie.cleaner.commits.retained</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p14307514238">Specifies the number of commits to retain. Data will be retained for <strong id="mrs_01_24097__b84038483111110">num_of_commits * time_between_commits</strong> (scheduled). This also directly translates into the number of datasets can be incrementally pulled.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p103077140311">10</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row183072014934"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p530716149313">hoodie.keep.min.commits, hoodie.keep.max.commits</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p19307131411316">Each commit is a small file in the <strong id="mrs_01_24097__b121339242811110">.hoodie</strong> directory. DFS typically does not support a large number of small files, so Hudi archives older commits into a sequential log. A commit is published atomically by renaming the commit file.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p3307191418316">20</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row193071914431"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p93071614434">hoodie.commits.archival.batch</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p16307514737">This parameter controls the number of commit instants read in memory as a batch and archived together.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p23071414938">10</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row7307171410319"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p33073141313">hoodie.parquet.small.file.limit</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p18307111418313">The value must be smaller than that of <strong id="mrs_01_24097__b205291533811110">maxFileSize</strong>. If <strong id="mrs_01_24097__b14459945811110">maxFileSize</strong> is set to <strong id="mrs_01_24097__b3008566311110">0</strong>, this function is disabled. Small files always exist because of the large number of insert records in a partition of batch processing. Hudi provides an option to solve the problem of small files by masking inserts into this partition as updates to existing small files. The size here is the minimum file size that is considered as a "small file size".</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p177874554380">104857600 byte</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row53071414630"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p1730718141934">hoodie.copyonwrite.insert.split.size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p153077149312">Specifies the parallelism for inserting and writing data. It is the number of inserts grouped for a single partition. Writing out 100 MB files with at least 1 KB records means 100 KB records exist in each file. Overprovision to 500 KB by default. To improve insert latency, adjust the value to match the number of records in a single file. If it is set to a smaller value, the file size will shrink (especially when <strong id="mrs_01_24097__b197141114411110">compactionSmallFileSize</strong> is set to <strong id="mrs_01_24097__b155681211111110">0</strong>).</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p830710141639">500000</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row3307161414313"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p730712145311">hoodie.copyonwrite.insert.auto.split</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p1330718141236">Specifies whether Hudi dynamically computes <strong id="mrs_01_24097__b127378687711110">insertSplitSize</strong> based on the last 24 commit metadata. This function is disabled by default.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p1730714141135">true</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row1030718149314"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p153071144310">hoodie.copyonwrite.record.size.estimate</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p13074144315">Specifies the average record size. If specified, Hudi will use this parameter and not compute dynamically based on the last 24 commit metadata. There is no default value. This is critical in computing the insert parallelism and packing inserts into small files.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p16308131415315">1024</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row1430819141230"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p9308714535">hoodie.compact.inline</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p930811147316">If this parameter is set to <strong id="mrs_01_24097__b74176561511110">true</strong>, compaction is triggered by the ingestion itself right after a commit or delta commit action as part of <strong id="mrs_01_24097__b169965193911110">insert</strong>, <strong id="mrs_01_24097__b40710908611110">upsert</strong>, or <strong id="mrs_01_24097__b183620034011110">bulk_insert</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p83081914539">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row930816141134"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p1930812141532">hoodie.compact.inline.max.delta.commits</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p430813141939">Specifies the maximum number of delta commits to be retained before inline compression is triggered.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p7308161420312">5</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row123081514337"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p8308171416316">hoodie.compaction.lazy.block.read</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p123087141233">When <strong id="mrs_01_24097__b63305130711110">CompactedLogScanner</strong> merges all log files, this parameter helps to choose whether the logblocks should be read lazily. Set it to <strong id="mrs_01_24097__b122838123511110">true</strong> to use I/O-intensive lazy block read (low memory usage) or <strong id="mrs_01_24097__b26016477211110">false</strong> to use memory-intensive immediate block read (high memory usage).</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p6308914231">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row1930831412316"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p113089146312">hoodie.compaction.reverse.log.read</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p130871416313"><strong id="mrs_01_24097__b125657732911110">HoodieLogFormatReader</strong> reads a log file in the forward direction from <strong id="mrs_01_24097__b206283163811110">pos=0</strong> to <strong id="mrs_01_24097__b86246597111110">pos=file_length</strong>. If this parameter is set to <strong id="mrs_01_24097__b173226601911110">true</strong>, Reader reads a log file in reverse direction from <strong id="mrs_01_24097__b58113614311110">pos=file_length</strong> to <strong id="mrs_01_24097__b158413366311110">pos=0</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p10308814637">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row1230820142037"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p1530813141931">hoodie.cleaner.parallelism</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p230841415314">Increase this parameter if cleaning becomes slow.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p530820140317">200</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row1430819141437"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p183081214239">hoodie.compaction.strategy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p1530811414317">Determines which file groups are selected for compaction during each compaction run. By default, Hudi selects the log file with most accumulated unmerged data.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p14559131053910">org.apache.hudi.table.action.compact.strategy.</p>
|
|
<p id="mrs_01_24097__p8559510183915">LogFileSizeBasedCompactionStrategy</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row193088144320"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p230816141539">hoodie.compaction.target.io</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p1630811149317">Specifies the number of MBs to spend during compaction run for <strong id="mrs_01_24097__b19229180011110">LogFileSizeBasedCompactionStrategy</strong>. This parameter can limit ingestion latency when compaction is run in inline mode.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p1871572615396">500 * 1024 MB</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row123082141736"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p93086145311">hoodie.compaction.daybased.target</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p1830881412310">Used by <strong id="mrs_01_24097__b184788985911110">org.apache.hudi.io.compact.strategy.DayBasedCompactionStrategy</strong> to denote the number of latest partitions to compact during a compaction run.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p93081214930">10</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row13308151413316"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p630841413310">hoodie.compaction.payload.class</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p113083141837">It needs to be same as class used during insert or upsert. Similar to writing, compaction also uses the record payload class to merge records in the log against each other, merge again with the base file, and produce the final record to be written after compaction.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p12308314437">org.apache.hudi.common.model.OverwriteWithLatestAvroPayload</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row938815599578"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p1638875917578">hoodie.schedule.compact.only.inline</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p15388105965711">Specifies whether to generate only a compression plan during a write operation. This parameter is valid only when <strong id="mrs_01_24097__b66692012145216">hoodie.compact.inline</strong> is set to <strong id="mrs_01_24097__b5521623195216">true</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p438820592573">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24097__row1680314475817"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.1 "><p id="mrs_01_24097__p880310414588">hoodie.run.compact.only.inline</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.1.4.1.2 "><p id="mrs_01_24097__p188031435811">Specifies whether to perform only the compression operation when the <strong id="mrs_01_24097__b9421201495717">run compaction</strong> command is executed using SQL. If the compression plan does not exist, no action is needed.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.1.4.1.3 "><p id="mrs_01_24097__p1580318435819">false</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24032.html">Configuration Reference</a></div>
|
|
</div>
|
|
</div>
|
|
|