forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
16 lines
1.3 KiB
HTML
16 lines
1.3 KiB
HTML
<a name="mrs_01_24074"></a><a name="mrs_01_24074"></a>
|
|
|
|
<h1 class="topictitle1">What Should I Do If Hudi Consumes Much Space in a Temporary Folder During Upsert?</h1>
|
|
<div id="body32001227"><div class="section" id="mrs_01_24074__en-us_topic_0000001173949978_section107038961220"><h4 class="sectiontitle">Question</h4><p id="mrs_01_24074__en-us_topic_0000001173949978_p162881034171512">Hudi consumes much space in a temporary folder during upsert.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24074__en-us_topic_0000001173949978_section654461315124"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_24074__en-us_topic_0000001173949978_p19980104312206">Hudi will spill part of input data to disk if the maximum memory for merge is reached when much input data is upserted.</p>
|
|
<p id="mrs_01_24074__en-us_topic_0000001173949978_p82881034151511">If the memory is sufficient, increase the memory of the Spark executor and add the <strong id="mrs_01_24074__en-us_topic_0000001173949978_b7523165711442">hoodie.memory.merge.fraction</strong> option, for example, <strong id="mrs_01_24074__en-us_topic_0000001173949978_b5797613154515">option("hoodie.memory.merge.fraction", "0.8")</strong>.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24070.html">Data Write</a></div>
|
|
</div>
|
|
</div>
|
|
|