forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
17 lines
2.1 KiB
HTML
17 lines
2.1 KiB
HTML
<a name="mrs_01_1472"></a><a name="mrs_01_1472"></a>
|
|
|
|
<h1 class="topictitle1">How Do I Configure Unsafe Memory in CarbonData?</h1>
|
|
<div id="body1595920217110"><div class="section" id="mrs_01_1472__s37677e1cc7354d7a8a40bdf414e8f2e6"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1472__a495614a5ef1648a9ab57237a76e881c1">How do I configure unsafe memory in CarbonData?</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1472__sefdf4b043a0040a2bc2a33d383f106bc"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1472__ac9778c609a92418dbdfdb50afbad93b2">In the Spark configuration, the value of <span class="parmname" id="mrs_01_1472__p59ae09c41416487a83e6f88320248e91"><b>spark.yarn.executor.memoryOverhead</b></span> must be greater than the sum of (<span class="parmname" id="mrs_01_1472__pb37e8bcd6e80439786bdf71da24ba666"><b>sort.inmemory.size.inmb</b></span> + <span class="parmname" id="mrs_01_1472__pec8186b03a394b57b5ff784068a08bf3"><b>Netty offheapmemory required</b></span>), or the sum of (<span class="parmname" id="mrs_01_1472__p889a61774b7947cda3be155e6f10cc80"><b>carbon.unsafe.working.memory.in.mb</b></span> + <span class="parmname" id="mrs_01_1472__pa3cf65eb3bd24f08a54ece3e3085eb54"><b>carbon.sort.inememory.storage.size.in.mb</b></span> + <span class="parmname" id="mrs_01_1472__p7ecc8e7d6f7f472fa4deefec92c02dcc"><b>Netty offheapmemory required</b></span>). Otherwise, if off-heap access exceeds the configured executor memory, Yarn may stop the executor.</p>
|
|
<p id="mrs_01_1472__a4f2805ae1daf476b83ae575eea55347f">If <strong id="mrs_01_1472__b1543228164916">spark.shuffle.io.preferDirectBufs</strong> is set to <strong id="mrs_01_1472__b164330816498">true</strong>, the netty transfer service in Spark takes off some heap memory (around 384 MB or 0.1 x executor memory) from <strong id="mrs_01_1472__b74334819492">spark.yarn.executor.memoryOverhead</strong>.</p>
|
|
<p id="mrs_01_1472__a83c155a2579443a2adaca95d74efbfbe">For details, see <a href="mrs_01_1947.html">Configuring Executor Off-Heap Memory</a>.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1457.html">CarbonData FAQ</a></div>
|
|
</div>
|
|
</div>
|
|
|