forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
25 lines
2.8 KiB
HTML
25 lines
2.8 KiB
HTML
<a name="mrs_01_1474"></a><a name="mrs_01_1474"></a>
|
|
|
|
<h1 class="topictitle1">Why Does Data Query or Loading Fail and "org.apache.carbondata.core.memory.MemoryException: Not enough memory" Is Displayed?</h1>
|
|
<div id="body1595920217113"><div class="section" id="mrs_01_1474__s32e8fb6882ae433d9dc51e0975a36376"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1474__a229d3f983c0447c8801c8a63ebbf2531">Why does data query or loading fail and "org.apache.carbondata.core.memory.MemoryException: Not enough memory" is displayed?</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1474__s715ca1e23e314af2ac4071272b0530ae"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1474__a53fde1a2b653437f917799eeeef4ea28">This exception is thrown when the out-of-heap memory required for data query and loading in the executor is insufficient.</p>
|
|
<p id="mrs_01_1474__a21b1c39be02d4e048c6faee76acbc24a">In this case, increase the values of <strong id="mrs_01_1474__b13523121135119">carbon.unsafe.working.memory.in.mb</strong> and <strong id="mrs_01_1474__b45294119516">spark.yarn.executor.memoryOverhead</strong>.</p>
|
|
<p id="mrs_01_1474__a5d25d2c085b44f769ada81894d743c39">For details, see <a href="mrs_01_1472.html">How Do I Configure Unsafe Memory in CarbonData?</a>.</p>
|
|
<p id="mrs_01_1474__aa5dc76b953964c9dbb095f44f100ace1">The memory is shared by data query and loading. Therefore, if the loading and query operations need to be performed at the same time, you are advised to set <strong id="mrs_01_1474__b135202011135114">carbon.unsafe.working.memory.in.mb</strong> and <strong id="mrs_01_1474__b12520131118510">spark.yarn.executor.memoryOverhead</strong> to a value greater than 2,048 MB.</p>
|
|
<p id="mrs_01_1474__a8579a48f0feb4c7f96df6f73d8e22095">The following formula can be used for estimation:</p>
|
|
<p id="mrs_01_1474__aefc3788228444bab8289f0c8ce039bd3">Memory required for data loading:</p>
|
|
<p id="mrs_01_1474__a3fe80cf6b46f4c2eb814c6f3f2f69dec">carbon.number.of.cores.while.loading [default value is 6] x Number of tables to load in parallel x offheap.sort.chunk.size.inmb [default value is 64 MB] + carbon.blockletgroup.size.in.mb [default value is 64 MB] + Current compaction ratio [64 MB/3.5])</p>
|
|
<p id="mrs_01_1474__ac8ee35f0a9c7441db1818f14956a4f2c">= Around 900 MB per table</p>
|
|
<p id="mrs_01_1474__a62c9197550b3418dbe8bc45ef46015bb">Memory required for data query:</p>
|
|
<p id="mrs_01_1474__a16cdc26693d841798559cc6706c3ed2b">(SPARK_EXECUTOR_INSTANCES. [default value is 2] x (carbon.blockletgroup.size.in.mb [default value: 64 MB] + carbon.blockletgroup.size.in.mb [default value = 64 MB x 3.5) x Number of cores per executor [default value: 1])</p>
|
|
<p id="mrs_01_1474__aa0ec0b13a24a4890a3263c299c60215e">= ~ 600 MB</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1457.html">CarbonData FAQ</a></div>
|
|
</div>
|
|
</div>
|
|
|