forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
20 lines
3.5 KiB
HTML
20 lines
3.5 KiB
HTML
<a name="mrs_01_1464"></a><a name="mrs_01_1464"></a>
|
|
|
|
<h1 class="topictitle1">Why INSERT INTO/LOAD DATA Task Distribution Is Incorrect and the Opened Tasks Are Less Than the Available Executors when the Number of Initial Executors<span id="ph8548202823810"> </span>Is Zero?</h1>
|
|
<div id="body1595920215968"><div class="section" id="mrs_01_1464__s41230294a0df4d63934ef2af8113fb17"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1464__a21fcee98ae82405885ee13a3daa6c497">Why <strong id="mrs_01_1464__a3c08220ff01542a8b3dc45387c34608d">INSERT INTO or LOAD DATA</strong> task distribution is incorrect, and the openedtasks are less than the available executors when the number of initial executors is zero<strong id="mrs_01_1464__afeaffae503de41f39028f8c80c07fdad">?</strong></p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1464__sbbb813ec80ce43d8b9ff69244f437b8a"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1464__a6dc0aa38305d4e7eb1c7186f654434cc">In case of INSERT INTO or LOAD DATA, CarbonData distributes one task per node. If the executors are not allocated from the distinct nodes then CarbonData will launch fewer tasks.</p>
|
|
<p id="mrs_01_1464__a940e8d03d60245de93f17ad2f59bd371"><strong id="mrs_01_1464__a640f08b5609249fb87ab521ebc39eefa">Solution</strong>:</p>
|
|
<p id="mrs_01_1464__a189c1d63701042b192ac004b27ea01dc">Configure higher value for the executor memory and core so that the yarn can launch only one executor per node.</p>
|
|
<ol id="mrs_01_1464__ob576d7e6910440529853bb1fc6c00949"><li id="mrs_01_1464__lc5d8cd22452b4f5f8ca7a064890099b4">Configure the number of the Executor cores.<ul id="mrs_01_1464__u5b4118bd0c8145d4b8cac162d22b1781"><li id="mrs_01_1464__l39204a8ba5434ceb9b48d1a77003f425">Configure the <span class="parmname" id="mrs_01_1464__p03513d0874d84d78ac647536faef696b"><b>spark.executor.cores</b></span> in <span class="filepath" id="mrs_01_1464__fc893bf23493147aebecd48079b0ac89d"><b>spark-defaults.conf</b></span> or the <span class="parmname" id="mrs_01_1464__p5455b27ce7f846cc8d9d7e632c84535d"><b>SPARK_EXECUTOR_CORES</b></span> in <span class="filepath" id="mrs_01_1464__fca85a300d2fe4a90863b6a3255317b6c"><b>spark-env.sh</b></span> appropriately.</li><li id="mrs_01_1464__l8b9e64a56aa24f07820d03b68b88356c">Add <span class="parmname" id="mrs_01_1464__p16f441996e5943c68a50b55083eafff2"><b>--executor-cores NUM</b></span> parameter to configure the cores during use the spark-submit command.</li></ul>
|
|
</li><li id="mrs_01_1464__ld4dd6b1f343442deb575d5c5d1d8387e">Configure the Executor memory.<ul id="mrs_01_1464__uf50b808f1ea143ea9cdf3402ba9dfc9c"><li id="mrs_01_1464__le04444b32c5b4cd2979897fbd08a276e">Configure the <span class="parmname" id="mrs_01_1464__pfc6f296899e54fbfb3b2c420621ab246"><b>spark.executor.memory</b></span> in <span class="filepath" id="mrs_01_1464__f59af5bd4028b4f11a228b0f1e417ee58"><b>spark-defaults.conf</b></span> or the <span class="parmname" id="mrs_01_1464__p9d7d4b530ae34eb58a1d89dcd3a12bdc"><b>SPARK_EXECUTOR_MEMORY</b></span> in <span class="filepath" id="mrs_01_1464__fe664d778aba64bb38a31606c2dbf5bbb"><b>spark-env.sh</b></span> appropriately.</li><li id="mrs_01_1464__l1f043c25559f4998bda6dc404554eba7">Add <span class="parmname" id="mrs_01_1464__p2f55f1eb322646ebb445d80f49f100f0"><b>--executor-memory MEM</b></span> parameter to configure the memory during use the spark-submit command.</li></ul>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1457.html">CarbonData FAQ</a></div>
|
|
</div>
|
|
</div>
|
|
|