forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
19 lines
2.9 KiB
HTML
19 lines
2.9 KiB
HTML
<a name="mrs_01_2043"></a><a name="mrs_01_2043"></a>
|
|
|
|
<h1 class="topictitle1">Why Is Memory Insufficient if 10 Terabytes of TPCDS Test Suites Are Consecutively Run in Beeline/JDBCServer Mode?</h1>
|
|
<div id="body1595920222661"><div class="section" id="mrs_01_2043__s37f28ed28f164da08b1dcb2457062784"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2043__a40f64b625f074fa69502558ee2b93add">When the driver memory is set to 10 GB and the 10 TB TPCDS test suites are continuously run in Beeline/JDBCServer mode, SQL statements fail to be executed due to insufficient driver memory. Why?</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_2043__s4b606433dbb64cba86c36a5bd282303e"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_2043__a94e9af52ffc04af3ab823402eb25fdc7">By default, 1000 UI data records of jobs and stages are reserved in the memory.</p>
|
|
<p id="mrs_01_2043__a1cac649e9a4f4756b81d3f5671dc4d33">The function of overflowing UI data to disks has been added to optimize large clusters. The overflow condition is that the size of UI data in each stage reaches the minimum threshold 5 MB. If the number of tasks in each stage is small, the size of UI data in the stage may not reach the threshold. As a result, the UI data in the stage is cached in the memory until the number of UI data records reaches the upper limit (1000 by default). Only then the old UI data is cleared from the memory.</p>
|
|
<p id="mrs_01_2043__a3bbbd5c0af284be4b8abe480ceb3631e">Therefore, before the old UI data is cleared, the UI data occupies a large amount of memory. As a result, the driver memory is insufficient when 10 terabytes of TPCDS test suites are executed.</p>
|
|
<p id="mrs_01_2043__af52cebbd1d60486aa8c9fade617dbd8b">Workaround:</p>
|
|
<ul id="mrs_01_2043__ufa8a399c1a684533ad7744a07efdea93"><li id="mrs_01_2043__lacd154d536f04820a3663a4a641a3b15">Set <span class="parmname" id="mrs_01_2043__parmname280984049112543"><b>spark.ui.retainedJobs</b></span> and <span class="parmname" id="mrs_01_2043__parmname1304332926112543"><b>spark.ui.retainedStages</b></span> based on service requirements to specify the number of UI data records of jobs and stages to be reserved. For details, see <a href="mrs_01_1931.html#mrs_01_1931__t681877b034a54c50a58b9e1864345ee4">Table 13</a> in <a href="mrs_01_1931.html">Common Parameters</a>.</li><li id="mrs_01_2043__l2bdb9ed8802949548c5d96f8af15fb79">If a large amount of UI data of jobs and stages needs to be reserved, increase the memory of the driver by setting the <span class="parmname" id="mrs_01_2043__peebdd22a61774a7aad7470bfb0adc49a"><b>spark.driver.memory</b></span> parameter. For details, see <a href="mrs_01_1931.html#mrs_01_1931__t846a81171d4c4af1908c5cf55578f022">Table 10</a> in <a href="mrs_01_1931.html">Common Parameters</a>.</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2022.html">Spark SQL and DataFrame</a></div>
|
|
</div>
|
|
</div>
|
|
|