forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
37 lines
3.8 KiB
HTML
37 lines
3.8 KiB
HTML
<a name="mrs_01_1952"></a><a name="mrs_01_1952"></a>
|
|
|
|
<h1 class="topictitle1">Configuring the Default Number of Data Blocks Divided by SparkSQL</h1>
|
|
<div id="body1595920207017"><div class="section" id="mrs_01_1952__sa495cb5f396b4610b9880fbade553007"><h4 class="sectiontitle">Scenarios</h4><p id="mrs_01_1952__a1045e435c0db4b12810ac1158e9c5948">By default, SparkSQL divides data into 200 data blocks during shuffle. In data-intensive scenarios, each data block may have excessive size. If a single data block of a task is larger than 2 GB, an error similar to the following will be reported while Spark attempts to fetch the data block:</p>
|
|
<pre class="screen" id="mrs_01_1952__seb1a6807e53a4d1db4daca0fb28d4006">Adjusted frame length exceeds 2147483647: 2717729270 - discarded</pre>
|
|
<p id="mrs_01_1952__aa142cd4398cb4f22a14929dee5c81aab">For example, setting the number of default data blocks to 200 causes SparkSQL to encounter an error in running a TPCDS 500-GB test. To avoid this, increase the number of default blocks in data-intensive scenarios.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1952__s4862f9a9a0ee4e97b6a1a8dffee0a15d"><h4 class="sectiontitle">Configuration parameters</h4><p id="mrs_01_1952__ad9da1e1c012f4f7fbf95227018f1ddb5"><strong id="mrs_01_1952__a19b85ce4b3544e2496f9ae7b00146e95">Navigation path for setting parameters:</strong></p>
|
|
<p id="mrs_01_1952__af07bffcb0ec84ae498154f2a634e4297">On Manager, choose <span class="menucascade" id="mrs_01_1952__menucascade1587206391"><b><span class="uicontrol" id="mrs_01_1952__uicontrol18871006396"><span id="mrs_01_1952__text13867012390">Cluster > <em id="mrs_01_1952__i8813015398">Name of the desired cluster</em> > </span>Service</span></b> > <b><span class="uicontrol" id="mrs_01_1952__uicontrol387006391">Spark2x</span></b> > <b><span class="uicontrol" id="mrs_01_1952__uicontrol198710013394">Configuration</span></b></span> and click <span class="parmvalue" id="mrs_01_1952__parmvalue19887083916"><b>All Configurations</b></span>. Enter a parameter name in the search box.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1952__tfdfbc9c242124d3dbd94d8db2cdadae4" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1952__r681a06e4f9a94bcf88e8395aaa7beafb"><th align="left" class="cellrowborder" valign="top" width="21.55%" id="mcps1.3.2.4.2.4.1.1"><p id="mrs_01_1952__aaacf5eb323b2478e8c38f01856242d4f"><strong id="mrs_01_1952__a822d74ab75f94be28e76c5b2a066b834">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="65.73%" id="mcps1.3.2.4.2.4.1.2"><p id="mrs_01_1952__af742340232ed422395eb63263d030366"><strong id="mrs_01_1952__ae3ae27c029d44a5ab75a79eb03defad4">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="12.72%" id="mcps1.3.2.4.2.4.1.3"><p id="mrs_01_1952__aa0f014da8d724f7dba760fe0c9c2b594"><strong id="mrs_01_1952__a4a55b0a09e264bd5b3eb3f4ce63615bf">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1952__rc3b9d8e7003b47a6afa47ad05369ae76"><td class="cellrowborder" valign="top" width="21.55%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_1952__af314f6d269cd45e89a4775c422833040">spark.sql.shuffle.partitions</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.73%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_1952__a0b446c0274bd41949f527706da7c128b">Indicates the default number of blocks divided during shuffle.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.72%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_1952__a168f0267e7604b91aab8524554601f5b">200</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|