forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
70 lines
8.5 KiB
HTML
70 lines
8.5 KiB
HTML
<a name="mrs_01_1961"></a><a name="mrs_01_1961"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Streaming Reading of Driver Execution Results</h1>
|
|
<div id="body1595920207216"><div class="section" id="mrs_01_1961__s7301938a48de46528f302a5a39912773"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1961__ab74bc9c0556e45a5bdad53b1d810bfd1">When a query statement is executed, the returned result may be large (containing more than 100,000 records). In this case, JDBCServer out of memory (OOM) may occur. Therefore, the data aggregation function is provided to avoid OOM without sacrificing the performance.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1961__s306e853eb7fc4336a3e90c2110c0620e"><h4 class="sectiontitle">Configuration</h4><p id="mrs_01_1961__adbf5d09f38f743a7ac62802295b4c51e">Two data aggregation function configuration parameters are provided. The two parameters are set in the <strong id="mrs_01_1961__b14894114343111">tunning</strong> option on the Spark JDBCServer server. After the setting is complete, restart JDBCServer.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1961__t1dac15ae5370449ea73622cf454ab451" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1961__re69fece14b684d2dadf874cc71aeb8dd"><th align="left" class="cellrowborder" valign="top" width="20.18%" id="mcps1.3.2.3.2.4.1.1"><p id="mrs_01_1961__ac5f761679d2d4c7d9b4325b446c5dcfa">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="63.970000000000006%" id="mcps1.3.2.3.2.4.1.2"><p id="mrs_01_1961__af69382a636df40fe8b107e1103f4fc09">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="15.85%" id="mcps1.3.2.3.2.4.1.3"><p id="mrs_01_1961__a0dff0aaaca3b4ec287c495a5509397bb">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1961__rd5cbd430544b489ab268fcf3ab5d5fcb"><td class="cellrowborder" valign="top" width="20.18%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1961__aaa404b17b2904ba792a956dc2336e219">spark.sql.bigdata.thriftServer.useHdfsCollect</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.970000000000006%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1961__a3e1b0f97028243f9a8f0be5ff9ea562a">Indicates whether to save result data to HDFS instead of the memory.</p>
|
|
<p id="mrs_01_1961__p497052611544">Advantages: The query result is stored in HDFS. Therefore, JDBCServer OOM does not occur.</p>
|
|
<p id="mrs_01_1961__p524611195614">Disadvantages: The query is slow.</p>
|
|
<ul id="mrs_01_1961__ue3533017daba41fea03c857fbcf5d165"><li id="mrs_01_1961__l502d4848b5eb4af19d6aab2cc487eb09"><strong id="mrs_01_1961__b18596102653210">true</strong>: Result data is saved to HDFS.</li><li id="mrs_01_1961__l43a3bd7dee78491d89d28a92e39c75bb"><strong id="mrs_01_1961__b187240546336">false</strong>: This function is disabled.<div class="notice" id="mrs_01_1961__note6531122417361"><span class="noticetitle"> NOTICE: </span><div class="noticebody"><p id="mrs_01_1961__p10531112443619">When <strong id="mrs_01_1961__b7771956133316">spark.sql.bigdata.thriftServer.useHdfsCollect</strong> is set to <strong id="mrs_01_1961__b14771125613336">true</strong>, result data is saved to HDFS. However, the job description on the native JobHistory page cannot be associated with the corresponding SQL statement. In addition, the execution ID in the spark-beeline command output is null. To solve the JDBCServer OOM problem and ensure correct information display, you are advised to set <strong id="mrs_01_1961__b77711556183313">spark.sql.userlocalFileCollect</strong>.</p>
|
|
</div></div>
|
|
</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15.85%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1961__a38f561f5d2b1421c9c8214ef30f139dd">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1961__row963492512568"><td class="cellrowborder" valign="top" width="20.18%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1961__p14635625135611">spark.sql.uselocalFileCollect</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.970000000000006%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1961__p163512252568">Indicates whether to save result data to the local disk instead of memory.</p>
|
|
<p id="mrs_01_1961__p7302153517573">Advantages: In the case of small data volume, the performance loss can be ignored compared with the data storage mode using the native memory. In the case of large data volume (hundreds of millions of data records), the performance is much better than that when data is stored in the HDFS and native memory.</p>
|
|
<p id="mrs_01_1961__p2034818145110">Disadvantages: Optimization is required. In the case of large data volume, it is recommended that the JDBCServer driver memory be 10 GB and each core of the executor be allocated with 3 GB memory.</p>
|
|
<ul id="mrs_01_1961__ul131161844312"><li id="mrs_01_1961__li101161647314"><strong id="mrs_01_1961__b19121726104017">true</strong>: This function is enabled.</li><li id="mrs_01_1961__li1979859315"><strong id="mrs_01_1961__b15981142894015">false</strong>: This function is disabled.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15.85%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1961__p26358252567">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1961__row7937102517510"><td class="cellrowborder" valign="top" width="20.18%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1961__p10938625953">spark.sql.collect.Hive</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.970000000000006%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1961__p14938525153">This parameter is valid only when <strong id="mrs_01_1961__b31701933194016">spark.sql.uselocalFileCollect</strong> is set to <strong id="mrs_01_1961__b1917510337406">true</strong>. It indicates whether to save the result data to a disk in direct serialization mode or in indirect serialization mode.</p>
|
|
<p id="mrs_01_1961__p1558919101188">Advantage: For queries of tables with a large number of partitions, the aggregation performance of the query results is better than that of the storage mode that query results are directly stored on the disk.</p>
|
|
<p id="mrs_01_1961__p531055017918">Disadvantages: The disadvantages are the same as those when <strong id="mrs_01_1961__b18272122311517">spark.sql.uselocalFileCollect</strong> is enabled.</p>
|
|
<ul id="mrs_01_1961__ul186622366611"><li id="mrs_01_1961__li159586481710"><strong id="mrs_01_1961__b1577602615511">true</strong>: This function is enabled.</li><li id="mrs_01_1961__li1066203617613"><strong id="mrs_01_1961__b14327728175116">false</strong>: This function is disabled.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15.85%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1961__p179387251751">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1961__row2014532031113"><td class="cellrowborder" valign="top" width="20.18%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1961__p414632061111">spark.sql.collect.serialize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.970000000000006%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1961__p116626410119">This parameter takes effect only when both <strong id="mrs_01_1961__b77432032105114">spark.sql.uselocalFileCollect</strong> and <strong id="mrs_01_1961__b1274883217512">spark.sql.collect.Hive</strong> are set to <strong id="mrs_01_1961__b6749203225117">true</strong>.</p>
|
|
<p id="mrs_01_1961__p173919546111">The function is to further improve performance.</p>
|
|
<ul id="mrs_01_1961__ul154721427171319"><li id="mrs_01_1961__li11472172715130"><strong id="mrs_01_1961__b53391951155118">java</strong>: Data is collected in Java serialization mode.</li><li id="mrs_01_1961__li24723275134"><strong id="mrs_01_1961__b15507654125112">kryo</strong>: Data is collected in kryo serialization mode. The performance is better than that when the Java serialization mode is used.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="15.85%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1961__p11468204116">java</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_1961__note19784155311417"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1961__p2784175394113"><strong id="mrs_01_1961__b179591414105217">spark.sql.bigdata.thriftServer.useHdfsCollect</strong> and <strong id="mrs_01_1961__b1896471465215">spark.sql.uselocalFileCollect</strong> cannot be set to <strong id="mrs_01_1961__b159641614115220">true</strong> at the same time.</p>
|
|
</div></div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|