forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
71 lines
12 KiB
HTML
71 lines
12 KiB
HTML
<a name="mrs_01_1964"></a><a name="mrs_01_1964"></a>
|
|
|
|
<h1 class="topictitle1"><strong id="en-us_topic_0000001173789672_b1708571440">Configuring Vector-based ORC Data Reading</strong></h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_1964__en-us_topic_0000001173789672_section880553614458"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1964__en-us_topic_0000001173789672_p88512036104518">ORC is a column-based storage format in the Hadoop ecosystem. It originates from Apache Hive and is used to reduce the Hadoop data storage space and accelerate the Hive query speed. Similar to Parquet, ORC is not a pure column-based storage format. In the ORC format, the entire table is split based on the row group, data in each row group is stored by column, and data is compressed as much as possible to reduce storage space consumption. Vector-based ORC data reading significantly improves the ORC data reading performance. In Spark2.3, SparkSQL supports vector-based ORC data reading (this function is supported in earlier Hive versions). Vector-based ORC data reading improves the data reading performance by multiple times.</p>
|
|
<p id="mrs_01_1964__en-us_topic_0000001173789672_p9851336144510">This function is controlled by the following parameters:</p>
|
|
<ul id="mrs_01_1964__en-us_topic_0000001173789672_ul8851133619453"><li id="mrs_01_1964__en-us_topic_0000001173789672_li285115367450"><strong id="mrs_01_1964__en-us_topic_0000001173789672_b485120368451">spark.sql.orc.enableVectorizedReader</strong>: Specifies whether vector-based ORC data reading is supported. The default value is <strong id="mrs_01_1964__en-us_topic_0000001173789672_b68517367453">true</strong>.</li><li id="mrs_01_1964__en-us_topic_0000001173789672_li485114362451"><strong id="mrs_01_1964__en-us_topic_0000001173789672_b1985163620454">spark.sql.codegen.wholeStage</strong>: Specifies whether to compile all stages of multiple operations into a Java method. The default value is <strong id="mrs_01_1964__en-us_topic_0000001173789672_b208514367454">true</strong>.</li><li id="mrs_01_1964__en-us_topic_0000001173789672_li1785113617457"><strong id="mrs_01_1964__en-us_topic_0000001173789672_b68511436114519">spark.sql.codegen.maxFields</strong>: Specifies the maximum number of fields (including nested fields) supported by all stages of codegen. The default value is <strong id="mrs_01_1964__en-us_topic_0000001173789672_b18851936164518">100</strong>.</li><li id="mrs_01_1964__en-us_topic_0000001173789672_li1485193615454"><strong id="mrs_01_1964__en-us_topic_0000001173789672_b10851153610455">spark.sql.orc.impl</strong>: Specifies whether Hive or Spark SQL native is used as the SQL execution engine to read ORC data. The default value is <strong id="mrs_01_1964__en-us_topic_0000001173789672_b1585111364457">hive</strong>.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1964__en-us_topic_0000001173789672_section16965344511"><h4 class="sectiontitle">Parameters</h4><p id="mrs_01_1964__en-us_topic_0000001173789672_p122414538455">Log in to FusionInsight Manager, choose <strong id="mrs_01_1964__en-us_topic_0000001173789672_b12293512913">Cluster > </strong><em id="mrs_01_1964__en-us_topic_0000001173789672_i132319512919">Name of the desired cluster</em><strong id="mrs_01_1964__en-us_topic_0000001173789672_b2229135091"> > Service<span id="mrs_01_1964__en-us_topic_0000001173789672_ph1769462519378">s</span> </strong>> <strong id="mrs_01_1964__en-us_topic_0000001173789672_a6bda9b42e87d47b8ba33b7bcf8d7f67a">Spark2x</strong>. On the displayed page, choose <strong id="mrs_01_1964__en-us_topic_0000001173789672_b5249538454">Configuration<span id="mrs_01_1964__en-us_topic_0000001173789672_ph1328193003720">s</span></strong> > <strong id="mrs_01_1964__en-us_topic_0000001173789672_b102425314512">All Configurations</strong>. Search for parameters listed in the following table and configure them as required.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1964__en-us_topic_0000001173789672_table9713468467" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_1964__en-us_topic_0000001173789672_row16759136124614"><th align="left" class="cellrowborder" valign="top" width="25.252525252525253%" id="mcps1.3.2.3.1.5.1.1"><p id="mrs_01_1964__en-us_topic_0000001173789672_p16759667463">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="39.39393939393939%" id="mcps1.3.2.3.1.5.1.2"><p id="mrs_01_1964__en-us_topic_0000001173789672_p47599612462">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="14.14141414141414%" id="mcps1.3.2.3.1.5.1.3"><p id="mrs_01_1964__en-us_topic_0000001173789672_p675919620463">Default Value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="21.21212121212121%" id="mcps1.3.2.3.1.5.1.4"><p id="mrs_01_1964__en-us_topic_0000001173789672_p147595694619">Value Range</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1964__en-us_topic_0000001173789672_row97592061469"><td class="cellrowborder" valign="top" width="25.252525252525253%" headers="mcps1.3.2.3.1.5.1.1 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p37599644616">spark.sql.orc.enableVectorizedReader</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.39393939393939%" headers="mcps1.3.2.3.1.5.1.2 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p6759565466">Specifies whether vector-based ORC data reading is supported. The default value is <strong id="mrs_01_1964__en-us_topic_0000001173789672_b1575912610465">true</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.14141414141414%" headers="mcps1.3.2.3.1.5.1.3 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p1075996184617">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.21212121212121%" headers="mcps1.3.2.3.1.5.1.4 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p775914664618">[true,false]</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1964__en-us_topic_0000001173789672_row8759106184617"><td class="cellrowborder" valign="top" width="25.252525252525253%" headers="mcps1.3.2.3.1.5.1.1 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p875920624613">spark.sql.codegen.wholeStage</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.39393939393939%" headers="mcps1.3.2.3.1.5.1.2 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p4759869467">Specifies whether to compile all stages of multiple operations into a Java method. The default value is <strong id="mrs_01_1964__en-us_topic_0000001173789672_b1675910617463">true</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.14141414141414%" headers="mcps1.3.2.3.1.5.1.3 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p207590611465">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.21212121212121%" headers="mcps1.3.2.3.1.5.1.4 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p1575919618463">[true,false]</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1964__en-us_topic_0000001173789672_row137591963460"><td class="cellrowborder" valign="top" width="25.252525252525253%" headers="mcps1.3.2.3.1.5.1.1 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p1575916134613">spark.sql.codegen.maxFields</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.39393939393939%" headers="mcps1.3.2.3.1.5.1.2 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p9759196134619">Specifies the maximum number of fields (including nested fields) supported by all stages of codegen. The default value is <strong id="mrs_01_1964__en-us_topic_0000001173789672_b375956154614">100</strong>.</p>
|
|
<p id="mrs_01_1964__en-us_topic_0000001173789672_p3759186204610"></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.14141414141414%" headers="mcps1.3.2.3.1.5.1.3 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p1075910654615">100</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.21212121212121%" headers="mcps1.3.2.3.1.5.1.4 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p575919664610">≥ 1</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1964__en-us_topic_0000001173789672_row1175916104615"><td class="cellrowborder" valign="top" width="25.252525252525253%" headers="mcps1.3.2.3.1.5.1.1 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p1375913634615">spark.sql.orc.impl</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.39393939393939%" headers="mcps1.3.2.3.1.5.1.2 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p177591669466">Specifies whether Hive or Spark SQL native is used as the SQL execution engine to read ORC data. The default value is <strong id="mrs_01_1964__en-us_topic_0000001173789672_b675986164613">hive</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.14141414141414%" headers="mcps1.3.2.3.1.5.1.3 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p1575914614464">hive</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.21212121212121%" headers="mcps1.3.2.3.1.5.1.4 "><p id="mrs_01_1964__en-us_topic_0000001173789672_p18759176124613">[hive,native]</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_1964__en-us_topic_0000001173789672_note10213182115464"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ol id="mrs_01_1964__en-us_topic_0000001173789672_ol18463048184617"><li id="mrs_01_1964__en-us_topic_0000001173789672_li94635483461">To use vector-based ORC data reading of SparkSQL, the following conditions must be met:<ul id="mrs_01_1964__en-us_topic_0000001173789672_ul04631248144616"><li id="mrs_01_1964__en-us_topic_0000001173789672_li1546334844611"><strong id="mrs_01_1964__en-us_topic_0000001173789672_b8478648124612">spark.sql.orc.enableVectorizedReader</strong> must be set to <strong id="mrs_01_1964__en-us_topic_0000001173789672_b17478134814613">true</strong> (default value). Generally, the value is not changed.</li><li id="mrs_01_1964__en-us_topic_0000001173789672_li1047814483460"><strong id="mrs_01_1964__en-us_topic_0000001173789672_b174786489466">spark.sql.codegen.wholeStage</strong> must be set to <strong id="mrs_01_1964__en-us_topic_0000001173789672_b1478348164618">true</strong> (default value). Generally, the value is not changed.</li><li id="mrs_01_1964__en-us_topic_0000001173789672_li164781648194617">The value of <strong id="mrs_01_1964__en-us_topic_0000001173789672_b247814811468">spark.sql.codegen.maxFields</strong> must be greater than or equal to the number of columns in scheme.</li><li id="mrs_01_1964__en-us_topic_0000001173789672_li447813488465">All data is of the AtomicType. Specifically, data is not null or of the UDT, array, or map type. If there is data of the preceding types, expected performance cannot be obtained.</li><li id="mrs_01_1964__en-us_topic_0000001173789672_li16478184815464"><strong id="mrs_01_1964__en-us_topic_0000001173789672_b44789482465">spark.sql.orc.impl</strong> must be set to <strong id="mrs_01_1964__en-us_topic_0000001173789672_b11478174819465">native</strong>. The default value is <strong id="mrs_01_1964__en-us_topic_0000001173789672_b1247804874619">hive</strong>.</li></ul>
|
|
</li><li id="mrs_01_1964__en-us_topic_0000001173789672_li17478148154614">If a task is submitted through the client, modification of the following parameters takes effect only after you download the client again: <strong id="mrs_01_1964__en-us_topic_0000001173789672_b447844819468">spark.sql.orc.enableVectorizedReader</strong>, <strong id="mrs_01_1964__en-us_topic_0000001173789672_b12478948184610">spark.sql.codegen.wholeStage</strong>, <strong id="mrs_01_1964__en-us_topic_0000001173789672_b164781248174612">spark.sql.codegen.maxFields</strong>, and <strong id="mrs_01_1964__en-us_topic_0000001173789672_b1947844894613">amd spark.sql.orc.impl</strong>.</li></ol>
|
|
</div></div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|