forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
42 lines
7.1 KiB
HTML
42 lines
7.1 KiB
HTML
<a name="mrs_01_0983"></a><a name="mrs_01_0983"></a>
|
|
|
|
<h1 class="topictitle1">Optimizing the Query Function Using Hive CBO</h1>
|
|
<div id="body1590395285876"><div class="section" id="mrs_01_0983__section7297162516331"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0983__p19297112593315">When joining multiple tables in Hive, Hive supports Cost-Based Optimization (CBO). The system automatically selects the optimal plan based on the table statistics, such as the data volume and number of files, to improve the efficiency of joining multiple tables. Hive needs to collect table statistics before CBO optimization.</p>
|
|
<div class="note" id="mrs_01_0983__note172971325123310"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0983__ul12971325113310"><li id="mrs_01_0983__li7297122543316">The CBO optimizes the joining sequence based on statistics and search criteria. However, the joining sequence may fail to be optimized in some special scenarios, such as data skew occurs and query condition values are not in the table.</li><li id="mrs_01_0983__li429702533312">When column statistics collection is enabled, Reduce operations must be performed for aggregation. For insert tasks without the Reduce phase, Reduce operations will be performed to collect statistics. </li><li id="mrs_01_0983__li10279193213345">This section applies to MRS 3.<em id="mrs_01_0983__i166187151806">x</em> or later clusters.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_0983__section7297122514331"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_0983__p1129742523314">You have logged in to the Hive client. For details, see <a href="mrs_01_0952.html">Using a Hive Client</a>.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0983__section78264215339"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_0983__ol4297192513337"><li id="mrs_01_0983__li3297925193318"><span>On the Manager UI, search for the <span class="parmname" id="mrs_01_0983__parmname529716258336"><b>hive.cbo.enable</b></span> parameter in the service configuration of the Hive component, and select <span class="parmvalue" id="mrs_01_0983__parmvalue5297182543320"><b>true</b></span> to enable the function permanently.</span></li><li id="mrs_01_0983__li8297122511335"><span>Collect statistics about the existing data in Hive tables manually.</span><p><p id="mrs_01_0983__p14297425183315">Run the following command to manually collect statistics: Statistics about only one table can be collected. If statistics about multiple tables need to be collected, the command needs to be executed repeatedly.</p>
|
|
<p id="mrs_01_0983__p132981025133310"><strong id="mrs_01_0983__b1298725193313">ANALYZE TABLE [db_name.]tablename [PARTITION(partcol1[=val1], partcol2[=val2], ...)]</strong></p>
|
|
<p id="mrs_01_0983__p92989251339"><strong id="mrs_01_0983__b18298625173313">COMPUTE STATISTICS</strong></p>
|
|
<p id="mrs_01_0983__p112981725123316"><strong id="mrs_01_0983__b32982025153313">[FOR COLUMNS]</strong></p>
|
|
<p id="mrs_01_0983__p729892520334"><strong id="mrs_01_0983__b172982025143313">[NOSCAN];</strong></p>
|
|
<div class="note" id="mrs_01_0983__note16298152523312"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0983__ul162982257333"><li id="mrs_01_0983__li12298825193316">When <strong id="mrs_01_0983__b1915152132218">FOR COLUMNS</strong> is specified, column-level statistics are collected.</li><li id="mrs_01_0983__li929882513332">When NOSCAN is specified, statistics about the file size and number of files will be collected, but specific files will not be scanned.</li></ul>
|
|
</div></div>
|
|
<p id="mrs_01_0983__p129810252335">For example:</p>
|
|
<p id="mrs_01_0983__p182981425113315"><strong id="mrs_01_0983__b6298525113311">analyze table table_name compute statistics;</strong></p>
|
|
<p id="mrs_01_0983__p14298152593317"><strong id="mrs_01_0983__b829812514339">analyze table table_name compute statistics for columns;</strong></p>
|
|
</p></li><li id="mrs_01_0983__li16298025133311"><span>Configure the automatic statistics collection function of Hive. After the function is enabled, new statistics will be collected only when you insert data by running the <strong id="mrs_01_0983__b158345122419">insert overwrite/into</strong> command.</span><p><ul id="mrs_01_0983__ul13299192513310"><li id="mrs_01_0983__li729972573317">Run the following commands on the Hive client to enable the statistics collection function temporarily:<p id="mrs_01_0983__p92991025133311"><a name="mrs_01_0983__li729972573317"></a><a name="li729972573317"></a><strong id="mrs_01_0983__en-us_topic_0116526959_b803579115218">set hive.stats.autogather = true;</strong> enables the automatic collection of table/partition-level statistics.</p>
|
|
<p id="mrs_01_0983__p02999253335"><strong id="mrs_01_0983__en-us_topic_0116526959_b293848815218">set hive.stats.column.autogather = true;</strong> enables the automatic collection of column-level statistics.</p>
|
|
<div class="note" id="mrs_01_0983__note8299825163314"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0983__ul16299172543312"><li id="mrs_01_0983__li1729942553314">The column-level statistics collection does not support complex data types, such as Map and Struct. </li><li id="mrs_01_0983__li829972515332">The automatic table-level statistics collection does not support Hive on HBase tables.</li></ul>
|
|
</div></div>
|
|
</li></ul>
|
|
<ul id="mrs_01_0983__ul7299325193310"><li id="mrs_01_0983__li9299122516330">On the Manager UI, search for the <span class="parmname" id="mrs_01_0983__pf4004df6eaac465599b2f6fc8bd8da21"><b>hive.stats.autogather</b></span> and <span class="parmname" id="mrs_01_0983__p0a4a4c0efd4a47e1975b6998556c3f67"><b>hive.stats.column.autogather</b></span> parameters in the service configuration of Hive, and select <strong id="mrs_01_0983__b1832475022617">true</strong> to enable the collection function permanently.</li></ul>
|
|
</p></li><li id="mrs_01_0983__li183005258337"><span>Run the following command to view statistics:</span><p><p id="mrs_01_0983__p6300122518338"><strong id="mrs_01_0983__b1430012543311">DESCRIBE FORMATTED table_name[.column_name] PARTITION partition_spec;</strong></p>
|
|
<p id="mrs_01_0983__p163005259335">For example:</p>
|
|
<p id="mrs_01_0983__p730042593311"><strong id="mrs_01_0983__b330010259335">desc formatted table_name;</strong></p>
|
|
<p id="mrs_01_0983__p11300825153311"><strong id="mrs_01_0983__b1230052517338">desc formatted table_name id;</strong></p>
|
|
<p id="mrs_01_0983__p193001125163311"><strong id="mrs_01_0983__b20300625153316">desc formatted table_name partition(time='2016-05-27');</strong></p>
|
|
<div class="note" id="mrs_01_0983__note103001325123315"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0983__p12300102517339">Partition tables only support partition-level statistics collection, so you must specify partitions to query statistics for partition tables.</p>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0977.html">Hive Performance Tuning</a></div>
|
|
</div>
|
|
</div>
|
|
|