forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
26 lines
6.5 KiB
HTML
26 lines
6.5 KiB
HTML
<a name="mrs_01_0978"></a><a name="mrs_01_0978"></a>
|
|
|
|
<h1 class="topictitle1">Creating Table Partitions</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_0978__en-us_topic_0000001173471358_s6f3b0ae5e79f4012bfdecd42e3888ad3"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_p33761356">During the Select query, Hive generally scans the entire table, which is time-consuming. To improve query efficiency, create table partitions based on service requirements and query dimensions.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0978__en-us_topic_0000001173471358_s5123f1749faf4f1884fd2637a7e79e9d"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_0978__en-us_topic_0000001173471358_ol67755492118"><li id="mrs_01_0978__en-us_topic_0000001173471358_li4153142615134"><span>Log in to the node where the Hive client has been installed as user <strong id="mrs_01_0978__en-us_topic_0000001173471358_b61542023103816">root</strong>.</span></li><li id="mrs_01_0978__en-us_topic_0000001173471358_li1877512412216"><span>Run the following command to go to the client installation directory, for example, <strong id="mrs_01_0978__en-us_topic_0000001173471358_b232414713717">/opt</strong><strong id="mrs_01_0978__en-us_topic_0000001173471358_b12324373716"></strong><strong id="mrs_01_0978__en-us_topic_0000001173471358_b153241574712">/client</strong>.</span><p><p class="litext" id="mrs_01_0978__en-us_topic_0000001173471358_p206251581217"><strong id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_b10523083">cd /opt/client</strong></p>
|
|
</p></li><li id="mrs_01_0978__en-us_topic_0000001173471358_li1577584162114"><span>Run the <strong id="mrs_01_0978__en-us_topic_0000001173471358_b19781449175211">source bigdata_env</strong> command to configure environment variables for the client.</span></li><li id="mrs_01_0978__en-us_topic_0000001173471358_li9775174142120"><span>Run the following command on the client for login:</span><p><p class="litext" id="mrs_01_0978__en-us_topic_0000001173471358_p1531231042117"><strong id="mrs_01_0978__en-us_topic_0000001173471358_a4952e7a738294f70b6f0d16a622aa4d3">kinit</strong> <em id="mrs_01_0978__en-us_topic_0000001173471358_abf4cdc2ed34c4e39b18842817e571752">Username</em></p>
|
|
</p></li><li id="mrs_01_0978__en-us_topic_0000001173471358_li1477518472118"><span>Run the following command to log in to the client tool:</span><p><p class="litext" id="mrs_01_0978__en-us_topic_0000001173471358_p5110181211213"><strong id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_b48832851">beeline</strong></p>
|
|
</p></li><li id="mrs_01_0978__en-us_topic_0000001173471358_li207768492120"><span>Select the static or dynamic partition.</span><p><ul id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_ul31450711"><li id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_li14620943">Static partition:<p id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_p64479624"><a name="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_li14620943"></a><a name="en-us_topic_0000001173471358_en-us_topic_0116526952_li14620943"></a>Manually enter a partition name, and use the keyword <strong id="mrs_01_0978__en-us_topic_0000001173471358_b51419502533">PARTITIONED BY</strong> to specify partition column name and data type when creating a table. During application development, use the <strong id="mrs_01_0978__en-us_topic_0000001173471358_b1638164512548">ALTER TABLE ADD PARTITION</strong> statement to add a partition and use the <strong id="mrs_01_0978__en-us_topic_0000001173471358_b15890115115412">LOAD DATA INTO PARTITION</strong> statement to load data to the partition, which supports only static partitions.</p>
|
|
</li><li id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_li25747420">Dynamic partition: Use a query command to insert results to a partition of a table. The partition can be a dynamic partition.<p id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_p30400195"><a name="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_li25747420"></a><a name="en-us_topic_0000001173471358_en-us_topic_0116526952_li25747420"></a>The dynamic partition can be enabled on the client tool by running the following command:</p>
|
|
<p id="mrs_01_0978__en-us_topic_0000001173471358_p11694950171614"><strong id="mrs_01_0978__en-us_topic_0000001173471358_b753426161711">set hive.exec.dynamic.partition=true</strong><strong id="mrs_01_0978__en-us_topic_0000001173471358_b77229510115">;</strong></p>
|
|
<p id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_p46496694">The default mode of the dynamic partition is strict. That is, at least a column must be specified as a static partition, under which dynamic sub-partitions can be created. You can run the following command to enable a completely dynamic partition:</p>
|
|
<p id="mrs_01_0978__en-us_topic_0000001173471358_p14171917171719"><strong id="mrs_01_0978__en-us_topic_0000001173471358_b16452142371716">set hive.exec.dynamic.partition.mode=nonstrict</strong><strong id="mrs_01_0978__en-us_topic_0000001173471358_b162313916114">;</strong></p>
|
|
</li></ul>
|
|
<div class="note" id="mrs_01_0978__en-us_topic_0000001173471358_en-us_topic_0116526952_note8135907"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0978__en-us_topic_0000001173471358_ul14713143051711"><li class="text" id="mrs_01_0978__en-us_topic_0000001173471358_li271363031719">The dynamic partition may cause a DML statement to create a large number of partitions and new mapping folders, which deteriorates system performance.</li><li id="mrs_01_0978__en-us_topic_0000001173471358_li1371314302172">If there are a large number of files, it takes a long time to run a SQL statement. You can run the <span class="parmname" id="mrs_01_0978__en-us_topic_0000001173471358_pd162c1f0ab4545d5b74a7b9f916436b0"><b>set mapreduce.input.fileinputformat.list-status.num-threads = 100;</b></span> statement before running a SQL statement to shorten the time. The parameter <span class="parmname" id="mrs_01_0978__en-us_topic_0000001173471358_p3050bfb0b39d4afab7d1b21de2fbe1fb"><b>mapreduce.input.fileinputformat.list-status.num-threads</b></span> can be set only after being added to the Hive whitelist.</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0977.html">Hive Performance Tuning</a></div>
|
|
</div>
|
|
</div>
|
|
|