forked from docs/doc-exports
Yang, Tong
48706b7552
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
243 lines
41 KiB
HTML
243 lines
41 KiB
HTML
<a name="mrs_01_0975"></a><a name="mrs_01_0975"></a>
|
|
|
|
<h1 class="topictitle1">Hive Supporting Transactions</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_0975__en-us_topic_0000001219230607_section28311819102618"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0975__en-us_topic_0000001219230607_p9831141962617">Hive supports transactions at the table and partition levels. When the transaction mode is enabled, transaction tables can be incrementally updated, deleted, and read, implementing atomicity, isolation, consistency, and durability of operations on transaction tables.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0975__en-us_topic_0000001219230607_section88312019102612"><h4 class="sectiontitle">Introduction to Transaction Features</h4><p id="mrs_01_0975__en-us_topic_0000001219230607_p1783181942614">A transaction is a group of unitized operations. These operations are either executed together or not executed together. A transaction is an inseparable unit of work. The four basic elements of a transaction are usually called ACID features, which are as follows:</p>
|
|
<ul id="mrs_01_0975__en-us_topic_0000001219230607_ul1583112192264"><li id="mrs_01_0975__en-us_topic_0000001219230607_li9831819152616">Atomicity: A transaction is an inseparable unit of work. All operations in a transaction occur or do not occur together.</li></ul>
|
|
<ul id="mrs_01_0975__en-us_topic_0000001219230607_ul17831119202619"><li id="mrs_01_0975__en-us_topic_0000001219230607_li1083111193261">Consistency: The database integrity constraints are not damaged before and after a transaction starts.</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li148314192264">Isolation: When multiple transactions are concurrently accessed, the transactions are isolated from each other. A transaction does not affect the running of other transactions. The impacts between transactions are as follows: dirty read, non-repeatable read, phantom read, and lost update.</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li98311919122615">Durability: After a transaction is complete, changes made by the transaction lock to the database are permanently stored in the database.</li></ul>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p6831121922610">Characteristics of transaction execution:</p>
|
|
<ul id="mrs_01_0975__en-us_topic_0000001219230607_ul108311119152612"><li id="mrs_01_0975__en-us_topic_0000001219230607_li8831219102617">A statement can be written to multiple partitions or tables. If the operation fails, the user cannot see partial write or insert. Even if data is frequently changed, operations can still be quickly performed.</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li13831161972617">Hive can automatically compress ACID transaction files without affecting concurrent queries. When querying many small partition files, automatic compression can improve query performance and metadata occupation.</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li11831101914269">Read semantics include snapshot isolation. When the read operation starts, the Hive data warehouse is logically locked. The read operation is not affected by any changes that occur during the operation.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_0975__en-us_topic_0000001219230607_section6831141911260"><h4 class="sectiontitle">Lock Mechanism</h4><p id="mrs_01_0975__en-us_topic_0000001219230607_p16832111911268">Transactions implement the ACID feature through the following two aspects:</p>
|
|
<ul id="mrs_01_0975__en-us_topic_0000001219230607_ul4832619152620"><li id="mrs_01_0975__en-us_topic_0000001219230607_li148329197266">Write-ahead logging ensures atomicity and durability.</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li183231932617">Locking ensures isolation.</li></ul>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0975__en-us_topic_0000001219230607_table4832819132613" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_0975__en-us_topic_0000001219230607_row1183212196267"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.3.4.1.3.1.1"><p id="mrs_01_0975__en-us_topic_0000001219230607_p18832101902614">Operation</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="70%" id="mcps1.3.3.4.1.3.1.2"><p id="mrs_01_0975__en-us_topic_0000001219230607_p15832181912263">Type of Held Locks</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0975__en-us_topic_0000001219230607_row12832121915265"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.4.1.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p2832111916266">Insert overwrite</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.4.1.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p8832161911267">If <strong id="mrs_01_0975__en-us_topic_0000001219230607_b1620013141985">hive.txn.xlock.iow</strong> is set to <strong id="mrs_01_0975__en-us_topic_0000001219230607_b5848125988">true</strong>, the exclusive lock is held. If <strong id="mrs_01_0975__en-us_topic_0000001219230607_b14230037980">hive.txn.xlock.iow</strong> is set to <strong id="mrs_01_0975__en-us_topic_0000001219230607_b6792114514818">false</strong>, the semi-shared lock is held.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__en-us_topic_0000001219230607_row158326198269"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.4.1.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p1883231962616">Insert</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.4.1.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p12832191916266">Shared lock. When performing this operation, you can perform read and write operations on the current table or partition.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__en-us_topic_0000001219230607_row383251918260"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.4.1.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p383210194260">Update/delete</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.4.1.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p2832121942613">Semi-shared lock. When this operation is performed, an operation of holding a shared lock can be performed, but an operation of holding an exclusive lock or a semi-shared lock cannot be performed.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__en-us_topic_0000001219230607_row1483221917267"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.4.1.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p9832191913263">Drop</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.4.1.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p148320199261">Exclusive lock. You cannot perform any other operations on the current table or partition when performing this operation.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_0975__en-us_topic_0000001219230607_note083212191262"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0975__en-us_topic_0000001219230607_p1483301917263">If a conflict caused by the lock mechanism exists in the write operation, the operation that preferentially holds the lock succeeds, and other operations fail.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_0975__en-us_topic_0000001219230607_section19833161942612"><h4 class="sectiontitle">Procedure</h4><p id="mrs_01_0975__en-us_topic_0000001219230607_p128337197268"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b1483361910263">Starting a Transaction</strong></p>
|
|
</div>
|
|
<ol id="mrs_01_0975__en-us_topic_0000001219230607_ol1583351916261"><li id="mrs_01_0975__en-us_topic_0000001219230607_li1783391922613"><span>Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>. Choose <strong id="mrs_01_0975__en-us_topic_0000001219230607_b599513012611">Cluster</strong> > <em id="mrs_01_0975__en-us_topic_0000001219230607_i106511161877">Name of the desired cluster</em> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b1629101314711">Services</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b131378151714">Hive</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b74611217575">Configurations</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b749711191278">All Configurations</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b824017221720">MetaStore(Role)</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b118948246720">Transaction</strong>.</span></li><li id="mrs_01_0975__en-us_topic_0000001219230607_li4833151919266"><span>Set <strong id="mrs_01_0975__en-us_topic_0000001219230607_b1764813217715">metastore.compactor.initiator.on</strong> to <strong id="mrs_01_0975__en-us_topic_0000001219230607_b197991834976">true</strong>.</span></li><li id="mrs_01_0975__en-us_topic_0000001219230607_li483331912269"><span>Set <strong id="mrs_01_0975__en-us_topic_0000001219230607_b832014016710">metastore.compactor.worker.threads</strong> to a positive integer.</span><p><div class="note" id="mrs_01_0975__en-us_topic_0000001219230607_note11833121982613"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0975__en-us_topic_0000001219230607_p483371913269"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b9192127399">metastore.compactor.worker.threads</strong>: Specifies the number of working threads for running the compression program on MetaStore. Set this parameter based on the actual requirements. If the value is too small, the transaction compression task is executed slowly. If the value is too large, the MetaStore execution performance deteriorates.</p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_0975__en-us_topic_0000001219230607_li8833101932618"><span>Log in to the Hive client and run the following command to enable the following parameters. For details, see <a href="mrs_01_0952.html">Using a Hive Client</a>.</span><p><p id="mrs_01_0975__en-us_topic_0000001219230607_p13833151915265"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b15834319202614">set hive.support.concurrency=true;</strong></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p11834131911265"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b1183481912619">set hive.exec.dynamic.partition.mode=nonstrict;</strong></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p16834121982614"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b48341819132615">set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;</strong></p>
|
|
</p></li></ol>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p48341719162617"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b1063075716105">Create a transaction table.</strong></p>
|
|
<ol start="5" id="mrs_01_0975__en-us_topic_0000001219230607_ol1183491962620"><li id="mrs_01_0975__en-us_topic_0000001219230607_li983420198262"><span>Run the following command to create a transaction table:</span><p><p id="mrs_01_0975__en-us_topic_0000001219230607_p1883421914267"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b78345199269">CREATE TABLE [IF NOT EXISTS] </strong><em id="mrs_01_0975__en-us_topic_0000001219230607_i13834191932615">[db_name.]table_name</em><strong id="mrs_01_0975__en-us_topic_0000001219230607_b78344191261"> (</strong><em id="mrs_01_0975__en-us_topic_0000001219230607_i20834919132614">col_name data_type </em><em id="mrs_01_0975__en-us_topic_0000001219230607_i283401962617">[COMMENT col_comment]</em><em id="mrs_01_0975__en-us_topic_0000001219230607_i1283411942620">, ...</em><strong id="mrs_01_0975__en-us_topic_0000001219230607_b1383414192266">) [ROW FORMAT row_format] STORED AS orc ...... TBLPROPERTIES ('transactional'='true'[,'groupId'='group1' ... ] );</strong></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p683441914267">For example:</p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p383431915261"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b98341819172615">CREATE TABLE acidTbl (a int, b int) STORED AS ORC TBLPROPERTIES ('transactional'='true');</strong></p>
|
|
<div class="note" id="mrs_01_0975__en-us_topic_0000001219230607_note1383481922612"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0975__en-us_topic_0000001219230607_ul108341019142618"><li id="mrs_01_0975__en-us_topic_0000001219230607_li19834131982616">Currently, the transactions support only the ORC format.</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li1983413194260">External tables are not supported.</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li148345191260">Sorted tables are not supported.</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li148340196265">To create a transaction table, you must add the table attribute <strong id="mrs_01_0975__en-us_topic_0000001219230607_b1369651131120">transactional'='true'</strong>.</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li1383512193261">The transaction table can be read and written only in transaction mode.</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p18835201992619"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b14568219124">Use the transaction table.</strong></p>
|
|
<ol start="6" id="mrs_01_0975__en-us_topic_0000001219230607_ol18354199261"><li id="mrs_01_0975__en-us_topic_0000001219230607_li208351319192611"><span>Run commands to use the transaction table. The following uses the <strong id="mrs_01_0975__en-us_topic_0000001219230607_b2122015121220">acidTbl</strong> table as an example:</span><p><ul id="mrs_01_0975__en-us_topic_0000001219230607_ul158354197269"><li id="mrs_01_0975__en-us_topic_0000001219230607_li14835019192616">Insert data into an existing transaction table:<p id="mrs_01_0975__en-us_topic_0000001219230607_p138351719102615"><a name="mrs_01_0975__en-us_topic_0000001219230607_li14835019192616"></a><a name="en-us_topic_0000001219230607_li14835019192616"></a><strong id="mrs_01_0975__en-us_topic_0000001219230607_b6835319142611">INSERT INTO acidTbl VALUES(1,1);</strong></p>
|
|
</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li2083591914267">Update an existing transaction table:<p id="mrs_01_0975__en-us_topic_0000001219230607_p38351519172614"><a name="mrs_01_0975__en-us_topic_0000001219230607_li2083591914267"></a><a name="en-us_topic_0000001219230607_li2083591914267"></a><strong id="mrs_01_0975__en-us_topic_0000001219230607_b683511992611">UPDATE acidTbl SET b = 10 where a = 1;</strong></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p1483511952615">The content of <strong id="mrs_01_0975__en-us_topic_0000001219230607_b127542829551345">acidTbl</strong> is changed to:</p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p48351719142615"><span><img id="mrs_01_0975__en-us_topic_0000001219230607_image5835111972616" src="en-us_image_0000001295739904.jpg"></span></p>
|
|
</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li9835201915260">Merge the old and new transaction tables:<p id="mrs_01_0975__en-us_topic_0000001219230607_p1835201952619"><a name="mrs_01_0975__en-us_topic_0000001219230607_li9835201915260"></a><a name="en-us_topic_0000001219230607_li9835201915260"></a>The <strong id="mrs_01_0975__en-us_topic_0000001219230607_b143342516051345">acidTbl_update</strong> table contains the following data:</p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p3835111972619"><span><img id="mrs_01_0975__en-us_topic_0000001219230607_image1383513191268" src="en-us_image_0000001296059708.jpg"></span></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p2835121917262"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b883514192267">MERGE INTO acidTbl AS a</strong></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p128351319112615"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b198351419162618">USING acidTbl_update AS b ON a.a = b.a</strong></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p108351819102616"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b78351019172618">WHEN MATCHED THEN UPDATE SET b = b. b</strong></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p118351619182612"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b16835219112614">WHEN NOT MATCHED THEN INSERT VALUES (b.a, b.b);</strong></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p9835119172611">The content of <strong id="mrs_01_0975__en-us_topic_0000001219230607_b111284905251345">acidTbl</strong> is changed to:</p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p178351719132611"><span><img id="mrs_01_0975__en-us_topic_0000001219230607_image983561915260" src="en-us_image_0000001348739733.jpg"></span></p>
|
|
<div class="note" id="mrs_01_0975__en-us_topic_0000001219230607_note1147111310198"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0975__en-us_topic_0000001219230607_p4665922121912">If "Error evaluating cardinality_violation" is displayed when you run the <strong id="mrs_01_0975__en-us_topic_0000001219230607_b6976133818512">merge</strong> command, check whether duplicate connection keys exist or run the <strong id="mrs_01_0975__en-us_topic_0000001219230607_b4411115355112">set hive.merge.cardinality.check=false</strong> command to avoid this exception.</p>
|
|
</div></div>
|
|
</li></ul>
|
|
<ul id="mrs_01_0975__en-us_topic_0000001219230607_ul168355198266"><li id="mrs_01_0975__en-us_topic_0000001219230607_li8835181912268">Delete records from the transaction table.<p id="mrs_01_0975__en-us_topic_0000001219230607_p158351819182611"><a name="mrs_01_0975__en-us_topic_0000001219230607_li8835181912268"></a><a name="en-us_topic_0000001219230607_li8835181912268"></a><strong id="mrs_01_0975__en-us_topic_0000001219230607_b48351919182610">DELETE FROM acidTbl where a = 2;</strong></p>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p18836141912618"><span><img id="mrs_01_0975__en-us_topic_0000001219230607_image8836619122616" src="en-us_image_0000001349259013.jpg"></span></p>
|
|
</li></ul>
|
|
</p></li></ol>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p168361919122617"><strong id="mrs_01_0975__en-us_topic_0000001219230607_b2836119182617">Checking the Transaction Execution Status</strong></p>
|
|
<ol start="7" id="mrs_01_0975__en-us_topic_0000001219230607_ol1836181942620"><li id="mrs_01_0975__en-us_topic_0000001219230607_li38362196266"><span>Run the following command to check the transaction execution status:</span><p><ul id="mrs_01_0975__en-us_topic_0000001219230607_ul198367192268"><li id="mrs_01_0975__en-us_topic_0000001219230607_li88361619112616">Check the lock:<p id="mrs_01_0975__en-us_topic_0000001219230607_p8836119172619"><a name="mrs_01_0975__en-us_topic_0000001219230607_li88361619112616"></a><a name="en-us_topic_0000001219230607_li88361619112616"></a><strong id="mrs_01_0975__en-us_topic_0000001219230607_b583691942619">show locks;</strong></p>
|
|
</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li12836419192618">Check the compression task:<p id="mrs_01_0975__en-us_topic_0000001219230607_p178364196265"><a name="mrs_01_0975__en-us_topic_0000001219230607_li12836419192618"></a><a name="en-us_topic_0000001219230607_li12836419192618"></a><strong id="mrs_01_0975__en-us_topic_0000001219230607_b16836419132620">show compactions;</strong></p>
|
|
</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li6836141912267">Check the task execution status:<p id="mrs_01_0975__en-us_topic_0000001219230607_p1683616194263"><a name="mrs_01_0975__en-us_topic_0000001219230607_li6836141912267"></a><a name="en-us_topic_0000001219230607_li6836141912267"></a><strong id="mrs_01_0975__en-us_topic_0000001219230607_b10836181942615">show transactions;</strong></p>
|
|
</li><li id="mrs_01_0975__en-us_topic_0000001219230607_li11836219192616">Interrupt a transaction:<p id="mrs_01_0975__en-us_topic_0000001219230607_p1836319122619"><a name="mrs_01_0975__en-us_topic_0000001219230607_li11836219192616"></a><a name="en-us_topic_0000001219230607_li11836219192616"></a><strong id="mrs_01_0975__en-us_topic_0000001219230607_b19836181916267">abort transaction</strong><strong id="mrs_01_0975__en-us_topic_0000001219230607_b966618152305">s</strong> <em id="mrs_01_0975__en-us_topic_0000001219230607_i3836161912619">TransactionId</em><strong id="mrs_01_0975__en-us_topic_0000001219230607_b12836619152613">;</strong></p>
|
|
</li></ul>
|
|
</p></li></ol>
|
|
<div class="section" id="mrs_01_0975__en-us_topic_0000001219230607_section1283631922615"><h4 class="sectiontitle">Configuring the Compression Function</h4><p id="mrs_01_0975__en-us_topic_0000001219230607_p9836119192619">HDFS does not support in-place file changing. For the new content, HDFS does not provide read consistency either. To provide these features on HDFS, we follow the standard approach used in other data warehouse tools: table or partition data is stored in a set of base files, and new, updated, as well as deleted records are stored in incremental files. Each transaction creates a new set of incremental files to change the table or partition. When read, the base files and the incremental files are merged and the changes of the update or deletion are applied.</p>
|
|
</div>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p283681918266">Writing a transaction table generates some small files in HDFS. Hive provides major and minor compression policies for combining these small files.</p>
|
|
<div class="section" id="mrs_01_0975__en-us_topic_0000001219230607_section295211137263"><h4 class="sectiontitle">Procedure of Automatic Compression</h4><ol id="mrs_01_0975__en-us_topic_0000001219230607_ol15836191992616"><li id="mrs_01_0975__en-us_topic_0000001219230607_li1983601992619"><span>Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>. Choose <strong id="mrs_01_0975__en-us_topic_0000001219230607_b3651213194920">Cluster</strong> > <em id="mrs_01_0975__en-us_topic_0000001219230607_i765191334910">Name of the desired cluster</em> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b16511313114917">Services</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b565171320496">Hive</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b10651151312496">Configurations</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b8652513194918">All Configurations</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b1165231394910">MetaStore(Role)</strong> > <strong id="mrs_01_0975__en-us_topic_0000001219230607_b14652111324920">Transaction</strong>.</span></li><li id="mrs_01_0975__en-us_topic_0000001219230607_li1183691919266"><span>Set the following parameters as required:</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0975__en-us_topic_0000001219230607_table383616195262" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0975__en-us_topic_0000001219230607_row17837119182615"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.14.2.2.2.1.2.3.1.1"><p id="mrs_01_0975__en-us_topic_0000001219230607_p98371819102616">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="70%" id="mcps1.3.14.2.2.2.1.2.3.1.2"><p id="mrs_01_0975__en-us_topic_0000001219230607_p983715193263">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0975__en-us_topic_0000001219230607_row9837191912264"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.14.2.2.2.1.2.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p11837101972617">hive.compactor.check.interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.14.2.2.2.1.2.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p198371819122614">Interval of executing compression threads. Unit: second. Default value: <strong id="mrs_01_0975__en-us_topic_0000001219230607_b15714112052618">300</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__en-us_topic_0000001219230607_row88378199263"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.14.2.2.2.1.2.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p17837819112616">hive.compactor.cleaner.run.interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.14.2.2.2.1.2.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p18837161913267">Interval of executing cleaning threads. Unit: millisecond. Default value: <strong id="mrs_01_0975__en-us_topic_0000001219230607_b3165172519268">5,000</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__en-us_topic_0000001219230607_row7837419122613"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.14.2.2.2.1.2.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p13837191912620">hive.compactor.delta.num.threshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.14.2.2.2.1.2.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p78373195265">Threshold of the number of incremental files that trigger minor compression. Default value: <strong id="mrs_01_0975__en-us_topic_0000001219230607_b15491142812262">10</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__en-us_topic_0000001219230607_row9837181992611"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.14.2.2.2.1.2.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p17837121952616">hive.compactor.delta.pct.threshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.14.2.2.2.1.2.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p483719198262">Ratio threshold of the total size of incremental files (delta) that trigger Major compression to the size of base files. The value <strong id="mrs_01_0975__en-us_topic_0000001219230607_b69131252181316">0.1</strong> indicates that Major compression is triggered when the ratio of the total size of delta files to the size of base files is 10%. Default value: <strong id="mrs_01_0975__en-us_topic_0000001219230607_b10108631181416">0.1</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__en-us_topic_0000001219230607_row128372193261"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.14.2.2.2.1.2.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p983711914268">hive.compactor.max.num.delta</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.14.2.2.2.1.2.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p4837161982619">Maximum number of incremental files that the compressor will attempt to process in a single job. Default value: <strong id="mrs_01_0975__en-us_topic_0000001219230607_b201491535132615">500</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__en-us_topic_0000001219230607_row138371119112610"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.14.2.2.2.1.2.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p16838171942610">metastore.compactor.initiator.on</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.14.2.2.2.1.2.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p1383891920269">Indicates whether to run the startup program thread and cleanup program thread on the MetaStore instance. The value must be <strong id="mrs_01_0975__en-us_topic_0000001219230607_b62153573256">true</strong>. Default value: <strong id="mrs_01_0975__en-us_topic_0000001219230607_b2374958172513">false</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__en-us_topic_0000001219230607_row11838141902613"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.14.2.2.2.1.2.3.1.1 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p17838131902611">metastore.compactor.worker.threads</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.14.2.2.2.1.2.3.1.2 "><p id="mrs_01_0975__en-us_topic_0000001219230607_p128381819192615">Number of compression program work threads running on MetaStore. If this parameter is set to <strong id="mrs_01_0975__en-us_topic_0000001219230607_b678010712275">0</strong>, no compression is performed. To use a transaction, you must set this parameter to a positive number on one or more instances of the MetaStore service. Unit: second Default value: <strong id="mrs_01_0975__en-us_topic_0000001219230607_b44521507351345">0</strong></p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_0975__en-us_topic_0000001219230607_li8838111919268"><span>Log in to the Hive client and perform compression. For details, see <a href="mrs_01_0952.html">Using a Hive Client</a>.</span><p><pre class="screen" id="mrs_01_0975__en-us_topic_0000001219230607_screen19838819132619">CREATE TABLE table_name (
|
|
id int, name string
|
|
)
|
|
CLUSTERED BY (id) INTO 2 BUCKETS STORED AS ORC
|
|
TBLPROPERTIES ("transactional"="true",
|
|
"compactor.mapreduce.map.memory.mb"="2048", -- Specify the properties of a compression map job.
|
|
"compactorthreshold.hive.compactor.delta.num.threshold"="4", -- If there are more than four incremental directories, slight compression is triggered.
|
|
"compactorthreshold.hive.compactor.delta.pct.threshold"="0.5" -- If the ratio of the incremental file size to the basic file size is greater than 50%, deep compression is triggered.
|
|
);</pre>
|
|
<p id="mrs_01_0975__en-us_topic_0000001219230607_p58388199263">or</p>
|
|
<pre class="screen" id="mrs_01_0975__en-us_topic_0000001219230607_screen8838419162616">ALTER TABLE table_name COMPACT 'minor' WITH OVERWRITE TBLPROPERTIES ("compactor.mapreduce.map.memory.mb"="3072"); -- Specify the properties of a compression map job.
|
|
ALTER TABLE table_name COMPACT 'major' WITH OVERWRITE TBLPROPERTIES ("tblprops.orc.compress.size"="8192"); -- Modify any other Hive table attributes.</pre>
|
|
<div class="note" id="mrs_01_0975__en-us_topic_0000001219230607_note1583851917262"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0975__en-us_topic_0000001219230607_p483817199269">After compression, small files are not deleted immediately. After the cleaner thread performs cleaning, the files are deleted in batches.</p>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0975__section994024482916"><h4 class="sectiontitle">Manual Compression Procedure</h4><p id="mrs_01_0975__p35232321813">If you do not want the system to automatically determine when to compress a table, configure the table attribute <strong id="mrs_01_0975__b1377291684619">NO_AUTO_Compaction</strong> to disable automatic compression. After automatic compression is disabled, you can still use the <strong id="mrs_01_0975__b1469182615212">ALTER Table /Partition Compact</strong> statement to perform manual compression.</p>
|
|
<div class="note" id="mrs_01_0975__note167321254131915"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0975__p167331854201914">This operation applies only to MRS 8.2.0 and later versions.</p>
|
|
</div></div>
|
|
<ol id="mrs_01_0975__ol63651330123"><li id="mrs_01_0975__li133651530223"><span>Log in to the Hive client by referring to <a href="mrs_01_0952.html">Using a Hive Client</a> and run the following commands to disable automatic compression when creating a table:</span><p><pre class="screen" id="mrs_01_0975__screen1052416326116"><span id="mrs_01_0975__ph1672665517543">CREATE TABLE <em id="mrs_01_0975__i1498920412710">table_name</em> (</span>
|
|
<span id="mrs_01_0975__ph872745517548"> id int, name string</span>
|
|
<span id="mrs_01_0975__ph472775516541">)</span>
|
|
<span id="mrs_01_0975__ph2727165565419">CLUSTERED BY (id) INTO 2 BUCKETS STORED AS ORC</span>
|
|
<span id="mrs_01_0975__ph87271855155410">TBLPROPERTIES ("transactional"="true",</span>
|
|
<span id="mrs_01_0975__ph1572816557549"> "NO_AUTO_COMPACTION"="true"</span>
|
|
<span id="mrs_01_0975__ph2072820554544">);</span></pre>
|
|
<div class="note" id="mrs_01_0975__note86282526121"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0975__p1262945216123">You can also run the following command to disable automatic compression after a table is created:</p>
|
|
<p id="mrs_01_0975__p1799872291312"><strong id="mrs_01_0975__b637416791418">ALTER TABLE </strong><em id="mrs_01_0975__i5669169191414">table_name</em><strong id="mrs_01_0975__b1137417141413"> set TBLPROPERTIES ("NO_AUTO_COMPACTION"="true");</strong></p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_0975__li19331420877"><span>Run the following command to set the compression type of the table. <strong id="mrs_01_0975__b41592163483">compaction_type</strong> indicates the compression type, which can be <strong id="mrs_01_0975__b561817228488">minor</strong> or <strong id="mrs_01_0975__b593802316487">major</strong>.</span><p><p id="mrs_01_0975__p79412204818"><strong id="mrs_01_0975__b1119123211818">ALTER TABLE </strong><em id="mrs_01_0975__i0677113216817">table_name</em><strong id="mrs_01_0975__b131953211819"> COMPACT 'compaction_type';</strong></p>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0975__section188939117304"><h4 class="sectiontitle">Procedure for Specifying a Queue for Running a Compression Task</h4><p id="mrs_01_0975__p10485125961916">This operation applies only to MRS 8.2.0 and later versions.</p>
|
|
</div>
|
|
<ol id="mrs_01_0975__ol842617241455"><li id="mrs_01_0975__li1157469141717"><a name="mrs_01_0975__li1157469141717"></a><a name="li1157469141717"></a><span>Create a queue.</span></li><li id="mrs_01_0975__li153341928132510"><span>Log in to FusionInsight Manager and choose <strong id="mrs_01_0975__b14121330104911">Cluster</strong> > <strong id="mrs_01_0975__b141343016498">Services</strong> > <strong id="mrs_01_0975__b19413153054910">Hive</strong>. Click <strong id="mrs_01_0975__b34139309494">Configuration</strong> then <strong id="mrs_01_0975__b6414103034911">All Configurations</strong>, click <strong id="mrs_01_0975__b18414143014916">MetaStore(Role)</strong>, and select <strong id="mrs_01_0975__b14140304497">Transaction</strong>.</span></li><li id="mrs_01_0975__li466795642613"><span>Set the following parameters as required:</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0975__table14119154818163" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0975__row1711717488164"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.17.3.2.1.2.3.1.1"><p id="mrs_01_0975__p12117204819169">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="70%" id="mcps1.3.17.3.2.1.2.3.1.2"><p id="mrs_01_0975__p81171048111612">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0975__row71187488160"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.17.3.2.1.2.3.1.1 "><p id="mrs_01_0975__p811719486168">hive.compactor.job.queue</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.17.3.2.1.2.3.1.2 "><p id="mrs_01_0975__p181171487161">The name of the Hadoop queue to which the compression job is submitted, that is, the name of the queue created in <a href="#mrs_01_0975__li1157469141717">1</a>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__row1711811484163"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.17.3.2.1.2.3.1.1 "><p id="mrs_01_0975__p911813486163">hive.compactor.check.interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.17.3.2.1.2.3.1.2 "><p id="mrs_01_0975__p16118104801613">The interval for executing the compression thread, in seconds. The default value is <strong id="mrs_01_0975__b581617119546">300</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__row111816483161"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.17.3.2.1.2.3.1.1 "><p id="mrs_01_0975__p15118154811619">hive.compactor.cleaner.run.interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.17.3.2.1.2.3.1.2 "><p id="mrs_01_0975__p21186488160">The interval for executing the clearance thread, in milliseconds. The default value is <strong id="mrs_01_0975__b173261710125510">5000</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__row0118948121611"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.17.3.2.1.2.3.1.1 "><p id="mrs_01_0975__p111182483161">hive.compactor.delta.num.threshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.17.3.2.1.2.3.1.2 "><p id="mrs_01_0975__p19118648111615">The threshold of the number of incremental files that triggers minor compression. The default value is <strong id="mrs_01_0975__b1909309564">10</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__row9118114861618"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.17.3.2.1.2.3.1.1 "><p id="mrs_01_0975__p411894861615">hive.compactor.delta.pct.threshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.17.3.2.1.2.3.1.2 "><p id="mrs_01_0975__p111181348171618">The ratio threshold of the total size of incremental files (delta) that trigger major compression to the size of base files. The value <strong id="mrs_01_0975__b396326477104125">0.1</strong> indicates that major compression is triggered when the ratio of the total size of delta files to the size of base files is 10%. The default value is <strong id="mrs_01_0975__b192145015817">0.1</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__row9118848131615"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.17.3.2.1.2.3.1.1 "><p id="mrs_01_0975__p2011854881615">hive.compactor.max.num.delta</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.17.3.2.1.2.3.1.2 "><p id="mrs_01_0975__p121187487169">The maximum number of incremental files that the compressor will attempt to process in a single job. The default value is <strong id="mrs_01_0975__b191043419016">500</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__row111192048201615"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.17.3.2.1.2.3.1.1 "><p id="mrs_01_0975__p14118548161620">metastore.compactor.initiator.on</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.17.3.2.1.2.3.1.2 "><p id="mrs_01_0975__p91191948131617">Whether to run the startup program thread and cleanup program thread on the MetaStore instance. To start a transaction, set this parameter to <strong id="mrs_01_0975__b812514101917">true</strong>. The default value is <strong id="mrs_01_0975__b145866122017">false</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0975__row11119114811612"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.17.3.2.1.2.3.1.1 "><p id="mrs_01_0975__p911912485162">metastore.compactor.worker.threads</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.17.3.2.1.2.3.1.2 "><p id="mrs_01_0975__p1211954851611">The number of compression program work threads running on MetaStore. If this parameter is set to <strong id="mrs_01_0975__b101764341913">0</strong>, no compression is performed. To use a transaction, set this parameter to a positive number on one or more instances of the MetaStore service. The unit is second. The default value is <strong id="mrs_01_0975__b2130451985104125">0</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_0975__li590921517548"><span>Log in to the Hive client and perform compression. For details, see <a href="mrs_01_0952.html">Using a Hive Client</a>.</span><p><pre class="screen" id="mrs_01_0975__mrs_01_0975_en-us_topic_0000001219230607_screen19838819132619">CREATE TABLE table_name (
|
|
id int, name string
|
|
)
|
|
CLUSTERED BY (id) INTO 2 BUCKETS STORED AS ORC
|
|
TBLPROPERTIES ("transactional"="true",
|
|
"compactor.mapreduce.map.memory.mb"="2048", -- Specify the properties of a compression map job.
|
|
"compactorthreshold.hive.compactor.delta.num.threshold"="4", -- If there are more than four incremental directories, slight compression is triggered.
|
|
"compactorthreshold.hive.compactor.delta.pct.threshold"="0.5" -- If the ratio of the incremental file size to the basic file size is greater than 50%, deep compression is triggered.
|
|
);</pre>
|
|
<p id="mrs_01_0975__mrs_01_0975_en-us_topic_0000001219230607_p58388199263">or</p>
|
|
<pre class="screen" id="mrs_01_0975__mrs_01_0975_en-us_topic_0000001219230607_screen8838419162616">ALTER TABLE table_name COMPACT 'minor' WITH OVERWRITE TBLPROPERTIES ("compactor.mapreduce.map.memory.mb"="3072"); -- Specify the properties of a compression map job.
|
|
ALTER TABLE table_name COMPACT 'major' WITH OVERWRITE TBLPROPERTIES ("tblprops.orc.compress.size"="8192"); -- Modify any other Hive table attributes.</pre>
|
|
<div class="note" id="mrs_01_0975__mrs_01_0975_en-us_topic_0000001219230607_note1583851917262"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0975__mrs_01_0975_en-us_topic_0000001219230607_p483817199269">After compression, small files are not deleted immediately. After the cleaner thread performs cleaning, the files are deleted in batches.</p>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0581.html">Using Hive</a></div>
|
|
</div>
|
|
</div>
|
|
|