forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
25 lines
4.3 KiB
HTML
25 lines
4.3 KiB
HTML
<a name="mrs_01_24121"></a><a name="mrs_01_24121"></a>
|
|
|
|
<h1 class="topictitle1">Hive Supporting ZSTD Compression Formats</h1>
|
|
<div id="body32001227"><p id="mrs_01_24121__en-us_topic_0000001219350847_p8060118">Zstandard (ZSTD) is an open-source lossless data compression algorithm. Its compression performance and compression ratio are better than those of other compression algorithms supported by Hadoop. Hive with this feature supports tables in ZSTD compression formats. The ZSTD compression formats supported by Hive include ORC, RCFile, TextFile, JsonFile, Parquet, Squence, and CSV.</p>
|
|
<p id="mrs_01_24121__en-us_topic_0000001219350847_p6243151312567">You can create a table in ZSTD compression format as follows:</p>
|
|
<ul id="mrs_01_24121__en-us_topic_0000001219350847_ul646017085713"><li id="mrs_01_24121__en-us_topic_0000001219350847_li44617095710">To create a table in ORC format, specify <strong id="mrs_01_24121__en-us_topic_0000001219350847_b1576628185710">TBLPROPERTIES("orc.compress"="zstd")</strong>.<p id="mrs_01_24121__en-us_topic_0000001219350847_p03601326115610"><strong id="mrs_01_24121__en-us_topic_0000001219350847_b102210118579">create table tab_1(...) stored as orc TBLPROPERTIES("orc.compress"="zstd");</strong></p>
|
|
</li><li id="mrs_01_24121__en-us_topic_0000001219350847_li1976010228572">To create a table in Parquet format, specify <strong id="mrs_01_24121__en-us_topic_0000001219350847_b1266081835814">TBLPROPERTIES("parquet.compression"="zstd")</strong>.<p id="mrs_01_24121__en-us_topic_0000001219350847_p1361122675613"><strong id="mrs_01_24121__en-us_topic_0000001219350847_b158243011570">create table tab_2(...) stored as parquet TBLPROPERTIES("parquet.compression"="zstd");</strong></p>
|
|
</li><li id="mrs_01_24121__en-us_topic_0000001219350847_li75202041185714">To create a table in other formats or common formats, run the following commands to set the <strong id="mrs_01_24121__en-us_topic_0000001219350847_b1816225417473">mapreduce.map.output.compress.codec</strong> and <strong id="mrs_01_24121__en-us_topic_0000001219350847_b16777174974710">mapreduce.output.fileoutputformat.compress.codec</strong> parameters to <strong id="mrs_01_24121__en-us_topic_0000001219350847_b285615020016">org.apache.hadoop.io.compress.ZStandardCode</strong>.<p id="mrs_01_24121__en-us_topic_0000001219350847_p163619265563"><strong id="mrs_01_24121__en-us_topic_0000001219350847_b1153501010588">set hive.exec.compress.output=true;</strong></p>
|
|
<p id="mrs_01_24121__en-us_topic_0000001219350847_p6361126125618"><strong id="mrs_01_24121__en-us_topic_0000001219350847_b1953671075820">set mapreduce.map.output.compress=true;</strong></p>
|
|
<p id="mrs_01_24121__en-us_topic_0000001219350847_p8361192645611"><strong id="mrs_01_24121__en-us_topic_0000001219350847_b2536191019583">set mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.ZStandardCodec;</strong></p>
|
|
<p id="mrs_01_24121__en-us_topic_0000001219350847_p1836182610563"><strong id="mrs_01_24121__en-us_topic_0000001219350847_b12537181045813">set mapreduce.output.fileoutputformat.compress=true;</strong></p>
|
|
<p id="mrs_01_24121__en-us_topic_0000001219350847_p9361172620568"><strong id="mrs_01_24121__en-us_topic_0000001219350847_b3538210205816">set mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.ZStandardCodec;</strong></p>
|
|
<p id="mrs_01_24121__en-us_topic_0000001219350847_p17361826145620"><strong id="mrs_01_24121__en-us_topic_0000001219350847_b55385107587">set hive.exec.compress.intermediate=true;</strong></p>
|
|
<p id="mrs_01_24121__en-us_topic_0000001219350847_p936112616564"><strong id="mrs_01_24121__en-us_topic_0000001219350847_b17539181015817">create table tab_3(...) stored as textfile;</strong></p>
|
|
</li></ul>
|
|
<div class="note" id="mrs_01_24121__en-us_topic_0000001219350847_note1033293241911"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24121__en-us_topic_0000001219350847_p262717297915">The SQL operations on a table compressed using ZSTD are the same as those on a common compressed table. A table compressed using ZSTD supports addition, deletion, query, and aggregation SQL operations.</p>
|
|
</div></div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0581.html">Using Hive</a></div>
|
|
</div>
|
|
</div>
|
|
|