vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0081.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

98 lines
16 KiB
HTML
Raw Blame History

<a name="dli_08_0081"></a><a name="dli_08_0081"></a>
<h1 class="topictitle1">Adding Partition Data (Only OBS Tables Supported)</h1>
<div id="body8662426"><div class="section" id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_sa8c6fdb072114386b7b01032f984fe52"><h4 class="sectiontitle">Function</h4><p id="dli_08_0081__p1367945754520">After an OBS partitioned table is created, no partition information is generated for the table. Partition information is generated only after you:</p>
<ul id="dli_08_0081__ul1072213452462"><li id="dli_08_0081__li672211454460">Insert data to the OBS partitioned table. After the data is inserted successfully, the partition metadata can be queried, for example, by partition columns.</li><li id="dli_08_0081__li887110105474">Copy the partition directory and data into the OBS path of the partitioned table, and run the partition adding statements described in this section to generate partition metadata. Then you can perform operations such as table query by partition columns.</li></ul>
<p id="dli_08_0081__p6091125412">The following describes how to use the <strong id="dli_08_0081__b14667112910516">ALTER TABLE</strong> statement to add a partition.</p>
</div>
<div class="section" id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_sf3ae5e78baa24dd6a4a555c4e188e824"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0081__en-us_topic_0114776175_screen1421122192712"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="p">[</span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="p">]</span>
<span class="w"> </span><span class="n">PARTITION</span><span class="w"> </span><span class="n">partition_specs1</span>
<span class="w"> </span><span class="p">[</span><span class="k">LOCATION</span><span class="w"> </span><span class="s1">'obs_path1'</span><span class="p">]</span>
<span class="w"> </span><span class="n">PARTITION</span><span class="w"> </span><span class="n">partition_specs2</span>
<span class="w"> </span><span class="p">[</span><span class="k">LOCATION</span><span class="w"> </span><span class="s1">'obs_path2'</span><span class="p">];</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_s0c845eb1e41148d79029d4f419b4fc4f"><h4 class="sectiontitle">Keywords</h4><ul id="dli_08_0081__ul85891167229"><li id="dli_08_0081__li145901169229">IF NOT EXISTS: prevents errors when partitions are repeatedly added.</li><li id="dli_08_0081__li145908169227">PARTITION: specifies a partition.</li><li id="dli_08_0081__li14590101622219">LOCATION: specifies the partition path.</li></ul>
</div>
<div class="section" id="dli_08_0081__en-us_topic_0114776188_en-us_topic_0093946905_s95436b2cebbc4c34b0e6ca8427d4f1d9"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0081__en-us_topic_0114776188_en-us_topic_0103084433_t2fa8597e0dde4884bd6412ac207ebea8" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="dli_08_0081__en-us_topic_0114776188_en-us_topic_0103084433_r16c75fd8fdbf41c79f9aee82b032351c"><th align="left" class="cellrowborder" valign="top" width="18.060000000000002%" id="mcps1.3.4.2.2.3.1.1"><p id="dli_08_0081__en-us_topic_0114776188_en-us_topic_0103084433_a9dd69379a3f54904a9c510fb817659ad">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="81.94%" id="mcps1.3.4.2.2.3.1.2"><p id="dli_08_0081__en-us_topic_0114776188_en-us_topic_0103084433_a7ede0d62539542baa7e367cf902bf2b2">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0081__row160712191411"><td class="cellrowborder" valign="top" width="18.060000000000002%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0081__p3608320149">table_name</p>
</td>
<td class="cellrowborder" valign="top" width="81.94%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0081__p160812219141">Table name</p>
</td>
</tr>
<tr id="dli_08_0081__row173647290145"><td class="cellrowborder" valign="top" width="18.060000000000002%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0081__p123641229111418">partition_specs</p>
</td>
<td class="cellrowborder" valign="top" width="81.94%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0081__p636492981417">Partition fields</p>
</td>
</tr>
<tr id="dli_08_0081__row1346183911611"><td class="cellrowborder" valign="top" width="18.060000000000002%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0081__p746133911612">obs_path</p>
</td>
<td class="cellrowborder" valign="top" width="81.94%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0081__p11461133901617">OBS path</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_sea542e2f5bc44b3597385ca653177f41"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_uf9be9f1548be4478ad1de55046a751cc"><li id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_l0e620e05220944fb92979894c176ccba">When you add a partition to a table, the table and the partition column (specified by PARTITIONED BY during table creation) must exist, and the partition to be added cannot be added repeatedly. Otherwise, an error is reported. You can use <strong id="dli_08_0081__b69978215479">IF NOT EXISTS</strong> to avoid errors if the partition does not exist.</li><li id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_l1d637f2dcd104c81bef048e0c9b11837">If tables are partitioned by multiple fields, you need to specify all partitioning fields in any sequence when adding partitions.</li><li id="dli_08_0081__en-us_topic_0114776175_li8194439172719">By default, parameters in <span class="parmname" id="dli_08_0081__en-us_topic_0114776175_parmname18832142192811"><b>partition_specs</b></span> contain parentheses (). For example: <b><span class="cmdname" id="dli_08_0081__en-us_topic_0114776175_cmdname1398714673114">PARTITION (dt='2009-09-09',city='xxx')</span></b>.</li><li id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_l0528cb4c739e4eea95cc64549ed27882">If you need to specify an OBS path when adding a partition, the OBS path must exist. Otherwise, an error occurs.</li><li id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_li12721542184816">To add multiple partitions, you need to use spaces to separate each set of <strong id="dli_08_0081__b1526911547106">LOCATION 'obs_path'</strong> in the <strong id="dli_08_0081__b3270145416105">PARTITION partition_specs</strong>. The following is an example:<p id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_p13257154474815"><b><span class="cmdname" id="dli_08_0081__en-us_topic_0114776175_cmdname3590192635213">PARTITION partition_specs LOCATION 'obs_path' PARTITION partition_specs LOCATION 'obs_path'</span></b></p>
</li><li id="dli_08_0081__li2075591192417">If the path specified in the new partition contains subdirectories (or nested subdirectories), all file types and content in the subdirectories are considered partition records.<p id="dli_08_0081__p786171920249"><a name="dli_08_0081__li2075591192417"></a><a name="li2075591192417"></a>Ensure that all file types and file content in the partition directory are the same as those in the table. Otherwise, an error is reported.</p>
<p id="dli_08_0081__p433481317244">You can set <strong id="dli_08_0081__b6233109171217">multiLevelDirEnable</strong> to <strong id="dli_08_0081__b323312981211">true</strong> in the <strong id="dli_08_0081__b6233109101219">OPTIONS</strong> statement to query the content in the subdirectory. The default value is <strong id="dli_08_0081__b13233794129">false</strong> (Note that this configuration item is a table attribute, exercise caution when performing this operation. Hive tables do not support this configuration item.)</p>
</li></ul>
</div>
<div class="section" id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_s3a0cfd3f6ee14bcea0d3dcc7f968457c"><h4 class="sectiontitle">Example</h4><ul id="dli_08_0081__en-us_topic_0114776175_en-us_topic_0093946864_u54c3db7d34564a45ae8e7b0b842c79c6"><li id="dli_08_0081__li1755910459015">The following example shows you how to add partition data when the OBS table is partitioned by a single column.<ol id="dli_08_0081__ol197591554103919"><li id="dli_08_0081__li207591754153911">Use the DataSource syntax to create an OBS table, and partition the table by column <strong id="dli_08_0081__b102123231795">external_data</strong>. The partition data is stored in <strong id="dli_08_0081__b12171143799">obs://bucketName/datapath</strong>.<pre class="screen" id="dli_08_0081__screen10910953414">create table testobstable(id varchar(128), external_data varchar(16)) using JSON OPTIONS (path 'obs://bucketName/datapath') PARTITIONED by (external_data);</pre>
</li><li id="dli_08_0081__li154708582390">Copy the partition directory to <strong id="dli_08_0081__b482955313917">obs://bucketName/datapath</strong>. In this example, copy all files in the partition column <strong id="dli_08_0081__b19718051919">external_data=22</strong> to <strong id="dli_08_0081__b235511951911">obs://bucketName/datapath</strong>.</li><li id="dli_08_0081__li727013184013">Run the following command to add partition data:<pre class="screen" id="dli_08_0081__screen1629605173913">ALTER TABLE testobstable ADD
PARTITION (external_data='22')
LOCATION 'obs://bucketName/datapath/external_data=22';</pre>
</li><li id="dli_08_0081__li880718518408">After the partition data is added successfully, you can perform operations such as data query based on the partition column.<pre class="screen" id="dli_08_0081__screen4887201513920">select * from testobstable where external_data='22';</pre>
</li></ol>
</li><li id="dli_08_0081__li5286185712916">The following example shows you how to add partition data when the OBS table is partitioned by multiple columns.<ol id="dli_08_0081__ol17232141194015"><li id="dli_08_0081__li8232131134011">Use the DataSource syntax to create an OBS table, and partition the table by columns <strong id="dli_08_0081__b4662633164317">external_data</strong> and <strong id="dli_08_0081__b596311054411">dt</strong>. The partition data is stored in <strong id="dli_08_0081__b26623339438">obs://bucketName/datapath</strong>.<div class="codecoloring" codetype="Sql" id="dli_08_0081__en-us_topic_0114776175_screen996824913275"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">testobstable</span><span class="p">(</span>
<span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">varchar</span><span class="p">(</span><span class="mi">128</span><span class="p">),</span>
<span class="w"> </span><span class="n">external_data</span><span class="w"> </span><span class="nb">varchar</span><span class="p">(</span><span class="mi">16</span><span class="p">),</span>
<span class="w"> </span><span class="n">dt</span><span class="w"> </span><span class="nb">varchar</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span>
<span class="p">)</span><span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">JSON</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span><span class="n">path</span><span class="w"> </span><span class="s1">'obs://bucketName/datapath'</span><span class="p">)</span><span class="w"> </span><span class="n">PARTITIONED</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="p">(</span><span class="n">external_data</span><span class="p">,</span><span class="w"> </span><span class="n">dt</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_08_0081__li1299151316403">Copy the partition directories to <strong id="dli_08_0081__b16629101116494">obs://bucketName/datapath</strong>. In this example, copy files in <strong id="dli_08_0081__b4512159194913">external_data=22</strong> and its subdirectory <strong id="dli_08_0081__b1632011717508">dt=2021-07-27</strong> to <strong id="dli_08_0081__b118920364508">obs://bucketName/datapath</strong>.</li><li id="dli_08_0081__li56396167408">Run the following command to add partition data:<div class="codecoloring" codetype="Sql" id="dli_08_0081__screen067231814010"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span>
<span class="w"> </span><span class="n">testobstable</span>
<span class="k">ADD</span>
<span class="w"> </span><span class="n">PARTITION</span><span class="w"> </span><span class="p">(</span><span class="n">external_data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'22'</span><span class="p">,</span><span class="w"> </span><span class="n">dt</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'2021-07-27'</span><span class="p">)</span><span class="w"> </span><span class="k">LOCATION</span><span class="w"> </span><span class="s1">'obs://bucketName/datapath/external_data=22/dt=2021-07-27'</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_08_0081__li1713233634010">After the partition data is added successfully, you can perform operations such as data query based on the partition columns.<div class="codecoloring" codetype="Sql" id="dli_08_0081__screen173441141154013"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">testobstable</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">external_data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'22'</span><span class="p">;</span>
<span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">testobstable</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">external_data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'22'</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="n">dt</span><span class="o">=</span><span class="s1">'2021-07-27'</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0080.html">Syntax for Partitioning a Table</a></div>
</div>
</div>
View Git Blame Copy Permalink