vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/en-us_topic_0000001621263317.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

130 lines
21 KiB
HTML
Raw Blame History

<a name="EN-US_TOPIC_0000001621263317"></a><a name="EN-US_TOPIC_0000001621263317"></a>
<h1 class="topictitle1">Specifying the Lifecycle of a Table When Creating the Table</h1>
<div id="body0000001621263317"><div class="section" id="EN-US_TOPIC_0000001621263317__section38321514165"><h4 class="sectiontitle">Function</h4><p id="EN-US_TOPIC_0000001621263317__p6931194918170">DLI provides table lifecycle management to allow you to specify the lifecycle of a table when creating the table. DLI determines whether to reclaim a table based on the table's last modification time and its lifecycle. By setting the lifecycle of a table, you can better manage a large number of tables, automatically delete data tables that are no longer used for a long time, and simplify the process of reclaiming data tables. Additionally, data restoration settings are supported to prevent data loss caused by misoperations.</p>
</div>
<div class="section" id="EN-US_TOPIC_0000001621263317__section1074415373234"><h4 class="sectiontitle">Table Reclamation Rules</h4><ul id="EN-US_TOPIC_0000001621263317__ul7798181022419"><li id="EN-US_TOPIC_0000001621263317__li2335162719291">When creating a table, use <strong id="EN-US_TOPIC_0000001621263317__b11463192161515">TBLPROPERTIES</strong> to specify the lifecycle of the table.<ul id="EN-US_TOPIC_0000001621263317__ul1650432992915"><li id="EN-US_TOPIC_0000001621263317__li117981810182415"><strong id="EN-US_TOPIC_0000001621263317__b936018309242">Non-partitioned table</strong><p id="EN-US_TOPIC_0000001621263317__p1410873615381">If the table is not a partitioned table, the system determines whether to reclaim the table after the lifecycle time based on the last modification time of the table.</p>
</li><li id="EN-US_TOPIC_0000001621263317__li129561715132410"><strong id="EN-US_TOPIC_0000001621263317__b16376113072410">Partitioned table</strong><p id="EN-US_TOPIC_0000001621263317__p1333542473">If the table is a partitioned table, the system determines whether the partition needs to be reclaimed based on the last modification time (LAST_ACCESS_TIME) of the partition. After the last partition of a partitioned table is reclaimed, the table is not deleted.</p>
<p id="EN-US_TOPIC_0000001621263317__p41372019194">Only table-level lifecycle management is supported for partitioned tables.</p>
</li></ul>
</li><li id="EN-US_TOPIC_0000001621263317__li490811406294">Lifecycle reclamation starts at a specified time every day to scan all partitions.<p id="EN-US_TOPIC_0000001621263317__p16416555536"><a name="EN-US_TOPIC_0000001621263317__li490811406294"></a><a name="li490811406294"></a>Lifecycle reclamation starts at a specified time every day. Reclamation only occurs if the last modification time of the table data (<strong id="EN-US_TOPIC_0000001621263317__b1558011356116">LAST_ACCESS_TIME</strong>) detected when scanning complete partitions exceeds the time specified by the lifecycle.</p>
<p id="EN-US_TOPIC_0000001621263317__p145017511538">Assume that the lifecycle of a partitioned table is one day and the last modification time of the partitioned data is 15:00 on May 20, 2023. If the table is scanned before 15:00 on May 20, 2023 (less than one day), the partitions in the table will not be reclaimed. If the last data modification time (<strong id="EN-US_TOPIC_0000001621263317__b111213521113">LAST_ACCESS_TIME</strong>) of a table partition exceeds the time specified by the lifecycle during reclamation scan on May 20, 2023, the partition will be reclaimed.</p>
</li><li id="EN-US_TOPIC_0000001621263317__li2918122115338">The lifecycle function periodically reclaims tables or partitions, which are reclaimed irregularly every day depending on the level of busyness of the service. It cannot ensure that a table or partition will be reclaimed immediately after its lifecycle expires.</li><li id="EN-US_TOPIC_0000001621263317__li19244155012335">After a table is deleted, all properties of the table, including the lifecycle, will be deleted. After a table with the same name is created again, the lifecycle of the table will be determined by the new property.</li></ul>
</div>
<div class="section" id="EN-US_TOPIC_0000001621263317__section0796155182119"><h4 class="sectiontitle">Constraints and Limitations</h4><ul id="EN-US_TOPIC_0000001621263317__ul162161589267"><li id="EN-US_TOPIC_0000001621263317__li1854174611252">Before using the lifecycle function, log in to the DLI console, choose <strong id="EN-US_TOPIC_0000001621263317__b13786112215140">Global Configuration</strong> &gt; <strong id="EN-US_TOPIC_0000001621263317__b1527753091417">Service Authorization</strong>, select <strong id="EN-US_TOPIC_0000001621263317__b42312321169">Tenant Administrator(Project-level)</strong>, and click <strong id="EN-US_TOPIC_0000001621263317__b1973011491717">Update</strong> on the <strong id="EN-US_TOPIC_0000001621263317__b1330751011617">Assign Agency Permissions</strong> page.</li><li id="EN-US_TOPIC_0000001621263317__li18263433152214">The table lifecycle function currently only supports creating tables and versioning tables using Hive and Datasource syntax.</li><li id="EN-US_TOPIC_0000001621263317__li753534652018">The unit of the lifecycle is in days. The value should be a positive integer.</li><li id="EN-US_TOPIC_0000001621263317__li15403171564215">The lifecycle can be set only at the table level. The lifecycle specified for a partitioned table applies to all partitions of the table.</li><li id="EN-US_TOPIC_0000001621263317__li1373233532617">After the lifecycle is set, DLI and OBS tables will support data backup. The backup directory for OBS tables needs to be set manually. The backup directory must be in the parallel file system and in the same bucket as the original table directory. It cannot have the same directory or subdirectory name as the original table.</li></ul>
</div>
<div class="section" id="EN-US_TOPIC_0000001621263317__section182651545162119"><h4 class="sectiontitle">Syntax</h4><ul id="EN-US_TOPIC_0000001621263317__ul1942664454817"><li id="EN-US_TOPIC_0000001621263317__li942624418487"><strong id="EN-US_TOPIC_0000001621263317__b2041013431482">Creating a DLI table using the Datasource syntax</strong><pre class="screen" id="EN-US_TOPIC_0000001621263317__screen3458194912483">CREATE TABLE table_name(name string, id int)
USING parquet
TBLPROPERTIES( "dli.lifecycle.days"=1 );</pre>
</li><li id="EN-US_TOPIC_0000001621263317__li14261444104818"><strong id="EN-US_TOPIC_0000001621263317__b39882418271">Creating a DLI table using the Hive syntax</strong><pre class="screen" id="EN-US_TOPIC_0000001621263317__screen099613478817">CREATE TABLE table_name(name string, id int)
stored as parquet
TBLPROPERTIES( "dli.lifecycle.days"=1 );</pre>
</li><li id="EN-US_TOPIC_0000001621263317__li18426104484812"><strong id="EN-US_TOPIC_0000001621263317__b447671914272">Creating an OBS table using the Datasource syntax</strong><pre class="screen" id="EN-US_TOPIC_0000001621263317__screen775817129215">CREATE TABLE table_name(name string, id int)
USING parquet
OPTIONS (path "obs://dli-test/table_name")
TBLPROPERTIES( "dli.lifecycle.days"=1, "external.table.purge"='true', "dli.lifecycle.trash.dir"='obs://dli-test/Lifecycle-Trash' );</pre>
</li><li id="EN-US_TOPIC_0000001621263317__li13426244114813"><strong id="EN-US_TOPIC_0000001621263317__b15897145662710">Creating an OBS table using the Hive syntax</strong><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001621263317__screen39591246182117"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="k">table_name</span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span><span class="w"> </span>
<span class="n">STORED</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">parquet</span><span class="w"> </span>
<span class="k">LOCATION</span><span class="w"> </span><span class="s1">'obs://dli-test/table_name'</span>
<span class="n">TBLPROPERTIES</span><span class="p">(</span><span class="w"> </span><span class="ss">&quot;dli.lifecycle.days&quot;</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="ss">&quot;external.table.purge&quot;</span><span class="o">=</span><span class="s1">'true'</span><span class="p">,</span><span class="w"> </span><span class="ss">&quot;dli.lifecycle.trash.dir&quot;</span><span class="o">=</span><span class="s1">'obs://dli-test/Lifecycle-Trash'</span><span class="w"> </span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</div>
<div class="section" id="EN-US_TOPIC_0000001621263317__en-us_topic_0114776165_en-us_topic_0093946907_s15f56ad46e8f476986860b44d58d0545"><h4 class="sectiontitle">Keywords</h4><ul id="EN-US_TOPIC_0000001621263317__en-us_topic_0114776165_en-us_topic_0093946907_ul19329628111649"><li id="EN-US_TOPIC_0000001621263317__li196161910443"><strong id="EN-US_TOPIC_0000001621263317__b122667196287">TBLPROPERTIES</strong>: Table properties, which can be used to extend the lifecycle of a table.</li><li id="EN-US_TOPIC_0000001621263317__li1786492322213"><strong id="EN-US_TOPIC_0000001621263317__b13726124717281">OPTIONS</strong>: path of the new table, which is applicable to OBS tables created using the Datasource syntax.</li><li id="EN-US_TOPIC_0000001621263317__li046561319182"><strong id="EN-US_TOPIC_0000001621263317__b78321363616">LOCATION</strong>: path of the new table, which is applicable to OBS tables created using the Hive syntax.</li></ul>
</div>
<div class="section" id="EN-US_TOPIC_0000001621263317__en-us_topic_0114776170_section1130610201331"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="EN-US_TOPIC_0000001621263317__table196711128152516" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="EN-US_TOPIC_0000001621263317__row1667272852519"><th align="left" class="cellrowborder" valign="top" width="28.062806280628067%" id="mcps1.3.6.2.2.4.1.1"><p id="EN-US_TOPIC_0000001621263317__p6672172882520">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="14.811481148114813%" id="mcps1.3.6.2.2.4.1.2"><p id="EN-US_TOPIC_0000001621263317__p18672142852512">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="57.125712571257125%" id="mcps1.3.6.2.2.4.1.3"><p id="EN-US_TOPIC_0000001621263317__p1672122802510">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="EN-US_TOPIC_0000001621263317__row4672122832510"><td class="cellrowborder" valign="top" width="28.062806280628067%" headers="mcps1.3.6.2.2.4.1.1 "><p id="EN-US_TOPIC_0000001621263317__p9672182817254">table_name</p>
</td>
<td class="cellrowborder" valign="top" width="14.811481148114813%" headers="mcps1.3.6.2.2.4.1.2 "><p id="EN-US_TOPIC_0000001621263317__p5672828192512">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="57.125712571257125%" headers="mcps1.3.6.2.2.4.1.3 "><p id="EN-US_TOPIC_0000001621263317__p567213287255">Name of the table whose lifecycle needs to be set</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001621263317__row1867232820259"><td class="cellrowborder" valign="top" width="28.062806280628067%" headers="mcps1.3.6.2.2.4.1.1 "><p id="EN-US_TOPIC_0000001621263317__p17672172882514">dli.lifecycle.days</p>
</td>
<td class="cellrowborder" valign="top" width="14.811481148114813%" headers="mcps1.3.6.2.2.4.1.2 "><p id="EN-US_TOPIC_0000001621263317__p26721128132518">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="57.125712571257125%" headers="mcps1.3.6.2.2.4.1.3 "><p id="EN-US_TOPIC_0000001621263317__p1067242842513">Lifecycle duration. The value must be a positive integer, in days.</p>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001621263317__row46721428172512"><td class="cellrowborder" valign="top" width="28.062806280628067%" headers="mcps1.3.6.2.2.4.1.1 "><p id="EN-US_TOPIC_0000001621263317__p8672202810253">external.table.purge</p>
</td>
<td class="cellrowborder" valign="top" width="14.811481148114813%" headers="mcps1.3.6.2.2.4.1.2 "><p id="EN-US_TOPIC_0000001621263317__p76731928142518">No</p>
</td>
<td class="cellrowborder" valign="top" width="57.125712571257125%" headers="mcps1.3.6.2.2.4.1.3 "><p id="EN-US_TOPIC_0000001621263317__p414511389433">This parameter is available only for OBS tables.</p>
<p id="EN-US_TOPIC_0000001621263317__p117432384315">Whether to clear data in the path when deleting a table or partition. The data is not cleared by default.</p>
<p id="EN-US_TOPIC_0000001621263317__p1867652694310">When this parameter is set to <strong id="EN-US_TOPIC_0000001621263317__b12962165311374">true</strong>:</p>
<ul id="EN-US_TOPIC_0000001621263317__ul19433173215434"><li id="EN-US_TOPIC_0000001621263317__li9433153217430">After a file is deleted from a non-partitioned OBS table, the table directory is also deleted.</li><li id="EN-US_TOPIC_0000001621263317__li14331232144311">The custom partition data in the partitioned OBS table is also deleted.</li></ul>
</td>
</tr>
<tr id="EN-US_TOPIC_0000001621263317__row778364252612"><td class="cellrowborder" valign="top" width="28.062806280628067%" headers="mcps1.3.6.2.2.4.1.1 "><p id="EN-US_TOPIC_0000001621263317__p10784154215260">dli.lifecycle.trash.dir</p>
</td>
<td class="cellrowborder" valign="top" width="14.811481148114813%" headers="mcps1.3.6.2.2.4.1.2 "><p id="EN-US_TOPIC_0000001621263317__p1178424218266">No</p>
</td>
<td class="cellrowborder" valign="top" width="57.125712571257125%" headers="mcps1.3.6.2.2.4.1.3 "><p id="EN-US_TOPIC_0000001621263317__p17263105724316">This parameter is available only for OBS tables.</p>
<p id="EN-US_TOPIC_0000001621263317__p127843425262">When <strong id="EN-US_TOPIC_0000001621263317__b192621425153911">external.table.purge</strong> is set to <strong id="EN-US_TOPIC_0000001621263317__b8513183212393">true</strong>, the backup directory will be deleted. By default, backup data is deleted seven days later.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="EN-US_TOPIC_0000001621263317__section9248104511445"><h4 class="sectiontitle">Example</h4><ul id="EN-US_TOPIC_0000001621263317__ul1924884519440"><li id="EN-US_TOPIC_0000001621263317__li1424817457446"><strong id="EN-US_TOPIC_0000001621263317__b4248184584417">Create the test_datasource_lifecycle table using the Datasource syntax. The lifecycle is set to 100 days.</strong><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001621263317__screen824814512449"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_datasource_lifecycle</span><span class="p">(</span><span class="n">id</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span><span class="w"> </span>
<span class="k">USING</span><span class="w"> </span><span class="n">parquet</span><span class="w"> </span>
<span class="n">TBLPROPERTIES</span><span class="p">(</span><span class="w"> </span><span class="ss">&quot;dli.lifecycle.days&quot;</span><span class="o">=</span><span class="mi">100</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001621263317__li1324954564414"><strong id="EN-US_TOPIC_0000001621263317__b15169125314112">Create the test_hive_lifecycle table using the Hive syntax. The lifecycle is set to 100 days.</strong><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001621263317__screen1824914504410"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_hive_lifecycle</span><span class="p">(</span><span class="n">id</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span><span class="w"> </span>
<span class="n">stored</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">parquet</span><span class="w"> </span>
<span class="n">TBLPROPERTIES</span><span class="p">(</span><span class="w"> </span><span class="ss">&quot;dli.lifecycle.days&quot;</span><span class="o">=</span><span class="mi">100</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001621263317__li524915454448"><strong id="EN-US_TOPIC_0000001621263317__b129001225174320">Create the test_datasource_lifecycle_obs table using the Datasource syntax. The lifecycle is set to 100 days. When the lifecycle expires, data is deleted by default and backed up to the obs://dli-test/ directory.</strong><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001621263317__screen42495451444"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_datasource_lifecycle_obs</span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span><span class="w"> </span>
<span class="k">USING</span><span class="w"> </span><span class="n">parquet</span><span class="w"> </span>
<span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span><span class="n">path</span><span class="w"> </span><span class="ss">&quot;obs://dli-test/xxx&quot;</span><span class="p">)</span>
<span class="n">TBLPROPERTIES</span><span class="p">(</span><span class="w"> </span><span class="ss">&quot;dli.lifecycle.days&quot;</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span><span class="w"> </span><span class="ss">&quot;external.table.purge&quot;</span><span class="o">=</span><span class="s1">'true'</span><span class="p">,</span><span class="w"> </span><span class="ss">&quot;dli.lifecycle.trash.dir&quot;</span><span class="o">=</span><span class="s1">'obs://dli-test/Lifecycle-Trash'</span><span class="w"> </span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001621263317__li9249245184418"><strong id="EN-US_TOPIC_0000001621263317__b3994103214437">Create the test_hive_lifecycle_obs table using the Hive syntax. The lifecycle is set to 100 days. When the lifecycle expires, data is deleted by default and backed up to the obs://dli-test/ directory.</strong><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001621263317__screen1324914519442"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_hive_lifecycle_obs</span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span><span class="w"> </span>
<span class="n">STORED</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">parquet</span><span class="w"> </span>
<span class="k">LOCATION</span><span class="w"> </span><span class="s1">'obs://dli-test/xxx'</span>
<span class="n">TBLPROPERTIES</span><span class="p">(</span><span class="w"> </span><span class="ss">&quot;dli.lifecycle.days&quot;</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span><span class="w"> </span><span class="ss">&quot;external.table.purge&quot;</span><span class="o">=</span><span class="s1">'true'</span><span class="p">,</span><span class="w"> </span><span class="ss">&quot;dli.lifecycle.trash.dir&quot;</span><span class="o">=</span><span class="s1">'obs://dli-test/Lifecycle-Trash'</span><span class="w"> </span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="en-us_topic_0000001571023676.html">Table Lifecycle Management</a></div>
</div>
</div>
View Git Blame Copy Permalink