vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dws/dev/dws_04_0185.html
Lu, Huayi a24ca60074 DWS DEVELOPER 811 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Lu, Huayi <luhuayi@huawei.com>
Co-committed-by: Lu, Huayi <luhuayi@huawei.com>
2023-01-19 13:37:49 +00:00

100 lines
23 KiB
HTML
Raw Blame History

<a name="EN-US_TOPIC_0000001146480559"></a><a name="EN-US_TOPIC_0000001146480559"></a>
<h1 class="topictitle1">Creating an OBS Foreign Table</h1>
<div id="body8662426"><div class="section" id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_section18973125145233"><h4 class="sectiontitle">Procedure</h4><ol id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_ol35241321145117"><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li4896932145117"><span>Set <strong id="EN-US_TOPIC_0000001146480559__b300637721205131">location</strong> of the foreign table based on the path planned in <a href="dws_04_0184.html">Uploading Data to OBS</a>.</span></li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li1959191381510"><span>Obtain the access keys (AK and SK) to access OBS. <span id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_ph223815664216">To obtain access keys, log in to the management console, click the username in the upper right corner, and select <strong id="EN-US_TOPIC_0000001146480559__b1961112435465">My Credentials</strong> from the menu. Then choose <strong id="EN-US_TOPIC_0000001146480559__b155812120479">Access Keys</strong> in the navigation tree on the left. On the <strong id="EN-US_TOPIC_0000001146480559__b0947181719476">Access Keys</strong> page, you can view the existing AKs or click <strong id="EN-US_TOPIC_0000001146480559__b1266915199485">Add Access Key</strong> to create an AK/SK pair.</span></span></li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li17114191619574"><span>Set data format parameters for the foreign table based on the formats of data to be imported. You need to collect the following source data information:</span><p><ul id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_ul139051349226"><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li48563411836"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b84235270620558">format</strong>: format of the source data file in the foreign table. OBS foreign tables support CSV and TEXT formats. The default value is <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b16236358131015">TEXT</strong>.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li159055491925"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b9336195941415">header</strong>: Whether the data file contains a table header. Only CSV files can have headers.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li16516134411511"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b18860125319121">delimiter</strong>: Delimiter specified to separate data fields in a file. If no delimiter is specified, the default one will be used.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li792155010295">For more parameters used for foreign tables, see data format parameters.</li></ul>
</p></li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li157481616912"><span>Plan the error tolerance of parallel import to specify how errors are handled during the import.</span><p><ul id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_ul9238152319136"><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li86291732056"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b139023071615">fill_missing_fields</strong>: When the last column in a row of the source data file is empty, this parameter specifies whether to report an error or set this field in the row to <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b5911304164">NULL</strong>.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li36291632355"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b191441935131616">ignore_extra_data</strong>: When the number of columns in the source data file is greater than that specified in the foreign table, this parameter specifies whether to report an error or ignore the extra columns.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li862913322057"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b610194251610">per node reject limit</strong>: This parameter specifies the number of data format errors allowed on each DN. If the number of errors recorded in the error table on a DN exceeds the specified value, the import fails and an error message will be reported. This parameter is optional.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li76292321855"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b15362551141616">compatible_illegal_chars</strong>: When an illegal character is encountered, this parameter specifies whether to import an error, or convert it and proceed with the import.<p id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0059778310_p13442883102042">The following describes the rules for converting an invalid character:</p>
<ul id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_ul693854213329"><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li1565715409325"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b948934019179">\0</strong> is converted to a space.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li1965714073218">Other invalid characters are converted to question marks (?).</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li2673124063212">If <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b2616124414179">NULL</strong>, <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b1261774416172">DELIMITER</strong>, <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b861918444173">QUOTE</strong>, or <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b19620744201718">ESCAPE</strong> is also set to a space or question mark, an error message such as "illegal chars conversion may confuse COPY escape 0x20" is displayed, prompting you to modify parameter settings that may cause import errors.</li></ul>
</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0059778310_li2281964215045"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b1631154201810">error_table_name</strong>: This parameter specifies the name of the table that records data format errors. After the parallel import, you can query this table for error details.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_li17679880154">For details about the parameters, see error tolerance parameters.</li></ul>
</p></li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li63392893151251"><span>Create an OBS table based on the parameter settings in the preceding steps. For details about how to create a foreign table, see CREATE FOREIGN TABLE (for GDS Import and Export).</span></li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_section1666367315324"><h4 class="sectiontitle">Example</h4><p id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_p758024815324">Create a foreign table in the <span id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_ph14619143164215">GaussDB(DWS)</span> database. Parameters are described as follows:</p>
<ul id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_ul4387289615324"><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li6015991215324"><strong id="EN-US_TOPIC_0000001146480559__b8797814117">Data format parameter access keys (AK and SK)</strong><ul id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_ul4111471015324"><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li3448807815324">Set <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b13487416441">access_key</strong> to the AK you have obtained.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li4207096515324">Set <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b4162118440">secret_access_key</strong> to the SK you have obtained.</li></ul>
<div class="note" id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_note5230496715324"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_a6cfc61fff8214256b92f9a530bfe162d">The values of <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b842352706181715">access_key</strong> and <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b842352706181721">secret_access_key</strong> are examples only.</p>
</div></div>
</li></ul>
<ul id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_ul884395015324"><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li1248669315324"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b192931013416">Set data format parameters.</strong><ul id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_ul478922215324"><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li4310300215324">Set <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b1092619323817">format</strong> to <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b89263320386">CSV</strong>.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li168228715324">Set <strong id="EN-US_TOPIC_0000001146480559__b13670630689528">encoding</strong> to <strong id="EN-US_TOPIC_0000001146480559__b4763077719528">UTF-8</strong>.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li204754415324">Configure <strong id="EN-US_TOPIC_0000001146480559__b1465438829528">encrypt</strong>. Its default value is <strong id="EN-US_TOPIC_0000001146480559__b9811766889528">off</strong>.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li3163337415324">Set <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_en-us_topic_0067789874_en-us_topic_0064722285_b84235270616570">delimiter</strong> to <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_en-us_topic_0067789874_en-us_topic_0064722285_b84235270616572">,</strong>.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li1216647215324">Retain the default value (double quotation marks) of <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b84235270616825">quote</strong>.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li5977450915324">Set <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b842352706144441">null</strong> (null value in a source data file) to a null string without quotation marks.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li2603673115324">Set <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b842352706144511">header</strong> (whether the exported data file contains the header row) to the default value <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b842352706144825">false</strong>. If the first row of the data file is not a header, retain the default value.<div class="note" id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_note43241561298"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_p5324106699">When exporting data from OBS, this parameter cannot be set to <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b751845051113">true</strong>. Use the default value <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b106776531218">false</strong>.</p>
</div></div>
</li></ul>
</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li3493670915324"><strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b5108145112337">Set fault-tolerant parameters for data import.</strong><ul id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_ul1130116915324"><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li3460165715324">Set <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b842352706145555">PER NODE REJECT LIMIT</strong> to <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b842352706145612">'unlimited'</strong>, indicating that all data format errors detected during data import are allowed.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li6388075015324">Set <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b842352706145756">LOG INTO</strong> to <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b84235270614586">product_info_err</strong>, indicating that the data format errors detected during data import are recorded in the <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b426610138145823">product_info_err</strong> table.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li1938289315324">If <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b106921052415130">fill_missing_fields</strong> is set to <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b842352706152217">true</strong> and the last column of a data row in a source data file is lost, the column is replaced with <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b84235270693653">NULL</strong> and no error message will be generated.</li><li id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_li4364586715324">If <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b84235270615425">ignore_extra_data</strong> is set to <strong id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001145491115_b84235270615429">true</strong> and the number of columns of the source data file is greater than that defined for the foreign table, the extra columns at the end of the row are ignored and no error message will be generated.</li></ul>
</li></ul>
<p id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_p297362615324">Based on the preceding settings, the foreign table is created using the following statements:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_screen2676263915324"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">DROP</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">product_info_ext</span><span class="p">;</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">product_info_ext</span><span class="w"></span>
<span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">product_price</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_time</span><span class="w"> </span><span class="nb">date</span><span class="w"> </span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_level</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="w"> </span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_name</span><span class="w"> </span><span class="nb">varchar</span><span class="p">(</span><span class="mi">200</span><span class="p">)</span><span class="w"> </span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_type1</span><span class="w"> </span><span class="nb">varchar</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span><span class="w"> </span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_type2</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="w"> </span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_monthly_sales_cnt</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_comment_time</span><span class="w"> </span><span class="nb">date</span><span class="w"> </span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_comment_num</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">product_comment_content</span><span class="w"> </span><span class="nb">varchar</span><span class="p">(</span><span class="mi">200</span><span class="p">)</span><span class="w"> </span>
<span class="p">)</span><span class="w"> </span>
<span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span><span class="w"> </span>
<span class="k">OPTIONS</span><span class="p">(</span><span class="w"></span>
<span class="k">LOCATION</span><span class="w"> </span><span class="s1">'obs://mybucket/input_data/product_info | obs://mybucket02/input_data/product_info'</span><span class="p">,</span><span class="w"></span>
<span class="n">FORMAT</span><span class="w"> </span><span class="s1">'CSV'</span><span class="w"> </span><span class="p">,</span><span class="w"></span>
<span class="k">DELIMITER</span><span class="w"> </span><span class="s1">','</span><span class="p">,</span><span class="w"></span>
<span class="k">encoding</span><span class="w"> </span><span class="s1">'utf8'</span><span class="p">,</span><span class="w"></span>
<span class="n">header</span><span class="w"> </span><span class="s1">'false'</span><span class="p">,</span><span class="w"></span>
<span class="n">ACCESS_KEY</span><span class="w"> </span><span class="s1">'access_key_value_to_be_replaced'</span><span class="p">,</span><span class="w"></span>
<span class="n">SECRET_ACCESS_KEY</span><span class="w"> </span><span class="s1">'secret_access_key_value_to_be_replaced'</span><span class="p">,</span><span class="w"></span>
<span class="n">fill_missing_fields</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span><span class="w"></span>
<span class="n">ignore_extra_data</span><span class="w"> </span><span class="s1">'true'</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="k">READ</span><span class="w"> </span><span class="k">ONLY</span><span class="w"> </span>
<span class="n">LOG</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">product_info_err</span><span class="w"> </span>
<span class="n">PER</span><span class="w"> </span><span class="n">NODE</span><span class="w"> </span><span class="n">REJECT</span><span class="w"> </span><span class="k">LIMIT</span><span class="w"> </span><span class="s1">'unlimited'</span><span class="p">;</span><span class="w"></span>
</pre></div></td></tr></table></div>
</div>
<p id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_p3332873115324">If the following information is displayed, the foreign table has been created:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001146480559__en-us_topic_0000001082830787_en-us_topic_0117443367_screen3152312415324"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"></span>
</pre></div></td></tr></table></div>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_04_0154.html">Importing CSV/TXT Data from the OBS</a></div>
</div>
</div>
View Git Blame Copy Permalink