vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dws/dev/dws_04_0186.html
Lu, Huayi e6fa411af0 DWS DEV 830.201 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Lu, Huayi <luhuayi@huawei.com>
Co-committed-by: Lu, Huayi <luhuayi@huawei.com>
2024-05-16 07:24:04 +00:00

78 lines
14 KiB
HTML
Raw Blame History

<a name="EN-US_TOPIC_0000001188642064"></a><a name="EN-US_TOPIC_0000001188642064"></a>
<h1 class="topictitle1">Importing Data</h1>
<div id="body8662426"><div class="section" id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_section7197641152126"><h4 class="sectiontitle">Context</h4><p id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_p24104669152116">Before importing data, you are advised to optimize your design and deployment based on the following excellent practices, helping maximize system resource utilization and improving data import performance.</p>
<ul id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_ul60056964152220"><li id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_li3641772152220">In most cases, OBS data import performance is limited by concurrent network access rate. Therefore, you are advised to deploy multiple buckets on the OBS server to import data in parallel from buckets, better utilizing DN data transfer.</li></ul>
<ul id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_ul32775952152220"><li id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_li26548112152220">Similar to the single table import, ensure that the I/O performance is greater than the maximum network throughput in the concurrent import.</li></ul>
<ul id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_ul34666403152220"><li id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_li43562172152220">Set GUC parameters <a href="dws_04_0899.html#EN-US_TOPIC_0000001188642160__sd65ae1e8226f4611819e91ce8b6a35fb">raise_errors_if_no_files</a>, <a href="dws_04_0899.html#EN-US_TOPIC_0000001188642160__s9d2ce4e6e9ea4f6a8a8df3e3a7ddadd8">partition_mem_batch</a>, and <a href="dws_04_0899.html#EN-US_TOPIC_0000001188642160__s004b2931955e4e549caeb98b2f2723af">partition_max_cache_size</a>. When importing data, specify whether to distinguish between the following two cases: no records exist in the data file or the data file does not exist. You also need to specify the number of caches and the size of data buffers.</li></ul>
<ul id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_ul56515233152220"><li id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_li38875049152220">If a table has an index, the index information is incrementally updated during the import, affecting data import performance. You are advised to delete the index from the target table before the import. You can create index again after the import is complete.</li></ul>
</div>
<div class="section" id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_section56367924141427"><h4 class="sectiontitle">Procedure</h4><ol id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_ol59612413141623"><li id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_li64341459141623"><span>Create a table in the <span id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001098971160_ph1692512422530">GaussDB(DWS)</span> database to store the data imported from the OBS. For details about the syntax, see CREATE TABLE.</span><p><p id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_p18350666442">The structure of the table must be consistent with that of the fields in the source data file. That is, the number of fields and field types must be the same. In addition, the structure of the target table must be the same as that of the foreign table. The field names can be different.</p>
</p></li><li id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_en-us_topic_0067790278_l5fa55bddf3a94bca8440f9d0e8f65059"><span>(Optional) If the target table has an index, the index information is incrementally updated during the import, affecting data import performance. You are advised to delete the index from the target table before the import. You can create index again after the import is complete.</span></li><li id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_en-us_topic_0067790278_lcf943e9ba6884253b8861a700f53eae9"><span>Import data.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_en-us_topic_0067790278_sf288fa132d354881891b8c1aed886882"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="p">[</span><span class="n">Target</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">name</span><span class="p">]</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="p">[</span><span class="k">Foreign</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">name</span><span class="p">]</span>
</pre></div></td></tr></table></div>
</div>
<ul id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_en-us_topic_0067790278_u16a187757057417589b06bab40d3c00d"><li id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_en-us_topic_0067790278_l573678319b484a55b263956ebc977c6e">If information similar to the following is displayed, the data has been imported. Query the error information table to check whether any data format errors occurred. For details, see <a href="dws_04_0187.html">Handling Import Errors</a>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_en-us_topic_0067790278_s94af2c28f6a84fcaae4451dc06657b2c"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">20</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_en-us_topic_0067790278_la6b00880731648a88f9cb12fefbdb4ae">If data fails to be loaded, rectify the problem by following the instructions provided in <a href="dws_04_0187.html">Handling Import Errors</a> and try again.</li></ul>
</p></li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_section104661228154210"><h4 class="sectiontitle">Example</h4><p id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_p1258336441">For example, create a table named <strong id="EN-US_TOPIC_0000001188642064__b1391834811582">product_info</strong>.</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_screen17865118123918"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">DROP</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="k">IF</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="n">product_info</span><span class="p">;</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">product_info</span>
<span class="p">(</span>
<span class="w"> </span><span class="n">product_price</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">product_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">product_time</span><span class="w"> </span><span class="nb">date</span><span class="w"> </span><span class="p">,</span>
<span class="w"> </span><span class="n">product_level</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="w"> </span><span class="p">,</span>
<span class="w"> </span><span class="n">product_name</span><span class="w"> </span><span class="nb">varchar</span><span class="p">(</span><span class="mi">200</span><span class="p">)</span><span class="w"> </span><span class="p">,</span>
<span class="w"> </span><span class="n">product_type1</span><span class="w"> </span><span class="nb">varchar</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span><span class="w"> </span><span class="p">,</span>
<span class="w"> </span><span class="n">product_type2</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span><span class="w"> </span><span class="p">,</span>
<span class="w"> </span><span class="n">product_monthly_sales_cnt</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="p">,</span>
<span class="w"> </span><span class="n">product_comment_time</span><span class="w"> </span><span class="nb">date</span><span class="w"> </span><span class="p">,</span>
<span class="w"> </span><span class="n">product_comment_num</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="p">,</span>
<span class="w"> </span><span class="n">product_comment_content</span><span class="w"> </span><span class="nb">varchar</span><span class="p">(</span><span class="mi">200</span><span class="p">)</span><span class="w"> </span>
<span class="p">)</span><span class="w"> </span>
<span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="n">orientation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">column</span><span class="p">,</span>
<span class="n">compression</span><span class="o">=</span><span class="n">middle</span>
<span class="p">)</span><span class="w"> </span>
<span class="n">DISTRIBUTE</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">HASH</span><span class="w"> </span><span class="p">(</span><span class="n">product_id</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<p id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_p715603221517">Run the following statement to import data from the <em id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001098971160_i842352697171154">product_info</em><strong id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001098971160_b84235270694545">_ext</strong> foreign table to the <em id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001098971160_i842352697171212">product_info</em> table:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188642064__en-us_topic_0000001082831121_en-us_topic_0117443368_en-us_topic_0103485416_en-us_topic_0093152947_screen15576104710154"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">product_info</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">product_info_ext</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_04_0154.html">Importing CSV/TXT Data from the OBS</a></div>
</div>
</div>
View Git Blame Copy Permalink