vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dws/dev/dws_04_0195.html
Lu, Huayi e6fa411af0 DWS DEV 830.201 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Lu, Huayi <luhuayi@huawei.com>
Co-committed-by: Lu, Huayi <luhuayi@huawei.com>
2024-05-16 07:24:04 +00:00

66 lines
13 KiB
HTML
Raw Blame History

<a name="EN-US_TOPIC_0000001233761817"></a><a name="EN-US_TOPIC_0000001233761817"></a>
<h1 class="topictitle1">Importing Data</h1>
<div id="body8662426"><p id="EN-US_TOPIC_0000001233761817__p1240415177247">This section describes how to create tables in <span id="EN-US_TOPIC_0000001233761817__text979156286">GaussDB(DWS)</span> and import data to the tables.</p>
<p id="EN-US_TOPIC_0000001233761817__p6701165249">Before importing all the data from a table containing over 10 million records, you are advised to import some of the data, and check whether there is data skew and whether the distribution keys need to be changed (for details, see <a href="dws_04_0228.html">Checking for Data Skew</a>). Troubleshoot the data skew if any. It is costly to address data skew and change the distribution keys after a large amount of data has been imported.</p>
<div class="section" id="EN-US_TOPIC_0000001233761817__s346fe31d510b413a9e00a4da16781f38"><h4 class="sectiontitle">Prerequisites</h4><p id="EN-US_TOPIC_0000001233761817__p1397022164117">The GDS server can communicate with GaussDB(DWS).</p>
<ul id="EN-US_TOPIC_0000001233761817__ul176821714429"><li id="EN-US_TOPIC_0000001233761817__li17685176420">You need to create an ECS as the GDS server.</li><li id="EN-US_TOPIC_0000001233761817__li424084620429">The created ECS and GaussDB(DWS) must belong to the same region, VPC, and subnet.</li></ul>
</div>
<div class="section" id="EN-US_TOPIC_0000001233761817__s512ba003c1d542e4b2df804651faea79"><h4 class="sectiontitle">Procedure</h4><ol id="EN-US_TOPIC_0000001233761817__oc008fca0a9e04a75a3b630745fc4bd45"><li id="EN-US_TOPIC_0000001233761817__l479f2d5c6f314d338a453ae9caa420e1"><span>Create a table in <span id="EN-US_TOPIC_0000001233761817__text1993120538">GaussDB(DWS)</span> to store imported data. For details, see CREATE TABLE.</span></li><li id="EN-US_TOPIC_0000001233761817__lb7959c4720a5411a9837a7b7bb7436ab"><span>(Optional) If the target table has an index, the index information is incrementally updated during the import, affecting data import performance. You are advised to delete the indexes of related tables before importing data. If the data uniqueness cannot be ensured, you are not advised to delete the unique indexes. You can create indexes again after the import is complete.</span><p><ol type="a" id="EN-US_TOPIC_0000001233761817__ob3209bcc7b174692b6b904496c6a0c08"><li id="EN-US_TOPIC_0000001233761817__lf674cf8d7e7f486db034f064391874ed">Assume that the ordinary index <strong id="EN-US_TOPIC_0000001233761817__b252816531944">product_idx</strong> exists in the <strong id="EN-US_TOPIC_0000001233761817__b842352706183910">product_id</strong> column of the target table <strong id="EN-US_TOPIC_0000001233761817__b842352706114125">product_info</strong>. Delete the index from the table before importing data.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761817__sb59fbde303ea4de691ddd3f4c578ba82"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">DROP</span><span class="w"> </span><span class="k">INDEX</span><span class="w"> </span><span class="n">product_idx</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761817__l467a258c446f468da639b8dfc41e3602">After importing the data, create the <strong id="EN-US_TOPIC_0000001233761817__b842352706153235">reasons_idx</strong> index again.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761817__sd8f30e59973a48c99bf71963415536ed"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">INDEX</span><span class="w"> </span><span class="n">product_idx</span><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">product_info</span><span class="p">(</span><span class="n">product_id</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001233761817__l675e212810b14373833bb5d4fa8f0b77"><span>Import data.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761817__s3b236958a0e54d71b6c339103242e3b0"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="p">[</span><span class="n">Target</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">name</span><span class="p">]</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="p">[</span><span class="k">Foreign</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">name</span><span class="p">]</span>
</pre></div></td></tr></table></div>
</div>
<ul id="EN-US_TOPIC_0000001233761817__ub621d008a7c3474b9db9f1f68c174461"><li id="EN-US_TOPIC_0000001233761817__l77526b6a51b94c6abb3f6ab122904097">If information similar to the following is displayed, the data has been imported. Query the error information table to check whether any data format errors occurred. For details, see <a href="dws_04_0196.html">Handling Import Errors</a>.<pre class="screen" id="EN-US_TOPIC_0000001233761817__sce91049b0bc54e32919be73b597c9e39">INSERT 0 9</pre>
</li><li id="EN-US_TOPIC_0000001233761817__l37c9171730e742f695e5d369f1e695d0">If data fails to be loaded, troubleshoot the problem by following the instructions provided in <a href="dws_04_0196.html">Handling Import Errors</a> and try again.</li></ul>
<div class="note" id="EN-US_TOPIC_0000001233761817__na88400ea8d584844921871bd201b4c9e"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="EN-US_TOPIC_0000001233761817__u10315cc2ffa64d329bd93cd5e7b4654e"><li id="EN-US_TOPIC_0000001233761817__li8963324134514">If a data loading error occurs, the entire data import task will fail.</li><li id="EN-US_TOPIC_0000001233761817__l9f079f661f2a41a7b1b25660d00c715a">Compile a batch-processing task script to concurrently import data. The degree of parallelism (DOP) depends on the server resource usage. You can test-import several tables, monitor resource utilization, and increase or reduce concurrency accordingly. Common resource monitoring commands include <strong id="EN-US_TOPIC_0000001233761817__b7618185013918">top</strong> for monitoring memory and CPU usage, <strong id="EN-US_TOPIC_0000001233761817__b1461845083914">iostat</strong> for monitoring I/O usage, and <strong id="EN-US_TOPIC_0000001233761817__b76181050193918">sar</strong> for monitoring networks. For details about application cases, see <a href="dws_04_0198.html#EN-US_TOPIC_0000001233761893__section197704383292">Data Import Using Multiple Threads</a>.</li><li id="EN-US_TOPIC_0000001233761817__l8b290ba70c714c33bc9446d2ac483e8f">If possible, more GDS servers can significantly improve the data import efficiency. For details about application cases, see <a href="dws_04_0198.html#EN-US_TOPIC_0000001233761893__section2041618243291">Parallel Import from Multiple Data Servers</a>.</li><li id="EN-US_TOPIC_0000001233761817__lfd88a221d9b145f28a119bbd12ff0b8a">In a scenario where many GDS servers import data concurrently, you can increase the TCP Keepalive interval for connections between GDS servers and DNs to ensure connection stability. (The recommended interval is 5 minutes.) TCP Keepalive settings of the cluster affect its fault detection response time.</li></ul>
</div></div>
</p></li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001233761817__s1c6c5420aeb94c46b58add6ea7cdac3f"><h4 class="sectiontitle">Example:</h4><ol id="EN-US_TOPIC_0000001233761817__oc2335a96c1d14cc6ba5d839f58385795"><li id="EN-US_TOPIC_0000001233761817__lb3e7146c681745d4a9b7db068eff3b77">Create a target table named <strong id="EN-US_TOPIC_0000001233761817__b842352706181119">reasons</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761817__s3092bce97b2c4a7facac7bd5efea5f44"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">reasons</span>
<span class="p">(</span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">DISTRIBUTE</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">HASH</span><span class="w"> </span><span class="p">(</span><span class="n">r_reason_sk</span><span class="p">);</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761817__l2e78ae74fd6b45f38a038cff4a918563">You are advised to delete the indexes from the target table before the import.<div class="p" id="EN-US_TOPIC_0000001233761817__p1850920458155"><a name="EN-US_TOPIC_0000001233761817__l2e78ae74fd6b45f38a038cff4a918563"></a><a name="l2e78ae74fd6b45f38a038cff4a918563"></a>Assume that an ordinary table index <strong id="EN-US_TOPIC_0000001233761817__b842352706164621">reasons_idx</strong> exists on the <strong id="EN-US_TOPIC_0000001233761817__b842352706164623">r_reason_id</strong> column in the <strong id="EN-US_TOPIC_0000001233761817__b10839195410103">reasons</strong> table. Delete the index before the import. Delete the index from the table.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761817__s524306896b53442383d324e3abb89825"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">DROP</span><span class="w"> </span><span class="k">INDEX</span><span class="w"> </span><span class="n">reasons_idx</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</div>
</li><li id="EN-US_TOPIC_0000001233761817__ld43c02d65aa94f088a74553471385c45">Import data from source data files through the <strong id="EN-US_TOPIC_0000001233761817__b84235270619577">foreign_tpcds_reasons</strong> foreign table to the <strong id="EN-US_TOPIC_0000001233761817__b842352706195713">reasons</strong> table.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761817__scfdb4070c13e42e8a3ce21fa89b2c257"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">reasons</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">foreign_tpcds_reasons</span><span class="w"> </span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761817__li1719419641613">You can create indexes again after the import is complete.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761817__screen1519466111619"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">INDEX</span><span class="w"> </span><span class="n">reasons_idx</span><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">reasons</span><span class="p">(</span><span class="n">r_reasons_id</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_04_0189.html">Using GDS to Import Data from a Remote Server</a></div>
</div>
</div>
View Git Blame Copy Permalink