vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dws/dev/dws_04_0195.html
Lu, Huayi a24ca60074 DWS DEVELOPER 811 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Lu, Huayi <luhuayi@huawei.com>
Co-committed-by: Lu, Huayi <luhuayi@huawei.com>
2023-01-19 13:37:49 +00:00

52 lines
9.6 KiB
HTML
Raw Blame History

<a name="EN-US_TOPIC_0000001145894887"></a><a name="EN-US_TOPIC_0000001145894887"></a>
<h1 class="topictitle1">Importing Data</h1>
<div id="body8662426"><p id="EN-US_TOPIC_0000001145894887__p1240415177247">This section describes how to create tables in <span id="EN-US_TOPIC_0000001145894887__text979156286">GaussDB(DWS)</span> and import data to the tables.</p>
<p id="EN-US_TOPIC_0000001145894887__p6701165249">Before importing all the data from a table containing over 10 million records, you are advised to import some of the data, and check whether there is data skew and whether the distribution keys need to be changed. Troubleshoot the data skew if any. It is costly to address data skew and change the distribution keys after a large amount of data has been imported.</p>
<div class="section" id="EN-US_TOPIC_0000001145894887__s346fe31d510b413a9e00a4da16781f38"><h4 class="sectiontitle">Prerequisites</h4><p id="EN-US_TOPIC_0000001145894887__p1397022164117">The GDS server can communicate with GaussDB(DWS).</p>
<ul id="EN-US_TOPIC_0000001145894887__ul176821714429"><li id="EN-US_TOPIC_0000001145894887__li17685176420">You need to create an ECS as the GDS server.</li><li id="EN-US_TOPIC_0000001145894887__li424084620429">The created ECS and GaussDB(DWS) must belong to the same region, VPC, and subnet.</li></ul>
</div>
<div class="section" id="EN-US_TOPIC_0000001145894887__s512ba003c1d542e4b2df804651faea79"><h4 class="sectiontitle">Procedure</h4><ol id="EN-US_TOPIC_0000001145894887__oc008fca0a9e04a75a3b630745fc4bd45"><li id="EN-US_TOPIC_0000001145894887__l479f2d5c6f314d338a453ae9caa420e1"><span>Create a table in <span id="EN-US_TOPIC_0000001145894887__text1993120538">GaussDB(DWS)</span> to store imported data. For details, see CREATE TABLE.</span></li><li id="EN-US_TOPIC_0000001145894887__l675e212810b14373833bb5d4fa8f0b77"><span>Import data.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001145894887__s3b236958a0e54d71b6c339103242e3b0"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="p">[</span><span class="n">Target</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">name</span><span class="p">]</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="p">[</span><span class="k">Foreign</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">name</span><span class="p">]</span><span class="w"></span>
</pre></div></td></tr></table></div>
</div>
<ul id="EN-US_TOPIC_0000001145894887__ub621d008a7c3474b9db9f1f68c174461"><li id="EN-US_TOPIC_0000001145894887__l77526b6a51b94c6abb3f6ab122904097">If information similar to the following is displayed, the data has been imported. Query the error information table to check whether any data format errors occurred. For details, see <a href="dws_04_0196.html">Handling Import Errors</a>.<pre class="screen" id="EN-US_TOPIC_0000001145894887__sce91049b0bc54e32919be73b597c9e39">INSERT 0 9</pre>
</li><li id="EN-US_TOPIC_0000001145894887__l37c9171730e742f695e5d369f1e695d0">If data fails to be loaded, troubleshoot the problem by following the instructions provided in <a href="dws_04_0196.html">Handling Import Errors</a> and try again.</li></ul>
<div class="note" id="EN-US_TOPIC_0000001145894887__na88400ea8d584844921871bd201b4c9e"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="EN-US_TOPIC_0000001145894887__u10315cc2ffa64d329bd93cd5e7b4654e"><li id="EN-US_TOPIC_0000001145894887__li8963324134514">If a data loading error occurs, the entire data import task will fail.</li><li id="EN-US_TOPIC_0000001145894887__l9f079f661f2a41a7b1b25660d00c715a">Compile a batch-processing task script to concurrently import data. The degree of parallelism (DOP) depends on the server resource usage. You can test-import several tables, monitor resource utilization, and increase or reduce concurrency accordingly. Common resource monitoring commands include <strong id="EN-US_TOPIC_0000001145894887__b7618185013918">top</strong> for monitoring memory and CPU usage, <strong id="EN-US_TOPIC_0000001145894887__b1461845083914">iostat</strong> for monitoring I/O usage, and <strong id="EN-US_TOPIC_0000001145894887__b76181050193918">sar</strong> for monitoring networks. For details about application cases, see .</li><li id="EN-US_TOPIC_0000001145894887__l8b290ba70c714c33bc9446d2ac483e8f">If possible, more GDS servers can significantly improve the data import efficiency. For details about application cases, see <a href="dws_04_0198.html#EN-US_TOPIC_0000001145494625__section2041618243291">Importing Data in Parallel from Multiple Data Servers</a>.</li><li id="EN-US_TOPIC_0000001145894887__lfd88a221d9b145f28a119bbd12ff0b8a">In a scenario where many GDS servers import data concurrently, you can increase the TCP Keepalive interval for connections between GDS servers and DNs to ensure connection stability. (The recommended interval is 5 minutes.) TCP Keepalive settings of the cluster affect its fault detection response time.</li></ul>
</div></div>
</p></li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001145894887__s1c6c5420aeb94c46b58add6ea7cdac3f"><h4 class="sectiontitle">Example:</h4><ol id="EN-US_TOPIC_0000001145894887__ol146395348432"><li id="EN-US_TOPIC_0000001145894887__li1163914342435"><span>Create a target table named <strong id="EN-US_TOPIC_0000001145894887__b842352706181119">reasons</strong>.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001145894887__s3092bce97b2c4a7facac7bd5efea5f44"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">reasons</span><span class="w"></span>
<span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="n">DISTRIBUTE</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">HASH</span><span class="w"> </span><span class="p">(</span><span class="n">r_reason_sk</span><span class="p">);</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</p></li><li id="EN-US_TOPIC_0000001145894887__li26401834174318"><span>Import data from source data files through the <strong id="EN-US_TOPIC_0000001145894887__b84235270619577">foreign_tpcds_reasons</strong> foreign table to the <strong id="EN-US_TOPIC_0000001145894887__b842352706195713">reasons</strong> table.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001145894887__scfdb4070c13e42e8a3ce21fa89b2c257"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">reasons</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">foreign_tpcds_reasons</span><span class="w"> </span><span class="p">;</span><span class="w"></span>
</pre></div></td></tr></table></div>
</div>
</p></li><li id="EN-US_TOPIC_0000001145894887__li9640193418434"><span>You can create indexes again after the import is complete.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001145894887__screen1519466111619"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">INDEX</span><span class="w"> </span><span class="n">reasons_idx</span><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">reasons</span><span class="p">(</span><span class="n">r_reasons_id</span><span class="p">);</span><span class="w"></span>
</pre></div></td></tr></table></div>
</div>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_04_0189.html">Using GDS to Import Data from a Remote Server</a></div>
</div>
</div>
View Git Blame Copy Permalink