vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dws/dev/dws_04_0198.html
Lu, Huayi e6fa411af0 DWS DEV 830.201 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Lu, Huayi <luhuayi@huawei.com>
Co-committed-by: Lu, Huayi <luhuayi@huawei.com>
2024-05-16 07:24:04 +00:00

296 lines
79 KiB
HTML
Raw Blame History

<a name="EN-US_TOPIC_0000001233761893"></a><a name="EN-US_TOPIC_0000001233761893"></a>
<h1 class="topictitle1">Example of Importing Data Using GDS</h1>
<div id="body8662426"><div class="section" id="EN-US_TOPIC_0000001233761893__section2041618243291"><a name="EN-US_TOPIC_0000001233761893__section2041618243291"></a><a name="section2041618243291"></a><h4 class="sectiontitle">Parallel Import from Multiple Data Servers</h4><p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a465f90ab92e44fd1ac2e5331d4eb6574">The data servers and the cluster reside on the same intranet. The IP addresses are <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b105745411175">192.168.0.90</strong> and <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b92822455713">192.168.0.91</strong>. Source data files are in CSV format.</p>
<ol id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ode83f3e54d66420c8e57326f89df4268"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lf1732e1798614b418733df5d8d8f2cfd">Log in to each GDS data server as user <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706164548">root</strong> and create the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706164555">/input_data</strong> directory for storing data files on the servers. The following takes the data server whose IP address is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b159571720988">192.168.0.90</strong> as an example. Operations on the other server are the same.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sbb53de52dbaf4f8ab79768fa5e2cc514"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a88c5d81c4d0b46f199912a95959b6ecb">mkdir -p</strong> <em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a1d3cdbe53d6b43b68e0e3de719ef2e0a">/input_data</em></pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l1b57a31aefde416980683b316b21816d">(Optional) Create a user and the user group it belongs to. The user is used to start GDS. If the user and user group exist, skip this step.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sbe89ef3a703a45a1a69f1dad29006ed6"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0117407660_b2851994648">groupadd</strong> gdsgrp
<strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a854bc4c25d4b4ce6800680e1c0f2d72f">useradd -g</strong> gdsgrp gds_user</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lc8cd42b5114a4f55a4d3ca2349c702cd">Evenly distribute CSV source data files to the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706112150">/input_data</strong> directories on the data servers.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_le7af74753c6643c9935ea885ddee7662">Change the owners of source data files and the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b126088571505">/input_data</strong> directory on each data server to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b3608957708">gds_user</strong>. The data server whose IP address is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b143785526911">192.168.0.90</strong> is used as an example.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sd7a1012c682d459ab495e70000eb2af5"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ac6448a40312640ce93d6b25f245769da">chown -R</strong> gds_user:gdsgrp /input_data </pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l7d929faa456a4aa3b0bdbbc3bbf95e03">Log in to each data server as user <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b94641440618">gds_user</strong> and start GDS.<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a78e5e9eaca374431b422cb3bd11d5376">The GDS installation path is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b22456619237">/opt/bin/dws/gds</strong>. Source data files are stored in <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b958131472316">/input_data/</strong>. The IP addresses of the data servers are <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1774133341119">192.168.0.90</strong> and <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b116163715116">192.168.0.91</strong>. The GDS listening port is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b370674341120">5000</strong>. GDS runs in daemon mode.</p>
<div class="p" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p14496202813204">Start GDS on the data server whose IP address is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b9241101919126">192.168.0.90</strong>.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sd1493e57611b4d0c879917bbd0d9ecb9"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a1370245064af4bf58338347d753449e8">/opt/bin/dws/gds/bin/gds -d</strong> <em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a9927edbdac1641c9a5f0b5df13b311e7">/input_data</em> <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_aace37c25c51444eabb0f7694bacb7bd7">-p </strong><em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ac49811766ffa482884400c77b0defb29">192.168.0.90:5000 </em><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a3037b9267090413fa8f54e1542ddbf65">-H</strong><em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a9078fa03c01c4900bcd755e5bab455ab"> </em>10.10.0.1/24 <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a765301ae3be5451eb3ed81736163c5fb">-D</strong></pre>
</div>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a707492c6f8924e33863f0a12d9c34c0f">Start GDS on the data server whose IP address is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b12214269123">192.168.0.91</strong>.</p>
<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_s47f18bb849b14c9db2490a2cf611b477"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a688abd3aec5242c8819bbe5c9542a972">/opt/bin/dws/gds/bin/gds -d</strong> <em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_aeac34deaf2cf474aa02c99ac38fbf127">/input_data</em> <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_aa409f7fd2b44481eb305d6c3471d05c8">-p </strong><em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_aa426205d390a49039e8bb9e0e45b53b6">192.168.0.91:5000 </em><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_af40fd292fa1d4a35b7a92d1df8a8648c">-H</strong><em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a5034820818d440c1bc936d9e7a21be73"> </em>10.10.0.1/24 <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a727bc594c8f14b90b44cf05bbf1acb6f"> -D</strong></pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li138592663012">Use a tool to connect to the database. </li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li7347253240">Create the target table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b178212000842027"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text79846483242027">tpcds.</span>reasons</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen5348657245"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">reasons</span>
<span class="p">(</span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ld643549316f44956bd6e12c41f3c86d5">Create the foreign table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b16125193019461"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text119331748202912">tpcds.</span>foreign_tpcds_reasons</strong> for receiving data from the data server.<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ac95945d49530478990f21732615003b4">Data export mode settings are as follows:</p>
<ul id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_u1cfdc0d9abb14624901e4bf38067dec5"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l85b86de2e5744b2992583dda59b68633">Set the import mode to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b9537171216820">Normal</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ld1c6accd1aeb447395235306585c872f">When GDS is started, the source data file directory is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0085032064_en-us_topic_0058967605_b1452941350101319_1">/input_data</strong> and the GDS listening port is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b17493480133">5000</strong>. Therefore, set <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0085032064_en-us_topic_0058967605_b1904485453101319_1">location</strong> to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0085032064_en-us_topic_0058967605_b1935725810101319_1">gsfs://192.168.0.90:5000/* | gsfs://192.168.0.91:5000/*</strong>.</li></ul>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a8d376fe2150b4b31a5045accf32abc68">Information about the data format is configured based on data format parameters specified during data export. The parameter configurations are as follows:</p>
<ul id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_u69a35a9beca94280ae994af65857fb96"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_la3061eb8a2e2490c91e67e469f16769a"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706165627">format</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706165631">CSV</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l0f5e28adc21940a5af6c8e774194e2cc"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706143434">encoding</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706161148">UTF-8</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l3f47d627abe64bae8f5ea5e32c183bd2"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0058967650_b84235270616570_1">delimiter</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0058967650_b84235270616572_1">E'\x08'</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l542a8fb4c8114e8eb652d575356068ef"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706165714">quote</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706165717">E'\x1b'</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l139d936194404f3f96573276c726d7ab"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b554252501012">null</strong> is set to an empty string without quotation marks.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li1881144018294"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b09471625123211">escape</strong> defaults to the value of <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b595002510325">quote</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l130cbb41925d438fa7f8a24d1c70115d"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706165753">header</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706171243">false</strong>, indicating that the first row is identified as a data row in an imported file.</li></ul>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a9feb1ca25bd64e62bca39d239602e12e">Configure import error tolerance parameters as follows:</p>
<ul id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ubbc3459a022f4ec892782a72745ae08b"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l1b09e44235484400b70292378d841e0d">Set <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b84235270610154">PER NODE REJECT LIMIT</strong> (number of allowed data format errors) to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1663913257310">unlimited</strong>. In this case, all the data format errors detected during data import will be tolerated.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l4fb119474b184c218959a414532f8053">Set <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b115081710205320">LOG INTO</strong> to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b850819103537">err_tpcds_reasons</strong>. The data format errors detected during data import will be recorded in the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b18510141075311">err_tpcds_reasons</strong> table.</li></ul>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a6cf3b0991f5e40ba916895db17efdbc5">Based on the above settings, the foreign table is created using the following statement:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sa82bfa7a31244461a5aab277de2ae7d3"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">foreign_tpcds_reasons</span>
<span class="p">(</span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span><span class="k">location</span><span class="w"> </span><span class="s1">'gsfs://192.168.0.90:5000/* | gsfs://192.168.0.91:5000/*'</span><span class="p">,</span><span class="w"> </span><span class="n">format</span><span class="w"> </span><span class="s1">'CSV'</span><span class="p">,</span><span class="k">mode</span><span class="w"> </span><span class="s1">'Normal'</span><span class="p">,</span><span class="w"> </span><span class="k">encoding</span><span class="w"> </span><span class="s1">'utf8'</span><span class="p">,</span><span class="w"> </span><span class="k">delimiter</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x08'</span><span class="p">,</span><span class="w"> </span><span class="n">quote</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x1b'</span><span class="p">,</span><span class="w"> </span><span class="k">null</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">fill_missing_fields</span><span class="w"> </span><span class="s1">'false'</span><span class="p">)</span><span class="w"> </span><span class="n">LOG</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">err_tpcds_reasons</span><span class="w"> </span><span class="n">PER</span><span class="w"> </span><span class="n">NODE</span><span class="w"> </span><span class="n">REJECT</span><span class="w"> </span><span class="k">LIMIT</span><span class="w"> </span><span class="s1">'unlimited'</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l9173a914c5fd452680f1045124efe9dc">Import data through the foreign table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b947718316513"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text12620246131812">tpcds.</span>foreign_tpcds_reasons</strong> to the target table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b104771231125111"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text5924132191913">tpcds.</span>reasons</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_s7810b86ffab5428799c88973b0550345"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">reasons</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">foreign_tpcds_reasons</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lf9d7d1cf88204e24805feb2437e8a937">Query data import errors in the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b84235270615248">err_tpcds_reasons</strong> table and rectify the errors (if any). For details, see <a href="dws_04_0196.html">Handling Import Errors</a>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_s7ee3163db2af4d87a87d420ab9524e72"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">err_tpcds_reasons</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l691363ba214242029c90f34866efcc04">After data import is complete, log in to each data server as user <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0085032064_b84235270617183">gds_user</strong> and stop GDS.<div class="p" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p1451242814209">The data server whose IP address is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b78974328255">192.168.0.90</strong> is used as an example. The GDS process ID is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1412464810254">128954</strong>.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_s0ff0feeaf6d74de898624df4d545a944"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a5eb857e102e54350b2bdcc57a2223c37">ps -ef|grep gds</strong>
gds_user <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a41051798abeb4cfeba2d3608fb6eab86">128954</strong> 1 0 15:03 ? 00:00:00 gds -d /input_data -p 192.168.0.90:5000 -D
gds_user 129003 118723 0 15:04 pts/0 00:00:00 grep gds
<strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a0db9da3db6af4a76ac290745fc925c3a">kill -9</strong> 128954</pre>
</div>
</li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001233761893__section197704383292"><a name="EN-US_TOPIC_0000001233761893__section197704383292"></a><a name="section197704383292"></a><h4 class="sectiontitle">Data Import Using Multiple Threads</h4><p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a41c7b34b58fa484daea23c9b58cd6545">The data servers and the cluster reside on the same intranet. The server IP address is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b20246114514268">192.168.0.90</strong>. Source data files are in CSV format. Data will be imported to two tables using multiple threads in <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b175531531309">Normal</strong> mode.</p>
<ol id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_o8e097d1292174fe99af701ff4ab00a06"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lc292aa115449439bba98cb727492d139">Log in to the GDS data server as user <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b842352706133453">root</strong>, and then create the data file directory <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b84235270613350">/input_data</strong> and its sub-directories <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b84235270613354">/input_data/import1/</strong> and <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b84235270613358">/input_data/import2/</strong>.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sc08176b3568945599582634a1d77035c">mkdir -p <em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_aab2547f29d2c4b1fb40c52dbbf79ed9c">/input_data</em></pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_la07b86d240794549a11e719c37373963">Store the source data files of the target table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b686912499614"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text174381754112">tpcds.</span>reasons1</strong> in <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b16870194920612">/input_data/import1/</strong> and the source data files of the target table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b15871149761"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text15185162417417">tpcds.</span>reasons2</strong> in <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b28723494620">/input_data/import2/</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lc481980cc5844c6dbbb1e90b34c1b85d">(Optional) Create a user and the user group it belongs to. The user is used to start GDS. If the user and user group already exist, skip this step.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sd54c58be74414495bf4acc64ca6f1425"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ad791eedc77524f009a5a3116b5f8455a">groupadd</strong> gdsgrp
<strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0117407660_b17515341034">useradd -g</strong> gdsgrp gds_user</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l978525e82045443e8651d1b4107d2a32">Change the owners of source data files and the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0085032064_b842352706163715">/input_data</strong> directory on the data server to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0085032064_b842352706103433">gds_user</strong>.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sf18823e70247418daf989e4348fc17f7">chown -R <em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0117407660_i91561415449">gds_user:gdsgrp</em><em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_aa7034cebe99b4bc19defd7e1c60f7828"> /input_data </em></pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lb45756cd3c7f4e98b0b3b32b96b01d58">Log in to the data server as user <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b3415134158">gds_user</strong> and start GDS.<div class="p" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p172969435223">The GDS installation path is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b31918531143">/opt/bin/dws/gds</strong>. Source data files are stored in <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1688910214519">/input_data/</strong>. The IP address of the data server is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b18783154615282">192.168.0.90</strong>. The GDS listening port is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b8740125920286">5000</strong>. GDS runs in daemon mode. The degree of parallelism is 2. A recursive directory is specified.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_s861f5b84b6ec402f88917c77a8725f30"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_adc519e55546c4e77b0e7e715f9fce958">/opt/bin/dws/gds/bin/gds -d</strong> <em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a26e4e935f59c4716a52191d3b536d80f">/input_data</em> <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a5af0deea7cbd4f719163914d14a3ff36">-p </strong><em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_acbdac7fd977648c1a8fd4b931392a7fd">192.168.0.90:5000 </em><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a58fb1ca5a2574e32aac402d07b6908dc">-H</strong><em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a6bc71b79f0744a838c57d230494377c3"> </em>10.10.0.1/24 <em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a7982b94152e14ac4ba8c1f8e004334fd"> </em><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a029be89589a7428b87959ec98c65e53f">-D -t </strong>2<strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a2a388f742d7b489b8e23d63c4c559eb8"> -r</strong></pre>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li883314550342">Use a tool to connect to the database. </li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li2618175173519">In the database, create the target tables <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b65861788151"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text20586148141511">tpcds.</span>reasons1</strong> and <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b158778181514"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text20587158151519">tpcds.</span>reasons2</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen9618558355"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">reasons1</span>
<span class="p">(</span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">)</span><span class="w"> </span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen13618352355"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">reasons2</span>
<span class="p">(</span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">)</span><span class="w"> </span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l8bc4d54a954c42ef9640b28f3d654888">In the database, create the foreign tables <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b3894151818162"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text63531327183211">tpcds.</span>foreign_tpcds_reasons1</strong> and <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b696610208162"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text1530205716327">tpcds.</span>foreign_tpcds_reasons2</strong> for the source data.<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a10010a0a667a41d09692c20c34365dc6">The foreign table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b9559132351611"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text17683135012324">tpcds.</span>foreign_tpcds_reasons1</strong> is used as an example to describe how to configure parameters in a foreign table.</p>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a7528b264232f40b8a8cd98876b561d5c">Data export mode settings are as follows:</p>
<ul id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ua5c54221399b46199e585903380c38a3"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lf2365add8ea347f3a0ec903547916b2a">Set the import mode to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1560736292">Normal</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lca350094bff14b85af63ac2c37045f0d">When GDS is started, the configured source data file directory is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1561856056">/input_data</strong> and the GDS listening port is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b472318168300">5000</strong>. However, source data files are actually stored in <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b669334314817">/input_data/import1/</strong>. Therefore, set <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1173394530">location</strong> to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b459727147">gsfs://192.168.0.90:5000/import1/*</strong>.</li></ul>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a18068bdc8b5848bbbac24fbf188577dd">Information about the data format is configured based on data format parameters specified during data export. The parameter configurations are as follows:</p>
<ul id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ub0e2cadbb5bd4bcda86a635f4be054a9"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ld1bcd94006ee419db465f36b0783fe3d"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1333053497">format</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b652893085">CSV</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lbf533f6c727e4057a232a32443f0e094"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b332281736">encoding</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1633416815">UTF-8</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l0a4ec79818e34f5f8375ed36bf6b10f8"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b912626142">delimiter</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1720087931">E'\x08'</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_la8cdba8723934b32b7125427fd968089"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1423979173">quote</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1212959670">E'\x1b'</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lc2b6f1f31ebf413ab8529cefb4e9afbe"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1598033437">null</strong> is set to an empty string without quotation marks.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li123938174309"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1876916492314">escape</strong> defaults to the value of <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b6770149113111">quote</strong>.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l6b3b96db12694ac9a77f794b4b0e041c"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1563174025619">header</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b45637409568">false</strong>, indicating that the first row is identified as a data row in an imported file.</li></ul>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a16fef2f92eb3499989cc0756f1015816">Configure import error tolerance parameters as follows:</p>
<ul id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ub890053d27f547d099b6b7ad71ddeeff"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l8905589a965642fba6d3543169ae2174">Set <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b253042815912">PER NODE REJECT LIMIT</strong> (number of allowed data format errors) to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b175458281198">unlimited</strong>. In this case, all the data format errors detected during data import will be tolerated.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_laed2f99667f04d5fae92a14288d4452a">Set <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1677914109254">LOG INTO</strong> to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b2781111013257">err_tpcds_reasons1</strong>. The data format errors detected during data import will be recorded in the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b14782171012253">err_tpcds_reasons1</strong> table.</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l081fccc5c6204faf87e8d133b8046f22">If the last column of a source data file is missing, the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b04311713151817">fill_missing_fields</strong> parameter is automatically set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0085032064_b842352706134758">NULL</strong>.</li></ul>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_aa6f08899b7074df691268403d8265e98">Based on the preceding settings, the foreign table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b12791232278"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text187166614614">tpcds.</span>foreign_tpcds_reasons1</strong> is created using the following statement:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sb351e6f271414321bf3670fd29769e0c"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">foreign_tpcds_reasons1</span>
<span class="p">(</span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">)</span><span class="w"> </span><span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span><span class="k">location</span><span class="w"> </span><span class="s1">'gsfs://192.168.0.90:5000/import1/*'</span><span class="p">,</span><span class="w"> </span><span class="n">format</span><span class="w"> </span><span class="s1">'CSV'</span><span class="p">,</span><span class="k">mode</span><span class="w"> </span><span class="s1">'Normal'</span><span class="p">,</span><span class="w"> </span><span class="k">encoding</span><span class="w"> </span><span class="s1">'utf8'</span><span class="p">,</span><span class="w"> </span><span class="k">delimiter</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x08'</span><span class="p">,</span><span class="w"> </span><span class="n">quote</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x1b'</span><span class="p">,</span><span class="w"> </span><span class="k">null</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span><span class="n">fill_missing_fields</span><span class="w"> </span><span class="s1">'on'</span><span class="p">)</span><span class="n">LOG</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">err_tpcds_reasons1</span><span class="w"> </span><span class="n">PER</span><span class="w"> </span><span class="n">NODE</span><span class="w"> </span><span class="n">REJECT</span><span class="w"> </span><span class="k">LIMIT</span><span class="w"> </span><span class="s1">'unlimited'</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a5ebfd2bfc4334f548eaf02d822a9aec4">Based on the preceding settings, the foreign table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b6346161315295"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text1093415203315">tpcds.</span>foreign_tpcds_reasons2</strong> is created using the following statement:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_s6b48e7add5304d58b05ad2bf131e4ae2"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">foreign_tpcds_reasons2</span>
<span class="p">(</span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">)</span><span class="w"> </span><span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span><span class="k">location</span><span class="w"> </span><span class="s1">'gsfs://192.168.0.90:5000/import2/*'</span><span class="p">,</span><span class="w"> </span><span class="n">format</span><span class="w"> </span><span class="s1">'CSV'</span><span class="p">,</span><span class="k">mode</span><span class="w"> </span><span class="s1">'Normal'</span><span class="p">,</span><span class="w"> </span><span class="k">encoding</span><span class="w"> </span><span class="s1">'utf8'</span><span class="p">,</span><span class="w"> </span><span class="k">delimiter</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x08'</span><span class="p">,</span><span class="w"> </span><span class="n">quote</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x1b'</span><span class="p">,</span><span class="w"> </span><span class="k">null</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span><span class="n">fill_missing_fields</span><span class="w"> </span><span class="s1">'on'</span><span class="p">)</span><span class="n">LOG</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">err_tpcds_reasons2</span><span class="w"> </span><span class="n">PER</span><span class="w"> </span><span class="n">NODE</span><span class="w"> </span><span class="n">REJECT</span><span class="w"> </span><span class="k">LIMIT</span><span class="w"> </span><span class="s1">'unlimited'</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_l9393c1d1179b45058b480628b524cd55">Import data through the foreign table <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1986665313171"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text197171042536">tpcds.</span>foreign_tpcds_reasons1</strong> to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b12569133191816"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text1674419561934">tpcds.</span>reasons1</strong> and through <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b9147141716186"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text156114485320">tpcds.</span>foreign_tpcds_reasons2</strong> to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b2052331814"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text18720621745">tpcds.</span>reasons2</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_s844aa16fc22a471aadf145aedee74868"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">reasons1</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">foreign_tpcds_reasons1</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_s1108f677243c45efb14cd6a66160ae2b"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">reasons2</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">foreign_tpcds_reasons2</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_le98d1523c9ce4b959db52648f97bf669">Query data import errors in the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b87801421375">err_tpcds_reasons1</strong> and <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b678116211278">err_tpcds_reasons2</strong> tables and rectify the errors (if any). For details, see <a href="dws_04_0196.html">Handling Import Errors</a>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_se2beb6c2f39943b88e94da5108cdcac8"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">err_tpcds_reasons1</span><span class="p">;</span>
<span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">err_tpcds_reasons2</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_lbcb9503544a24337943a3090dab25df4">After data import is complete, log in to the data server as user <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_en-us_topic_0085032744_b372029376175142">gds_user</strong> and stop GDS.<div class="p" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p1737384372211">The GDS process ID is <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b9841743173511">128954</strong>.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_sd8448e03f3444b95b108c8d00bd888c1"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_aa5d3e598bacd4153a9d65bfc531b041f">ps -ef|grep gds</strong>
gds_user <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a2cc1b01346444017a110839b75cae7ac">128954</strong> 1 0 15:03 ? 00:00:00 gds -d /input_data -p 192.168.0.90:5000 -D -t 2 -r
gds_user 129003 118723 0 15:04 pts/0 00:00:00 grep gds
<strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_a01e470f496a54cd3b44061380ca97de8">kill -9</strong> 128954</pre>
</div>
</li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001233761893__section1115254593118"><h4 class="sectiontitle">Importing Data Through a Pipe File</h4><ol id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ol134874524597"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li34875528591"><span>Start the GDS.</span><p><pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen943410482717"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b103925137261">/opt/bin/dws/gds/bin/gds</strong> -d /***/gds_data/ -D -p 192.168.0.1:7789 -l /***/gds_log/aa.log -H 0/0 -t 10 -D</pre>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p19156181213910">If you need to set the timeout interval of a pipe, use the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b91671341820">--pipe-timeout</strong> parameter.</p>
</p></li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li969042513241"><span>Import data.</span><p><ol type="a" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ol143432393245"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li17343103962414">Log in to the database and create an internal table.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen146761906818"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_pipe_1</span><span class="p">(</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">sex</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li1376543102420">Create a read-only foreign table.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen16711128385"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">foreign_test_pipe_tr</span><span class="p">(</span><span class="w"> </span><span class="k">like</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span><span class="k">LOCATION</span><span class="w"> </span><span class="s1">'gsfs://192.168.0.1:7789/foreign_test_pipe.pipe'</span><span class="p">,</span><span class="w"> </span><span class="n">FORMAT</span><span class="w"> </span><span class="s1">'text'</span><span class="p">,</span><span class="w"> </span><span class="k">DELIMITER</span><span class="w"> </span><span class="s1">','</span><span class="p">,</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">EOL</span><span class="w"> </span><span class="s1">'0x0a'</span><span class="w"> </span><span class="p">,</span><span class="n">file_type</span><span class="w"> </span><span class="s1">'pipe'</span><span class="p">,</span><span class="n">auto_create_pipe</span><span class="w"> </span><span class="s1">'false'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li3952105122413">Execute the import statement. The statement is blocked.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen209517138817"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe_1</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">foreign_test_pipe_tr</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li760365011817"><span>Import data through the GDS pipes.</span><p><ol type="a" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ol360161922910"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li11601171917292">Log in to GDS and go to the GDS data directory.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen1037219322919">cd /***/gds_data/ </pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li136035256296">Create a pipe. If <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b18257101218188">auto_create_pipe</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b102633123187">true</strong>, skip this step.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen97964253919">mkfifo foreign_test_pipe.pipe; </pre>
<div class="note" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_note845153919568"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p1187812409562">A pipe will be automatically cleared after an operation is complete. To perform another operation, create a pipe file again.</p>
</div></div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li10941133282912">Write data to the pipe.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen485611611919">cat postgres_public_foreign_test_pipe_tw.txt &gt; foreign_test_pipe.pipe</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li810113529298">To read the compressed file to the pipe, run the following command.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen1485496142212">gzip -d &lt; out.gz &gt; foreign_test_pipe.pipe</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li2646800306">To read the HDFS file to the pipe, run the following command.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen751614404910">hdfs dfs -cat - /user/hive/***/test_pipe.txt &gt; foreign_test_pipe.pipe</pre>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li11851387011"><span>View the result returned by the import statement.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen147057116108"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe_1</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">foreign_test_pipe_tr</span><span class="p">;</span>
<span class="k">INSERT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">4</span>
<span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">test_pipe_1</span><span class="p">;</span>
<span class="n">id</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">sex</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">name</span>
<span class="c1">----+-----+----------------</span>
<span class="mi">3</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">11111111111111</span>
<span class="mi">1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">11111111111111</span>
<span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">11111111111111</span>
<span class="mi">4</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">11111111111111</span>
<span class="p">(</span><span class="mi">4</span><span class="w"> </span><span class="k">rows</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</p></li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001233761893__section5317154873219"><h4 class="sectiontitle">Importing Data Through Multi-Process Pipes</h4><p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p13983747104213">GDS also supports importing data through multi-process pipes. That is, one foreign table corresponds to multiple GDSs.</p>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p4983114714213">The following takes importing a local file as an example.</p>
<ol id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ol399912371166"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li17063819613"><span>Start multiple GDSs. If the GDSs have been started, skip this step.</span><p><pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen19706155917155"><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b127091593269">/opt/bin/dws/gds/bin/gds</strong> -d /***/gds_data/ -D -p 192.168.0.1:7789 -l /***/gds_log/aa.log -H 0/0 -t 10 -D
<strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b898612219278">/opt/bin/dws/gds/bin/gds</strong> -d /***/gds_data_1/ -D -p 192.168.0.1:7790 -l /***/gds_log_1/aa.log -H 0/0 -t 10 -D</pre>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p168901411191015">If you need to set the timeout interval of a pipe, use the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b12420132751813">--pipe-timeout</strong> parameter.</p>
</p></li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li020217500464"><span>Import data.</span><p><ol type="a" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ol942214539468"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li9422155310467">Log in to the database and create an internal table.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen895591561614"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_pipe</span><span class="p">(</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">sex</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li42061656164616">Create a read-only foreign table.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen5247122111610"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">foreign_test_pipe_tr</span><span class="p">(</span><span class="w"> </span><span class="k">like</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span><span class="k">LOCATION</span><span class="w"> </span><span class="s1">'gsfs://192.168.0.1:7789/foreign_test_pipe.pipe|gsfs://192.168.0.1:7790/foreign_test_pipe.pipe'</span><span class="p">,</span><span class="w"> </span><span class="n">FORMAT</span><span class="w"> </span><span class="s1">'text'</span><span class="p">,</span><span class="w"> </span><span class="k">DELIMITER</span><span class="w"> </span><span class="s1">','</span><span class="p">,</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">EOL</span><span class="w"> </span><span class="s1">'0x0a'</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="n">file_type</span><span class="w"> </span><span class="s1">'pipe'</span><span class="p">,</span><span class="w"> </span><span class="n">auto_create_pipe</span><span class="w"> </span><span class="s1">'false'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li1195135813466">Execute the import statement. The statement is blocked.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen192592681618"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe_1</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">foreign_test_pipe_tr</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li833194712189"><span>Import data through the GDS pipes.</span><p><ol type="a" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ol68921939184818"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li2276184564815">Log in to GDS and go to each GDS data directory.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen16276114544813">cd /***/gds_data/
cd /***/gds_data_1/</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li113121919114916">Create a pipe. If <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b1584934241818">auto_create_pipe</strong> is set to <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b68553427187">true</strong>, skip this step.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen11313131915492">mkfifo foreign_test_pipe.pipe; </pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li2222111219508">Read each pipe and write the new file to the pipes.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen71131623111715">cat postgres_public_foreign_test_pipe_tw.txt &gt; foreign_test_pipe.pipe</pre>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li8121523273"><span>View the result returned by the import statement.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen757193020177"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe_1</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">foreign_test_pipe_tr</span><span class="p">;</span>
<span class="k">INSERT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">4</span>
<span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">test_pipe_1</span><span class="p">;</span>
<span class="n">id</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">sex</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">name</span>
<span class="c1">----+-----+----------------</span>
<span class="mi">3</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">11111111111111</span>
<span class="mi">1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">11111111111111</span>
<span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">11111111111111</span>
<span class="mi">4</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="mi">11111111111111</span>
<span class="p">(</span><span class="mi">4</span><span class="w"> </span><span class="k">rows</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</p></li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001233761893__section46286123419"><h4 class="sectiontitle">Direct Data Import Between Clusters</h4><ol id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ol1324715291578"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li44781847977"><span>Start the GDS. (If the process has been started, skip this step.)</span><p><pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen1430619488275">gds -d /***/gds_data/ -D -p GDS_IP:GDS_PORT -l /***/gds_log/aa.log -H 0/0 -t 10 -D</pre>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p19306948122720">If you need to set the timeout interval of a pipe, use the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b15829230198">--pipe-timeout</strong> parameter.</p>
</p></li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li3306648182712"><span>Export data from the source database.</span><p><ol type="a" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ol1030611489279"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li1830674817277">Log in to the target database, create an internal table, and write data to the table.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen1814314454316"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text4798436885"></span>CREATE TABLE test_pipe( id integer not null, sex text not null, name text );
<span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text162239396819"></span>INSERT INTO test_pipe values(1,2,'11111111111111');
<span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text7417411182"></span>INSERT INTO test_pipe values(2,2,'11111111111111');
<span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text1568410422084"></span>INSERT INTO test_pipe values(3,2,'11111111111111');
<span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text460518452818"></span>INSERT INTO test_pipe values(4,2,'11111111111111');
<span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text353419472812"></span>INSERT INTO test_pipe values(5,2,'11111111111111');</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li1430794862716">Create a write-only foreign table.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen7307174815272"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text15641122016911"></span>CREATE FOREIGN TABLE foreign_test_pipe( id integer not null, age text not null, name text ) SERVER gsmpp_server OPTIONS (LOCATION 'gsfs://GDS_IP:GDS_PORT/', FORMAT 'text', DELIMITER ',', NULL '', EOL '0x0a' ,file_type 'pipe') WRITE ONLY;</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li8307184814279">Execute the import statement. The statement is blocked.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen123075486272"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text5520725197"></span>INSERT INTO foreign_test_pipe SELECT * FROM test_pipe;</pre>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li153070489272"><span>Import data to the target cluster.</span><p><ol type="a" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ol2307144862710"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li173073486272">Create an internal table.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen133071048132720"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text10313372917"></span>CREATE TABLE test_pipe (id integer not null, sex text not null, name text);</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li13071248152720">Create a read-only foreign table.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen199117201494"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text1753804013915"></span>CREATE FOREIGN TABLE foreign_test_pipe(like test_pipe) SERVER gsmpp_server OPTIONS (LOCATION 'gsfs://GDS_IP:GDS_PORT/', FORMAT 'text', DELIMITER ',', NULL '', EOL '0x0a' , file_type 'pipe', auto_create_pipe 'false');</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li230724812719">Run the following command to import data to the table.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen19308748152712"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text523384313910"></span>INSERT INTO test_pipe SELECT * FROM foreign_test_pipe;</pre>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li10248142913712"><span>View the result returned by the import statement from the target cluster.</span><p><pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen6308948122716"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text17823143961019"></span>SELECT * FROM test_pipe;
id | sex | name
----+-----+----------------
3 | 2 | 11111111111111
6 | 2 | 11111111111111
7 | 2 | 11111111111111
1 | 2 | 11111111111111
2 | 2 | 11111111111111
4 | 2 | 11111111111111
5 | 2 | 11111111111111
8 | 2 | 11111111111111
9 | 2 | 11111111111111
(9 rows)</pre>
</p></li></ol>
<div class="note" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_note791753711206"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p1691703717209">By default, the pipeline file exported from or imported to GDS is named in the format of <em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_i1329927101911">Database name</em>_<em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_i436112781914">Schema name</em>_<em id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_i7363271197">Foreign table name </em><strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b5367278196">.pipe</strong>. Therefore, the database name and schema name of the target cluster must be the same as those of the source cluster. If the database or schema is inconsistent, you can specify the same pipe file in the URL of the <strong id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_b85068287195">location</strong>.</p>
<p id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_p16835165918385">Example:</p>
<ul id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_ul14237144916155"><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li52378494159">Pipe name specified by a write-only foreign table.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen6515144103914"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text18281520142"></span>CREATE FOREIGN TABLE foreign_test_pipe(id integer not null, age text not null, name text) SERVER gsmpp_server OPTIONS (LOCATION 'gsfs://GDS_IP:GDS_PORT/foreign_test_pipe.pipe', FORMAT 'text', DELIMITER ',', NULL '', EOL '0x0a' ,file_type 'pipe') WRITE ONLY;</pre>
</li><li id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_li102371949111515">Pipe name specified by a read-only foreign table.<pre class="screen" id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_screen039515278158"><span id="EN-US_TOPIC_0000001233761893__en-us_topic_0000001188482272_text1513652016154"></span>CREATE FOREIGN TABLE foreign_test_pipe(like test_pipe) SERVER gsmpp_server OPTIONS (LOCATION 'gsfs://GDS_IP:GDS_PORT/foreign_test_pipe.pipe', FORMAT 'text', DELIMITER ',', NULL '', EOL '0x0a' ,file_type 'pipe',auto_create_pipe 'false');</pre>
</li></ul>
</div></div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_04_0189.html">Using GDS to Import Data from a Remote Server</a></div>
</div>
</div>
View Git Blame Copy Permalink