vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dws/dev/dws_04_0268.html
Lu, Huayi e6fa411af0 DWS DEV 830.201 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Lu, Huayi <luhuayi@huawei.com>
Co-committed-by: Lu, Huayi <luhuayi@huawei.com>
2024-05-16 07:24:04 +00:00

358 lines
66 KiB
HTML
Raw Blame History

<a name="EN-US_TOPIC_0000001188482140"></a><a name="EN-US_TOPIC_0000001188482140"></a>
<h1 class="topictitle1">Examples of Exporting Data Using GDS</h1>
<div id="body8662426"><div class="section" id="EN-US_TOPIC_0000001188482140__sdcad1f9356214a9f8912535e46f27537"><h4 class="sectiontitle">Exporting Data in Remote Mode</h4><p id="EN-US_TOPIC_0000001188482140__a4bf3b2440fbf4f52b6c6639e14224786">The data server and the cluster reside on the same intranet, the IP address of the data server is <strong id="EN-US_TOPIC_0000001188482140__b103731546123919">192.168.0.90</strong>, and data source files are in CSV format. In this scenario, data is exported in parallel in <strong id="EN-US_TOPIC_0000001188482140__b9116185712418">Remote</strong> mode.</p>
<p id="EN-US_TOPIC_0000001188482140__a46c3af2df2be45d4ba98deb083e90924">To export data in parallel in <strong id="EN-US_TOPIC_0000001188482140__b19241185122518">Remote</strong> mode, perform the following operations:</p>
<ol id="EN-US_TOPIC_0000001188482140__o9a0388cdf1e542d287a5540ae7baf502"><li id="EN-US_TOPIC_0000001188482140__ld4537905065b4df3aa5e83bf33ac481f">Log in to the GDS data server as user <strong id="EN-US_TOPIC_0000001188482140__b9305631141213">root</strong>, create the <strong id="EN-US_TOPIC_0000001188482140__b830517319124">/output_data</strong> directory for storing data files, and create user <strong id="EN-US_TOPIC_0000001188482140__b963043161711">gds_user</strong> and its user group.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__s78555e7342fb451fb59b0bd36209795b"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">mkdir</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="o">/</span><span class="n">output_data</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__l1b57a31aefde416980683b316b21816d">(Optional) Create a user and the user group it belongs to. The user is used to start GDS. If the user and user group exist, skip this step.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__sbe89ef3a703a45a1a69f1dad29006ed6"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">groupadd</span><span class="w"> </span><span class="n">gdsgrp</span>
<span class="n">useradd</span><span class="w"> </span><span class="o">-</span><span class="k">g</span><span class="w"> </span><span class="n">gdsgrp</span><span class="w"> </span><span class="n">gds_user</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__l24130c54425d46a0a6649eb9330ac0e8">Change the owner of the <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0085032744_b372029376172051">/output_data</strong> directory on the data server to <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0085032744_b84235270610312">gds_user</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__s84dc6d0d43b647278c078f1f8af92efa"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">chown</span><span class="w"> </span><span class="o">-</span><span class="n">R</span><span class="w"> </span><span class="n">gds_user</span><span class="p">:</span><span class="n">gdsgrp</span><span class="w"> </span><span class="o">/</span><span class="n">output_data</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__lcb5f581a8d1c49859759f52e647f1e68">Log in to the data server as user <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0085032744_b372029376172120">gds_user</strong> and start GDS.<div class="p" id="EN-US_TOPIC_0000001188482140__a25f16b50be834206b5d988f0bf25b4c8">The GDS installation path is <strong id="EN-US_TOPIC_0000001188482140__b4642519162812">/opt/bin/dws/gds</strong>. Exported data files are stored in <strong id="EN-US_TOPIC_0000001188482140__b76429192285">/output_data/</strong>. The IP address of the data server is <strong id="EN-US_TOPIC_0000001188482140__b1627094544215">192.168.0.90</strong>. The GDS listening port is <strong id="EN-US_TOPIC_0000001188482140__b83568484425">5000</strong>. GDS runs in daemon mode.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__sda5fd2f566f840d4887090dcc815b2bd"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="o">/</span><span class="n">opt</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">dws</span><span class="o">/</span><span class="n">gds</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">gds</span><span class="w"> </span><span class="o">-</span><span class="n">d</span><span class="w"> </span><span class="o">/</span><span class="n">output_data</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="mi">192</span><span class="p">.</span><span class="mi">168</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">90</span><span class="p">:</span><span class="mi">5000</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="mi">10</span><span class="p">.</span><span class="mi">10</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="o">/</span><span class="mi">24</span><span class="w"> </span><span class="o">-</span><span class="n">D</span>
</pre></div></td></tr></table></div>
</div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__l955109db195e4df0b8cb0c8302b0a4db">In the database, create the foreign table <strong id="EN-US_TOPIC_0000001188482140__b10909105514010">foreign_tpcds_reasons</strong> for receiving data from the data server.<p id="EN-US_TOPIC_0000001188482140__a74b966859e4d466392e1a4c012105b9f">Data export mode settings are as follows:</p>
<ul id="EN-US_TOPIC_0000001188482140__u6fbf09b30fa24a02aff41c2e4e3421e0"><li id="EN-US_TOPIC_0000001188482140__l2ab20bef959246f099a6f1f79d706d81">The directory for storing exported files is <strong id="EN-US_TOPIC_0000001188482140__b372029376172233">/output_data/</strong> and the GDS listening port is <strong id="EN-US_TOPIC_0000001188482140__b189631964441">5000</strong> when GDS is started. The directory created for storing exported files is <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0085032744_b372029376172254">/output_data/</strong>. Therefore, the <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0085032744_b37202937617234">location</strong> parameter is set to <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0085032744_b37202937617239">gsfs://192.168.0.90:5000/</strong>.</li></ul>
<p id="EN-US_TOPIC_0000001188482140__a532202bd8e50499fb6c472047f1d7266">Data format parameter settings are as follows:</p>
<ul id="EN-US_TOPIC_0000001188482140__uc2215176ef004d86a5851eb452470f5b"><li id="EN-US_TOPIC_0000001188482140__l7ca5051d0da343948517138d7b9818d6"><strong id="EN-US_TOPIC_0000001188482140__b842352706165627">format</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b842352706165631">CSV</strong>.</li><li id="EN-US_TOPIC_0000001188482140__lbf7f52ea70814df0bc8ba44c84711fce"><strong id="EN-US_TOPIC_0000001188482140__b842352706143434">encoding</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b842352706161148">UTF-8</strong>.</li><li id="EN-US_TOPIC_0000001188482140__ld341b52fd0584aa494c6961abc82c5ca"><strong id="EN-US_TOPIC_0000001188482140__b18967331741">delimiter</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b58961333547">E'\x08'</strong>.</li><li id="EN-US_TOPIC_0000001188482140__l329b644c674644e88bd3a522add2450d"><strong id="EN-US_TOPIC_0000001188482140__b842352706165714">quote</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b842352706165717">E'\x1b'</strong>.</li><li id="EN-US_TOPIC_0000001188482140__l3baf5e5eedd14a2cabeb02bf5ba34eed"><strong id="EN-US_TOPIC_0000001188482140__b554252501012">null</strong> is set to an empty string without quotation marks.</li><li id="EN-US_TOPIC_0000001188482140__li79983214235"><strong id="EN-US_TOPIC_0000001188482140__b3394931124512">escape</strong> defaults to the value of <strong id="EN-US_TOPIC_0000001188482140__b1639443117453">quote</strong>.</li><li id="EN-US_TOPIC_0000001188482140__l6e7cbd6e683c45cfaa47999c9a149d05"><strong id="EN-US_TOPIC_0000001188482140__b151131842133616">header</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b16113104213619">false</strong>, indicating that the first row is identified as a data row in an exported file.</li></ul>
<p id="EN-US_TOPIC_0000001188482140__a72f693e13d40407dba47e20ce9543f98">Based on the above settings, the foreign table is created using the following statement:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__sf4b7b16acd6345a8a9bfbb644aed3130"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">foreign_tpcds_reasons</span>
<span class="p">(</span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">)</span><span class="w"> </span>
<span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span>
<span class="k">OPTIONS</span><span class="w"> </span>
<span class="p">(</span>
<span class="k">LOCATION</span><span class="w"> </span><span class="s1">'gsfs://192.168.0.90:5000/'</span><span class="p">,</span><span class="w"> </span>
<span class="n">FORMAT</span><span class="w"> </span><span class="s1">'CSV'</span><span class="p">,</span>
<span class="k">ENCODING</span><span class="w"> </span><span class="s1">'utf8'</span><span class="p">,</span>
<span class="k">DELIMITER</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x08'</span><span class="p">,</span><span class="w"> </span>
<span class="n">QUOTE</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x1b'</span><span class="p">,</span><span class="w"> </span>
<span class="k">NULL</span><span class="w"> </span><span class="s1">''</span>
<span class="p">)</span><span class="w"> </span>
<span class="k">WRITE</span><span class="w"> </span><span class="k">ONLY</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__l9dbbfaf5608c424f91983b2da930e214">In the database, export data to data files through the foreign table <strong id="EN-US_TOPIC_0000001188482140__b1439165213111">foreign_tpcds_reasons</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__s923bd88f69064af2bb3b61dd87416741"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">foreign_tpcds_reasons</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">reason</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__l83cce3771fcd4e1e835f2f47c6e25001">After data export is complete, log in to the data server as user <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0085032744_b372029376175142">gds_user</strong> and stop GDS.<div class="p" id="EN-US_TOPIC_0000001188482140__a4d1bf02bbbca4499aded2e9fb0d172ba">The GDS process ID is <strong id="EN-US_TOPIC_0000001188482140__b11482734465">128954</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__s4fe052d30dba4624b044380a78d960bf"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">ps</span><span class="w"> </span><span class="o">-</span><span class="n">ef</span><span class="o">|</span><span class="n">grep</span><span class="w"> </span><span class="n">gds</span>
<span class="n">gds_user</span><span class="w"> </span><span class="mi">128954</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">15</span><span class="p">:</span><span class="mi">03</span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="w"> </span><span class="n">gds</span><span class="w"> </span><span class="o">-</span><span class="n">d</span><span class="w"> </span><span class="o">/</span><span class="n">output_data</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="mi">192</span><span class="p">.</span><span class="mi">168</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">90</span><span class="p">:</span><span class="mi">5000</span><span class="w"> </span><span class="o">-</span><span class="n">D</span>
<span class="n">gds_user</span><span class="w"> </span><span class="mi">129003</span><span class="w"> </span><span class="mi">118723</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">15</span><span class="p">:</span><span class="mi">04</span><span class="w"> </span><span class="n">pts</span><span class="o">/</span><span class="mi">0</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="w"> </span><span class="n">grep</span><span class="w"> </span><span class="n">gds</span>
<span class="n">kill</span><span class="w"> </span><span class="o">-</span><span class="mi">9</span><span class="w"> </span><span class="mi">128954</span>
</pre></div></td></tr></table></div>
</div>
</div>
</li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001188482140__s855daf73006d4e05ba6d04f8db74e7f6"><a name="EN-US_TOPIC_0000001188482140__s855daf73006d4e05ba6d04f8db74e7f6"></a><a name="s855daf73006d4e05ba6d04f8db74e7f6"></a><h4 class="sectiontitle">Exporting Data Using Multiple Threads</h4><p id="EN-US_TOPIC_0000001188482140__a6ea339cc06174f5783ef17efa4e0c072">The data server and the cluster reside on the same intranet, the IP address of the data server is <strong id="EN-US_TOPIC_0000001188482140__b6259329194715">192.168.0.90</strong>, and data source files are in CSV format. In this scenario, data is concurrently exported to two target tables using multiple threads in <strong id="EN-US_TOPIC_0000001188482140__b186946719490">Remote</strong> mode.</p>
<p id="EN-US_TOPIC_0000001188482140__afc0d8043841e415980397eb4daeb63f2">To concurrently export data using multiple threads in <strong id="EN-US_TOPIC_0000001188482140__b2092018175413">Remote</strong> mode, perform the following operations:</p>
<ol id="EN-US_TOPIC_0000001188482140__o2b7bb4d1b885432cb7dc251831f558da"><li id="EN-US_TOPIC_0000001188482140__ldb4da38e458c47d88063fa270e2c566b">Log in to the GDS data server as user <strong id="EN-US_TOPIC_0000001188482140__b57682817226">root</strong>, create the <strong id="EN-US_TOPIC_0000001188482140__b2076910817228">/output_data</strong> directory for storing data files, and create the database user and its user group.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__s2b312a2cdd5c45f4ba1a775ea26de6a6"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">mkdir</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="o">/</span><span class="n">output_data</span>
<span class="n">groupadd</span><span class="w"> </span><span class="n">gdsgrp</span>
<span class="n">useradd</span><span class="w"> </span><span class="o">-</span><span class="k">g</span><span class="w"> </span><span class="n">gdsgrp</span><span class="w"> </span><span class="n">gds_user</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__l5db37895dbaf4e58b4321d494e5d1342">Change the owner of the <strong id="EN-US_TOPIC_0000001188482140__b296490004">/output_data</strong> directory on the data server to <strong id="EN-US_TOPIC_0000001188482140__b1458620748">gds_user</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__sa2dd8dda39884fdcbcf342448633ba5c"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">chown</span><span class="w"> </span><span class="o">-</span><span class="n">R</span><span class="w"> </span><span class="n">gds_user</span><span class="p">:</span><span class="n">gdsgrp</span><span class="w"> </span><span class="o">/</span><span class="n">output_data</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__ld2c41791b1cc4c5285d48ae380ff7f23">Log in to the data server as user <strong id="EN-US_TOPIC_0000001188482140__b174161458102212">gds_user</strong> and start GDS.<div class="p" id="EN-US_TOPIC_0000001188482140__a75a988d7150e436889ed7f764dbf1abd">The GDS installation path is <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0085032744_b3720293761817">/opt/bin/dws/gds</strong>. Exported data files are stored in <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0085032744_b37202937618115">/output_data/</strong>. The IP address of the data server is <strong id="EN-US_TOPIC_0000001188482140__b1724421112496">192.168.0.90</strong>. The GDS listening port is <strong id="EN-US_TOPIC_0000001188482140__b544915165491">5000</strong>. GDS runs in daemon mode. The degree of parallelism is 2.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__s7791d200070a4add972a2c11db854842"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="o">/</span><span class="n">opt</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">dws</span><span class="o">/</span><span class="n">gds</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">gds</span><span class="w"> </span><span class="o">-</span><span class="n">d</span><span class="w"> </span><span class="o">/</span><span class="n">output_data</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="mi">192</span><span class="p">.</span><span class="mi">168</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">90</span><span class="p">:</span><span class="mi">5000</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="mi">10</span><span class="p">.</span><span class="mi">10</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="o">/</span><span class="mi">24</span><span class="w"> </span><span class="o">-</span><span class="n">D</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__ld521900340f64063a359f8ec145383a1">In <span id="EN-US_TOPIC_0000001188482140__text1893218442">GaussDB(DWS)</span>, create the foreign tables <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0125634759_b894734124412">foreign_tpcds_reasons1</strong> and <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0125634759_b842352706155915">foreign_tpcds_reasons2</strong> for receiving data from the data server.<ul id="EN-US_TOPIC_0000001188482140__u17b613cb33be435e95ada54073ca2da2"><li id="EN-US_TOPIC_0000001188482140__l52bbad736a0a4deb8c6239773fdedbda">Data export mode settings are as follows:<ul id="EN-US_TOPIC_0000001188482140__u7fae3b75f8114b309ee250164a951cf1"><li id="EN-US_TOPIC_0000001188482140__la07e3bacf69347538b2dd610ff1d8be9">The directory for storing exported files is <strong id="EN-US_TOPIC_0000001188482140__b1048213454494">/output_data/</strong> and the GDS listening port is <strong id="EN-US_TOPIC_0000001188482140__b114821445164913">5000</strong> when GDS is started. The directory created for storing exported files is <strong id="EN-US_TOPIC_0000001188482140__b6794205134913">/output_data/</strong>. Therefore, the <strong id="EN-US_TOPIC_0000001188482140__b143805914453">location</strong> parameter is set to <strong id="EN-US_TOPIC_0000001188482140__b438015919455">gsfs://192.168.0.90:5000/</strong>.</li></ul>
</li><li id="EN-US_TOPIC_0000001188482140__ldb671c8309fa415b98373b1e59428c52">Data format parameter settings are as follows:<ul id="EN-US_TOPIC_0000001188482140__u2f25d570ce7e448b9b1a85642018db58"><li id="EN-US_TOPIC_0000001188482140__la94b8d957a114f1dbd04480e9d9ff947"><strong id="EN-US_TOPIC_0000001188482140__b60275465">format</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b1692698245">CSV</strong>.</li><li id="EN-US_TOPIC_0000001188482140__l8bb0675b3eb04ba6bcfcac6acd7e8f67"><strong id="EN-US_TOPIC_0000001188482140__b738548862">encoding</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b1667566538">UTF-8</strong>.</li><li id="EN-US_TOPIC_0000001188482140__la29d3cb7e95e4054b88cf91194c2daf2"><strong id="EN-US_TOPIC_0000001188482140__b06293510420">delimiter</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b166316351148">E'\x08'</strong>.</li><li id="EN-US_TOPIC_0000001188482140__l97fc4774a9a249819077ede9d22d2d06"><strong id="EN-US_TOPIC_0000001188482140__b611395850">quote</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b427253482">E'\x1b'</strong>.</li><li id="EN-US_TOPIC_0000001188482140__lf3539c39860645c094bb66d2a33de0ab"><strong id="EN-US_TOPIC_0000001188482140__b1797739391">null</strong> is set to an empty string without quotation marks.</li><li id="EN-US_TOPIC_0000001188482140__lcf191e63fc704080a882c8abeaf82f08"><strong id="EN-US_TOPIC_0000001188482140__b17451298503">escape</strong> defaults to the value of <strong id="EN-US_TOPIC_0000001188482140__b11457209115013">quote</strong>.</li><li id="EN-US_TOPIC_0000001188482140__l144b35da7b8d41e4ad28b7746e228054"><strong id="EN-US_TOPIC_0000001188482140__b146811268433">header</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b94687263435">false</strong>, indicating that the first row is identified as a data row in an exported file.</li></ul>
</li></ul>
<p id="EN-US_TOPIC_0000001188482140__a9812744e60b2445283a90189b2ebafce">Based on the preceding settings, the foreign table <strong id="EN-US_TOPIC_0000001188482140__b4338191235">foreign_tpcds_reasons1</strong> is created using the following statement:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__s31fdffdc7ad3444190f415b63e209abc"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">foreign_tpcds_reasons1</span>
<span class="p">(</span><span class="w"> </span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">)</span><span class="w"> </span>
<span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span><span class="w"> </span>
<span class="k">OPTIONS</span><span class="w"> </span>
<span class="p">(</span>
<span class="k">LOCATION</span><span class="w"> </span><span class="s1">'gsfs://192.168.0.90:5000/'</span><span class="p">,</span><span class="w"> </span>
<span class="n">FORMAT</span><span class="w"> </span><span class="s1">'CSV'</span><span class="p">,</span>
<span class="k">ENCODING</span><span class="w"> </span><span class="s1">'utf8'</span><span class="p">,</span><span class="w"> </span>
<span class="k">DELIMITER</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x08'</span><span class="p">,</span><span class="w"> </span>
<span class="n">QUOTE</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x1b'</span><span class="p">,</span><span class="w"> </span>
<span class="k">NULL</span><span class="w"> </span><span class="s1">''</span>
<span class="p">)</span><span class="w"> </span>
<span class="k">WRITE</span><span class="w"> </span><span class="k">ONLY</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
<p id="EN-US_TOPIC_0000001188482140__a411aae6fd01b451fbb2564515664f138">Based on the preceding settings, the foreign table <strong id="EN-US_TOPIC_0000001188482140__b151361340164614">foreign_tpcds_reasons2</strong> is created using the following statement:</p>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__sadae6271ec644734a09c29475da670e5"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">foreign_tpcds_reasons2</span>
<span class="p">(</span><span class="w"> </span>
<span class="w"> </span><span class="n">r_reason_sk</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_id</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span>
<span class="w"> </span><span class="n">r_reason_desc</span><span class="w"> </span><span class="nb">char</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="p">)</span><span class="w"> </span>
<span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span><span class="w"> </span>
<span class="k">OPTIONS</span><span class="w"> </span>
<span class="p">(</span>
<span class="k">LOCATION</span><span class="w"> </span><span class="s1">'gsfs://192.168.0.90:5000/'</span><span class="p">,</span><span class="w"> </span>
<span class="n">FORMAT</span><span class="w"> </span><span class="s1">'CSV'</span><span class="p">,</span><span class="w"> </span>
<span class="k">DELIMITER</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x08'</span><span class="p">,</span><span class="w"> </span>
<span class="n">QUOTE</span><span class="w"> </span><span class="n">E</span><span class="s1">'\x1b'</span><span class="p">,</span><span class="w"> </span>
<span class="k">NULL</span><span class="w"> </span><span class="s1">''</span>
<span class="p">)</span><span class="w"> </span>
<span class="k">WRITE</span><span class="w"> </span><span class="k">ONLY</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__l060f64b7504940fc99ec63e143c7b546">In the database, export data from table <strong id="EN-US_TOPIC_0000001188482140__b1738319311478">reasons1</strong> through the foreign table <strong id="EN-US_TOPIC_0000001188482140__b102711951483">foreign_tpcds_reasons1</strong> and from table <strong id="EN-US_TOPIC_0000001188482140__b16803141414819">reasons2</strong> through the foreign table <strong id="EN-US_TOPIC_0000001188482140__b1267772454819">foreign_tpcds_reasons2</strong> to <strong id="EN-US_TOPIC_0000001188482140__en-us_topic_0058967650_b37202937618421">/output_data</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__sea0f7f34561a48aca8450e52b26b6b2c"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">foreign_tpcds_reasons1</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">reason</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__s0455fa60cbf14f2ba7fa907c38031be3"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">foreign_tpcds_reasons2</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">tpcds</span><span class="p">.</span><span class="n">reason</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__ldb2968d060f84be094f5c20b549cf5f3">After data export is complete, log in to the data server as user <strong id="EN-US_TOPIC_0000001188482140__b1088293854814">gds_user</strong> and stop GDS.<div class="p" id="EN-US_TOPIC_0000001188482140__a7483e2c69c094c52a3cf824f6ce55b19">The GDS process ID is <strong id="EN-US_TOPIC_0000001188482140__b1798518377518">128954</strong>.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__sa7aaeb365f45405aba5d06f7d088a9eb"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">ps</span><span class="w"> </span><span class="o">-</span><span class="n">ef</span><span class="o">|</span><span class="n">grep</span><span class="w"> </span><span class="n">gds</span>
<span class="n">gds_user</span><span class="w"> </span><span class="mi">128954</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">15</span><span class="p">:</span><span class="mi">03</span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="w"> </span><span class="n">gds</span><span class="w"> </span><span class="o">-</span><span class="n">d</span><span class="w"> </span><span class="o">/</span><span class="n">output_data</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="mi">192</span><span class="p">.</span><span class="mi">168</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">90</span><span class="p">:</span><span class="mi">5000</span><span class="w"> </span><span class="o">-</span><span class="n">D</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span>
<span class="n">gds_user</span><span class="w"> </span><span class="mi">129003</span><span class="w"> </span><span class="mi">118723</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">15</span><span class="p">:</span><span class="mi">04</span><span class="w"> </span><span class="n">pts</span><span class="o">/</span><span class="mi">0</span><span class="w"> </span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="w"> </span><span class="n">grep</span><span class="w"> </span><span class="n">gds</span>
<span class="n">kill</span><span class="w"> </span><span class="o">-</span><span class="mi">9</span><span class="w"> </span><span class="mi">128954</span>
</pre></div></td></tr></table></div>
</div>
</div>
</li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001188482140__section206932582119"><h4 class="sectiontitle">Exporting Data Through a Pipe</h4><ol id="EN-US_TOPIC_0000001188482140__ol82421990535"><li id="EN-US_TOPIC_0000001188482140__li6244119145316"><span>Start GDS.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen135315351579"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">gds</span><span class="w"> </span><span class="o">-</span><span class="n">d</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_data</span><span class="o">/</span><span class="w"> </span><span class="o">-</span><span class="n">D</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="mi">192</span><span class="p">.</span><span class="mi">168</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">7789</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_log</span><span class="o">/</span><span class="n">aa</span><span class="p">.</span><span class="n">log</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="o">-</span><span class="n">D</span>
</pre></div></td></tr></table></div>
</div>
<p id="EN-US_TOPIC_0000001188482140__p1152131474119">If you need to set the timeout interval of a pipe, use the <strong id="EN-US_TOPIC_0000001188482140__b76121830122214">--pipe-timeout</strong> parameter.</p>
</p></li><li id="EN-US_TOPIC_0000001188482140__li1390185824920"><span>Export data.</span><p><ol type="a" id="EN-US_TOPIC_0000001188482140__ol1817971012508"><li id="EN-US_TOPIC_0000001188482140__li131793107508">Log in to the database, create an internal table, and write data to the table.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen323918010593"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_pipe</span><span class="p">(</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">sex</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">;</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="k">values</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="s1">'11111111111111'</span><span class="p">);</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="k">values</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="s1">'11111111111111'</span><span class="p">);</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="k">values</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="s1">'11111111111111'</span><span class="p">);</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="k">values</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="s1">'11111111111111'</span><span class="p">);</span>
<span class="k">INSERT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">1</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li17478201417506">Create a write-only foreign table.<pre class="screen" id="EN-US_TOPIC_0000001188482140__screen62601859102512"><span id="EN-US_TOPIC_0000001188482140__text1039115588345"></span>CREATE FOREIGN TABLE foreign_test_pipe_tw( id integer not null, age text not null, name text ) SERVER gsmpp_server OPTIONS (LOCATION 'gsfs://192.168.0.1:7789/', FORMAT 'text', DELIMITER ',', NULL '', EOL '0x0a' ,file_type 'pipe', auto_create_pipe 'false') WRITE ONLY;</pre>
</li><li id="EN-US_TOPIC_0000001188482140__li47991418135213">Execute the export statement. In this case, the statements are blocked.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen175603226012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">foreign_test_pipe_tw</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">test_pipe</span><span class="p">;</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001188482140__li149561245205218"><span>Export data through the GDS pipes.</span><p><ol type="a" id="EN-US_TOPIC_0000001188482140__ol15910111110109"><li id="EN-US_TOPIC_0000001188482140__li139111811131016">Log in to GDS and go to the GDS data directory.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen9788294017"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">cd</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_data</span><span class="o">/</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li51338155105">Create a pipe. If <strong id="EN-US_TOPIC_0000001188482140__b127901253142214">auto_create_pipe</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b10790165322213">true</strong>, skip this step.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen1514862454012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">mkfifo</span><span class="w"> </span><span class="n">postgres_public_foreign_test_pipe_tw</span><span class="p">.</span><span class="n">pipe</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
<div class="note" id="EN-US_TOPIC_0000001188482140__note1739511267561"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="EN-US_TOPIC_0000001188482140__p11348177103810">A pipe will be automatically cleared after an operation is complete. To perform another operation, create a pipe file again.</p>
</div></div>
</li><li id="EN-US_TOPIC_0000001188482140__li97952111103">Read data from the pipe and write it to a new file.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen1493183634012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">cat</span><span class="w"> </span><span class="n">postgres_public_foreign_test_pipe_tw</span><span class="p">.</span><span class="n">pipe</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="n">postgres_public_foreign_test_pipe_tw</span><span class="p">.</span><span class="n">txt</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li1624962831013">To compress the exported files, run the following command:<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen14320838125815"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">gzip</span><span class="w"> </span><span class="o">-</span><span class="mi">9</span><span class="w"> </span><span class="o">-</span><span class="k">c</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">postgres_public_foreign_test_pipe_tw</span><span class="p">.</span><span class="n">pipe</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="k">out</span><span class="p">.</span><span class="n">gz</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li2679153791019">To export the content from the pipe to the HDFS server, run the following command:<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen137382145512"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">cat</span><span class="w"> </span><span class="n">postgres_public_foreign_test_pipe_tw</span><span class="p">.</span><span class="n">pipe</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">hdfs</span><span class="w"> </span><span class="n">dfs</span><span class="w"> </span><span class="o">-</span><span class="n">put</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">/</span><span class="k">user</span><span class="o">/</span><span class="n">hive</span><span class="cm">/***/</span><span class="n">test_pipe</span><span class="p">.</span><span class="n">txt</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001188482140__li19597125335218"><span>Verify the exported data.</span><p><ol type="a" id="EN-US_TOPIC_0000001188482140__ol986619499153"><li id="EN-US_TOPIC_0000001188482140__li686654941512">Check whether the exported file is correct.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen3187123512519"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">cat</span><span class="w"> </span><span class="n">postgres_public_foreign_test_pipe_tw</span><span class="p">.</span><span class="n">txt</span>
<span class="mi">3</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">4</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li3258165316153">View the compressed file.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen113614487516"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">vim</span><span class="w"> </span><span class="k">out</span><span class="p">.</span><span class="n">gz</span>
<span class="mi">3</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">4</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li155211259181516">View the data exported to the HDFS server.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen168718416619"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">hdfs</span><span class="w"> </span><span class="n">dfs</span><span class="w"> </span><span class="o">-</span><span class="n">cat</span><span class="w"> </span><span class="o">/</span><span class="k">user</span><span class="o">/</span><span class="n">hive</span><span class="cm">/***/</span><span class="n">test_pipe</span><span class="p">.</span><span class="n">txt</span>
<span class="mi">3</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">4</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</p></li></ol>
</div>
<div class="section" id="EN-US_TOPIC_0000001188482140__section9408862212"><h4 class="sectiontitle">Exporting Data Through Multi-Process Pipes</h4><p id="EN-US_TOPIC_0000001188482140__p842218541105">GDS also supports importing and exporting data through multi-process pipes. That is, one foreign table corresponds to multiple GDSs.</p>
<p id="EN-US_TOPIC_0000001188482140__p379766184215">The following takes exporting a local file as an example.</p>
<ol id="EN-US_TOPIC_0000001188482140__ol20408174771311"><li id="EN-US_TOPIC_0000001188482140__li9408104721314"><span>Start multiple GDSs.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen9565938191215"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">gds</span><span class="w"> </span><span class="o">-</span><span class="n">d</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_data</span><span class="o">/</span><span class="w"> </span><span class="o">-</span><span class="n">D</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="mi">192</span><span class="p">.</span><span class="mi">168</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">7789</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_log</span><span class="o">/</span><span class="n">aa</span><span class="p">.</span><span class="n">log</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="o">-</span><span class="n">D</span>
<span class="n">gds</span><span class="w"> </span><span class="o">-</span><span class="n">d</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_data_1</span><span class="o">/</span><span class="w"> </span><span class="o">-</span><span class="n">D</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="w"> </span><span class="mi">192</span><span class="p">.</span><span class="mi">168</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">7790</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_log</span><span class="o">/</span><span class="n">aa</span><span class="p">.</span><span class="n">log</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="o">-</span><span class="n">D</span>
</pre></div></td></tr></table></div>
</div>
<p id="EN-US_TOPIC_0000001188482140__p133521190431">If you need to set the timeout interval of a pipe, use the <strong id="EN-US_TOPIC_0000001188482140__b1215171642311">--pipe-timeout</strong> parameter.</p>
</p></li><li id="EN-US_TOPIC_0000001188482140__li1725917883612"><span>Export data.</span><p><ol type="a" id="EN-US_TOPIC_0000001188482140__ol412981103614"><li id="EN-US_TOPIC_0000001188482140__li141296112362">Log in to the database and create an internal table.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen1332163311134"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="p">(</span><span class="n">id</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">sex</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="nb">text</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li1884141515444">Write data.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen1987415714411"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="k">values</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="s1">'11111111111111'</span><span class="p">);</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="k">values</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="s1">'11111111111111'</span><span class="p">);</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="k">values</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="s1">'11111111111111'</span><span class="p">);</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">test_pipe</span><span class="w"> </span><span class="k">values</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="s1">'11111111111111'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li13149191411361">Create a write-only foreign table.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen393885161317"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">foreign_test_pipe_tw</span><span class="p">(</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">age</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="k">not</span><span class="w"> </span><span class="k">null</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="n">SERVER</span><span class="w"> </span><span class="n">gsmpp_server</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span><span class="k">LOCATION</span><span class="w"> </span><span class="s1">'gsfs://192.168.0.1:7789/|gsfs://192.168.0.1:7790/'</span><span class="p">,</span><span class="w"> </span><span class="n">FORMAT</span><span class="w"> </span><span class="s1">'text'</span><span class="p">,</span><span class="w"> </span><span class="k">DELIMITER</span><span class="w"> </span><span class="s1">','</span><span class="p">,</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">EOL</span><span class="w"> </span><span class="s1">'0x0a'</span><span class="w"> </span><span class="p">,</span><span class="n">file_type</span><span class="w"> </span><span class="s1">'pipe'</span><span class="p">,</span><span class="w"> </span><span class="n">auto_create_pipe</span><span class="w"> </span><span class="s1">'false'</span><span class="p">)</span><span class="w"> </span><span class="k">WRITE</span><span class="w"> </span><span class="k">ONLY</span><span class="p">;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li20505220153614">Execute the export statement. In this case, the statements are blocked.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen123719184149"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">foreign_test_pipe_tw</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">test_pipe</span><span class="p">;</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001188482140__li14257155511383"><span>Export data through the GDS pipes.</span><p><ol type="a" id="EN-US_TOPIC_0000001188482140__ol199213263914"><li id="EN-US_TOPIC_0000001188482140__li945575119395">Log in to GDS and go to each GDS data directory.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen465135973914"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">cd</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_data</span><span class="o">/</span><span class="w"> </span>
<span class="n">cd</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_data_1</span><span class="o">/</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li12492811203920">Create a pipe. If <strong id="EN-US_TOPIC_0000001188482140__b48481629192314">auto_create_pipe</strong> is set to <strong id="EN-US_TOPIC_0000001188482140__b19848182913237">true</strong>, skip this step.<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen438619016152"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">mkfifo</span><span class="w"> </span><span class="n">postgres_public_foreign_test_pipe_tw</span><span class="p">.</span><span class="n">pipe</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</li><li id="EN-US_TOPIC_0000001188482140__li95801819203911">Read each pipe and write the new file to the pipes.<pre class="screen" id="EN-US_TOPIC_0000001188482140__screen5662812151">cat postgres_public_foreign_test_pipe_tw.pipe &gt; postgres_public_foreign_test_pipe_tw.txt</pre>
</li></ol>
</p></li><li id="EN-US_TOPIC_0000001188482140__li1724519137161"><span>Verify the exported data.</span><p><div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen853063471517"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">cat</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_data</span><span class="o">/</span><span class="n">postgres_public_foreign_test_pipe_tw</span><span class="p">.</span><span class="n">txt</span>
<span class="mi">3</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
</pre></div></td></tr></table></div>
</div>
<div class="codecoloring" codetype="Sql" id="EN-US_TOPIC_0000001188482140__screen219884691516"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">cat</span><span class="w"> </span><span class="cm">/***/</span><span class="n">gds_data_1</span><span class="o">/</span><span class="n">postgres_public_foreign_test_pipe_tw</span><span class="p">.</span><span class="n">txt</span>
<span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
<span class="mi">4</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">11111111111111</span>
</pre></div></td></tr></table></div>
</div>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_04_0261.html">Using GDS to Export Data to a Remote Server</a></div>
</div>
</div>
View Git Blame Copy Permalink