forked from docs/doc-exports
Lu, Huayi
a24ca60074
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Lu, Huayi <luhuayi@huawei.com> Co-committed-by: Lu, Huayi <luhuayi@huawei.com>
122 lines
18 KiB
HTML
122 lines
18 KiB
HTML
<a name="EN-US_TOPIC_0000001099633040"></a><a name="EN-US_TOPIC_0000001099633040"></a>
|
|
|
|
<h1 class="topictitle1">Data Migration to GaussDB(DWS)</h1>
|
|
<div id="body8662426"><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p1113234919179"><span id="EN-US_TOPIC_0000001099633040__ph12421185745313">GaussDB(DWS)</span> provides flexible methods for importing data. You can import data from different sources to <span id="EN-US_TOPIC_0000001099633040__ph84281057155311">GaussDB(DWS)</span>. The features of each method are listed in <a href="#EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_table10273436143718">Table 1</a>. You can select a method as required. You are advised to use <span id="EN-US_TOPIC_0000001099633040__ph1755183719118">GaussDB(DWS)</span> with Cloud Data Migration (CDM), and Data Lake Factory (DLF). CDM is used for batch data migration, and DLF orchestrates and schedules the entire ETL process and provides a visualized development environment.</p>
|
|
<div class="note" id="EN-US_TOPIC_0000001099633040__note195562296559"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="EN-US_TOPIC_0000001099633040__ul1775310215573"><li id="EN-US_TOPIC_0000001099633040__li1779132205718">DRS, CDM, OBS, and MRS are cloud services.</li><li id="EN-US_TOPIC_0000001099633040__li69301637125813">GDS, DSC, and <strong id="EN-US_TOPIC_0000001099633040__b723572282310">gs_restore</strong>, and <strong id="EN-US_TOPIC_0000001099633040__b757465011165">gs_dump</strong> are internal tools.</li></ul>
|
|
</div></div>
|
|
|
|
<div class="tablenoborder"><a name="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_table10273436143718"></a><a name="en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_table10273436143718"></a><table cellpadding="4" cellspacing="0" summary="" id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_table10273436143718" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Data migration methods</caption><thead align="left"><tr id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_row1627416369375"><th align="left" class="cellrowborder" valign="top" width="24.15%" id="mcps1.3.3.2.5.1.1"><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p3274133613372">Data Migration Mode</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="17.02%" id="mcps1.3.3.2.5.1.2"><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p12274143611371">Supported Data Source/Database</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="38.49%" id="mcps1.3.3.2.5.1.3"><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p78031146143918">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20.34%" id="mcps1.3.3.2.5.1.4"><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p152740367379">Advantage</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_row152747367379"><td class="cellrowborder" valign="top" width="24.15%" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p17274103611372"><a href="dws_04_0181.html">Importing Data from OBS in Parallel</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.02%" headers="mcps1.3.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p144051814119">TEXT, CSV, ORC, and CarbonData data file formats</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.49%" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p180334663913">You can import data in TEXT, CSV, ORC, or CarbonData format from OBS to <span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_ph14538135614212">GaussDB(DWS)</span> for query, and can remotely read data from OBS.</p>
|
|
<p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_p1449517184610">It is recommended for <span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001145651023_ph15721958174314">GaussDB(DWS)</span>.</p>
|
|
</td>
|
|
<td class="cellrowborder" rowspan="2" valign="top" width="20.34%" headers="mcps1.3.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p1227413363374">This method features high performance and flexible scale-out.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_row9466165383511"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_p102809945217"><a href="dws_04_0189.html">Using GDS to Import Data from a Remote Server</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p35972011183617">TEXT and CSV data file formats</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p8803246143911">You can use the GDS tool provided by <span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001145651023_ph123261154122518">GaussDB(DWS)</span> to import data from the remote server to <span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001145651023_ph1712113508258">GaussDB(DWS)</span> in parallel. Multiple DNs are used for the import. This method is efficient and applicable to importing a large amount of data to the database.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_row10229161163513"><td class="cellrowborder" valign="top" width="24.15%" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_p568681813351"><a href="dws_04_0210.html">Importing Data from MRS to a Cluster</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.02%" headers="mcps1.3.3.2.5.1.2 "><ul id="EN-US_TOPIC_0000001099633040__ul17601122510517"><li id="EN-US_TOPIC_0000001099633040__li560182525111">Hive</li><li id="EN-US_TOPIC_0000001099633040__li176012025205117">Spark</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.49%" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_p19686181811355">You can configure a <span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001145651023_ph884632792510">GaussDB(DWS)</span> cluster to connect to an MRS cluster, and read data from the HDFS of MRS to <span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001145651023_ph1735794592518">GaussDB(DWS)</span>.</p>
|
|
<div class="note" id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_note10687418143516"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_p86877189358">This import method is not supported currently.</p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.34%" headers="mcps1.3.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_p92330118358">This method features high performance and flexible scale-out.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001099633040__row44092055131619"><td class="cellrowborder" valign="top" width="24.15%" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__p103325720179"><a href="dws_04_0949.html">Importing Data from One GaussDB(DWS) Cluster to Another</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.02%" headers="mcps1.3.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001099633040__p3332157201716">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.49%" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__p10333127101714">Data communication between two GaussDB(DWS) clusters is supported. You can use foreign tables to access and import data across GaussDB(DWS) clusters.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.34%" headers="mcps1.3.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001099633040__p73334781716">This method is applicable to data synchronization between multiple GaussDB(DWS) clusters.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001099633040__row26104493165"><td class="cellrowborder" valign="top" width="24.15%" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__p131192016172"><a href="dws_04_0208.html">Using the gsql Meta-Command \COPY to Import Data</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.02%" headers="mcps1.3.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001099633040__p1731192014170">Local files</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.49%" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__p431162021715">Unlike the SQL <strong id="EN-US_TOPIC_0000001099633040__b8721956113911">COPY</strong> statement, the <strong id="EN-US_TOPIC_0000001099633040__b872313566398">\copy</strong> command can be used to read data from or write data to only local files on a gsql client.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.34%" headers="mcps1.3.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001099633040__p33222019173">This method is easy-to-operate and suitable for importing a small amount of data to the database.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001099633040__row1055515362177"><td class="cellrowborder" valign="top" width="24.15%" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__p03105015178"><a href="dws_04_0203.html">Running the COPY FROM STDIN Statement to Import Data</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.02%" headers="mcps1.3.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001099633040__p173205019173">Other files or databases</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.49%" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__p164195019177">When you use Java to develop applications, the CopyManager interface of the JDBC driver is invoked to write data from files or other databases to <span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001145651023_ph6667121019273">GaussDB(DWS)</span>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.34%" headers="mcps1.3.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001099633040__p14125012177">Data is directly written from other databases to <span id="EN-US_TOPIC_0000001099633040__ph164175051710">GaussDB(DWS)</span>. Service data does not need to be stored in files.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_row1127453617378"><td class="cellrowborder" valign="top" width="24.15%" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p13274143610370"><a href="dws_04_0219.html">Using CDM to Migrate Data to GaussDB(DWS)</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.02%" headers="mcps1.3.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p027533615376"></p>
|
|
<ul id="EN-US_TOPIC_0000001099633040__ul33322237441"><li id="EN-US_TOPIC_0000001099633040__li633292354415">MySQL</li><li id="EN-US_TOPIC_0000001099633040__li17332132384413">PostgreSQL</li><li id="EN-US_TOPIC_0000001099633040__li4332152364418">Microsoft SQL Server</li><li id="EN-US_TOPIC_0000001099633040__li1533212324414">Oracle</li><li id="EN-US_TOPIC_0000001099633040__li183322231447">IBM Db2</li><li id="EN-US_TOPIC_0000001099633040__li833262313441">SAP HANA</li><li id="EN-US_TOPIC_0000001099633040__li18332142384414">GaussDB(DWS)</li><li id="EN-US_TOPIC_0000001099633040__li633282314412">Hive</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.49%" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p4422519145022">CDM can migrate various types of data in batches between homogeneous and heterogeneous data sources. CDM migrates data to <span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001145651023_ph1064716136266">GaussDB(DWS)</span> using the <strong id="EN-US_TOPIC_0000001099633040__b9246155212259">COPY</strong> statement or the GDS parallel import method.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.34%" headers="mcps1.3.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p1727503633720">This method supports data import from abundant data sources and is easy-to-operate.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_row115562143311"><td class="cellrowborder" valign="top" width="24.15%" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p14551221153312"><a href="dws_01_0127.html">Using DSC to Migrate SQL Scripts</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.02%" headers="mcps1.3.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p1555172118337">Databases, NoSQL, file systems, and big data platforms</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.49%" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p1280315465394">For details, see the documents of the third-party ETL tool.</p>
|
|
<p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_p118431812345"><span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001145651023_ph129691947132617">GaussDB(DWS)</span> provides the DSC tool to migrate Teradata/Oracle scripts to <span id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001145651023_ph344105219265">GaussDB(DWS)</span>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.34%" headers="mcps1.3.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001099633040__en-us_topic_0000001082926999_en-us_topic_0140501319_en-us_topic_0145255637_p139781232353">This method supports abundant data sources and provides powerful data conversion through OBS.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001099633040__row1599061413221"><td class="cellrowborder" valign="top" width="24.15%" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__p1991121422211"><a href="dws_04_0269.html">Using gs_dump and gs_dumpall to Export Metadata</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.02%" headers="mcps1.3.3.2.5.1.2 "><ul id="EN-US_TOPIC_0000001099633040__ul6884122713717"><li id="EN-US_TOPIC_0000001099633040__li68847276371">Plaintext</li><li id="EN-US_TOPIC_0000001099633040__li2038265822017">Custom</li><li id="EN-US_TOPIC_0000001099633040__li1994149112112">Directory</li><li id="EN-US_TOPIC_0000001099633040__li153212303373">.tar</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.49%" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__p1434365072312"><strong id="EN-US_TOPIC_0000001099633040__b1498172110199">gs_dump</strong> exports a single database or its objects. <strong id="EN-US_TOPIC_0000001099633040__b928412441911">gs_dumpall</strong> exports all databases or global objects in a cluster.</p>
|
|
<p id="EN-US_TOPIC_0000001099633040__p12991414132219">To migrate database information, you can use a tool to import the exported metadata to a target database.</p>
|
|
</td>
|
|
<td class="cellrowborder" rowspan="2" valign="top" width="20.34%" headers="mcps1.3.3.2.5.1.4 "><p id="EN-US_TOPIC_0000001099633040__p10991514192215"></p>
|
|
<p id="EN-US_TOPIC_0000001099633040__p182901946145112">This method is applicable to metadata migration.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="EN-US_TOPIC_0000001099633040__row428944655111"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.5.1.1 "><p id="EN-US_TOPIC_0000001099633040__p152901746125112"><a href="dws_04_0209.html">Using gs_restore to Import Data</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.5.1.2 "><p id="EN-US_TOPIC_0000001099633040__p182900464517">SQL, TMP, and TAR file formats</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.5.1.3 "><p id="EN-US_TOPIC_0000001099633040__p710361761911">During database migration, you can use the gs_restore tool to import the file exported using the gs_dump tool to a GaussDB(DWS) cluster. In this way, metadata, such as table definitions and database object definitions, is imported. The following data needs to be imported:</p>
|
|
<ul id="EN-US_TOPIC_0000001099633040__ul914882413106"><li id="EN-US_TOPIC_0000001099633040__li11148142451011">All database object definitions</li><li id="EN-US_TOPIC_0000001099633040__li1014852419109">Definition of each database object</li><li id="EN-US_TOPIC_0000001099633040__li3148624151015">Definition of each schema</li><li id="EN-US_TOPIC_0000001099633040__li7148132411108">Definition of each table</li></ul>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dws_04_0985.html">Data Migration</a></div>
|
|
</div>
|
|
</div>
|
|
|