vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide/mrs_01_24454.html
Yang, Tong 6182f91ba8 MRS component operation guide_normal 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-09 14:55:21 +00:00

219 lines
24 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<a name="mrs_01_24454"></a><a name="mrs_01_24454"></a>
<h1 class="topictitle1">Using Sqoop from Scratch</h1>
<div id="body1595845458238"><p id="mrs_01_24454__p271318616124">Sqoop is an open-source tool for transferring data between Hadoop (Hive) and traditional databases (such as MySQL and PostgreSQL). It can transfer data from a relational database (such as MySQL, Oracle, and PostgreSQL) to HDFS of Hadoop and the other way around.</p>
<div class="section" id="mrs_01_24454__section19493164011271"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24454__ul122727255111"><li id="mrs_01_24454__li119023261116">You have selected the Sqoop component when creating a cluster of MRS 3.1.0 or later.</li><li id="mrs_01_24454__li827212259113">You have installed the client. For details, see <a href="mrs_01_2127.html">Installing a Client (Version 3.x or Later)</a>. For example, the installation directory of the client is <strong id="mrs_01_24454__b1650068707114941">/opt/client</strong>. The client directory in the following operations is an example. Change it to the actual installation directory.</li></ul>
</div>
<div class="section" id="mrs_01_24454__section4468434234"><h4 class="sectiontitle">Exporting Data From HDFS to MySQL Using the <strong id="mrs_01_24454__b1286145454">sqoop export</strong> Command</h4><ol id="mrs_01_24454__ol91839444238"><li id="mrs_01_24454__li99217282412"><span>Log in to the node where the client is located.</span></li><li id="mrs_01_24454__li109168252241"><span>Run the following command to initialize environment variables:</span><p><p id="mrs_01_24454__p19654172612410"><strong id="mrs_01_24454__b551203313258">source /opt/client/bigdata_env</strong></p>
</p></li><li id="mrs_01_24454__li115201937182512"><span>Run the following command to operate the Sqoop client:</span><p><p id="mrs_01_24454__p17304549263"><strong id="mrs_01_24454__b0461131782610">sqoop export --connect jdbc:mysql://10.100.231.134:3306/test --username root --password xxxxxx --table component13 -export-dir hdfs://hacluster/user/hive/warehouse/component_test3 --fields-terminated-by ',' -m 1</strong></p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24454__table1935416343262" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_24454__row63546345263"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.3.2.3.2.2.2.3.1.1"><p id="mrs_01_24454__p10355193412269">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="70%" id="mcps1.3.3.2.3.2.2.2.3.1.2"><p id="mrs_01_24454__p935514341262">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24454__row33551345265"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p13554346261">-direct</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p235583482615">Imports data to a relational database using a database import tool, for example, mysqlimport of MySQL, more efficient than the JDBC connection mode.</p>
</td>
</tr>
<tr id="mrs_01_24454__row435553402612"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p235510345260">-export-dir &lt;dir&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p035512342267">Specifies the source directory for storing data in the HDFS.</p>
</td>
</tr>
<tr id="mrs_01_24454__row764065102911"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p20640251295">-m or -num-mappers &lt;n&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p1364118582912">Starts <em id="mrs_01_24454__i2506865318">n</em> (4 by default) maps to import data concurrently. The value cannot be greater than the maximum number of maps in a cluster.</p>
</td>
</tr>
<tr id="mrs_01_24454__row483935202916"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p198391156297">-table &lt;table-name&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p1583915132915">Specifies the relational database table to be imported.</p>
</td>
</tr>
<tr id="mrs_01_24454__row83619615290"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p13361563297">-update-key &lt;col-name&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p153612620294">Specifies the column used for updating the existing data in a relational database.</p>
</td>
</tr>
<tr id="mrs_01_24454__row9925201310329"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p1292501316321">-update-mode &lt;mode&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p7925151310325">Specifies how updates are performed. The value can be <strong id="mrs_01_24454__b134135133185">updateonly</strong> or <strong id="mrs_01_24454__b176516141816">allowinsert</strong>. This parameter is used only when the relational data table does not contain the data record to be imported. For example, if the HDFS data to be imported to the destination table contains a data record <strong id="mrs_01_24454__b1137714842119">id=1</strong> and the table contains an existing data record <strong id="mrs_01_24454__b85981133192215">id=2</strong>, the update will fail.</p>
</td>
</tr>
<tr id="mrs_01_24454__row1142121423214"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p20142161412328">-input-null-string &lt;null-string&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p3142151417320">This parameter is optional. If it is not specified, <strong id="mrs_01_24454__b19657385274">null</strong> will be used.</p>
</td>
</tr>
<tr id="mrs_01_24454__row1753451418320"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p1553481413214">-input-null-non-string &lt;null-string&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p1349161103316">This parameter is optional. If it is not specified, <strong id="mrs_01_24454__b16696194818274">null</strong> will be used.</p>
</td>
</tr>
<tr id="mrs_01_24454__row22072218339"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p62076213334">-staging-table &lt;staging-table-name&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p1425543111392">Creates a table with the same data structure as the destination table for storing data before it is imported to the destination table.</p>
<p id="mrs_01_24454__p14519132453513">This parameter ensures the transaction security when data is imported to a relational database table. Due to multiple transactions during an import, this parameter can prevent other transactions from being affected when one transaction fails. For example, the imported data is incorrect or duplicate records exist.</p>
</td>
</tr>
<tr id="mrs_01_24454__row119891622173318"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.3.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p119894223336">-clear-staging-table</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.3.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p298962233319">Clears data in the staging table before data is imported if the staging-table is not empty.</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li></ol>
</div>
<div class="section" id="mrs_01_24454__section1294043815363"><h4 class="sectiontitle">Importing Data from MySQL to Hive Using the sqoop import Command</h4><ol id="mrs_01_24454__ol10810181011370"><li id="mrs_01_24454__li15810710153715"><span>Log in to the node where the client is located.</span></li><li id="mrs_01_24454__li881091012370"><span>Run the following command to initialize environment variables:</span><p><p id="mrs_01_24454__p158101410103712"><strong id="mrs_01_24454__b198105107372">source /opt/client/bigdata_env</strong></p>
</p></li><li id="mrs_01_24454__li158105103373"><span>Run the following command to operate the Sqoop client:</span><p><p id="mrs_01_24454__p4950225153718"><strong id="mrs_01_24454__b49641330143715">sqoop import --connect jdbc:mysql://10.100.231.134:3306/test --username root --password xxxxxx --table component --hive-import --hive-table component_test2 --delete-target-dir --fields-terminated-by "," -m 1 --as-textfile</strong></p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24454__table69791040193719" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_24454__row1980040193716"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.4.2.3.2.2.2.3.1.1"><p id="mrs_01_24454__p5980144017379">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="70%" id="mcps1.3.4.2.3.2.2.2.3.1.2"><p id="mrs_01_24454__p3980154013720">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24454__row2980240113717"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p115245103719">-append</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p162164511370">Appends data to an existing dataset in the HDFS. Once this parameter is used, Sqoop imports data to a temporary directory, renames the temporary file where the data is stored, and moves the file to a formal directory to avoid duplicate file names in the directory.</p>
</td>
</tr>
<tr id="mrs_01_24454__row17854184723710"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p28558479378">-as-avrodatafile</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p1285512472374">Imports data to a data file in the Avro format.</p>
</td>
</tr>
<tr id="mrs_01_24454__row1465104815379"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p965144816375">-as-sequencefile</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p1651748163715">Imports data to a sequence file.</p>
</td>
</tr>
<tr id="mrs_01_24454__row1926104817377"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p5261174883713">-as-textfile</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p14261174863718">Import data to a text file. After the text file is generated, you can run SQL statements in Hive to query the result.</p>
</td>
</tr>
<tr id="mrs_01_24454__row54531448193713"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p445310485374">-boundary-query &lt;statement&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p1787412258308">Specifies the SQL statement for performing boundary query. Before importing data, use a SQL statement to obtain a result set and import the data in the result set. The data format can be <strong id="mrs_01_24454__b1887442523014">-boundary-query 'select id,creationdate from person where id = 3'</strong> (indicating a data record whose ID is 3) or <strong id="mrs_01_24454__b19874625103010">select min(&lt;split-by&gt;), max(&lt;split-by&gt;) from &lt;table name&gt;</strong>.</p>
<p id="mrs_01_24454__p851112455405">The fields to be queried cannot contain fields whose data type is string. Otherwise, the error message "java.sql.SQLException: Invalid value for getLong()" is displayed.</p>
</td>
</tr>
<tr id="mrs_01_24454__row2030893319405"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p4308173316407">-columns&lt;col,col,col...&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p1830813337408">Specifies the fields to be imported. The format is <strong id="mrs_01_24454__b9809125811152"><em id="mrs_01_24454__i69701084166">-Column id</em>,<em id="mrs_01_24454__i15228141321616">Username</em></strong>.</p>
</td>
</tr>
<tr id="mrs_01_24454__row1495163314010"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p1049573354014">-direct</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p4495133144019">Imports data to a relational database using a database import tool, for example, mysqlimport of MySQL, more efficient than the JDBC connection mode.</p>
</td>
</tr>
<tr id="mrs_01_24454__row969343354012"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p2693333124019">-direct-split-size</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p13693113313406">Splits the imported streams by byte. Especially when data is imported from PostgreSQL using the direct mode, a file that reaches the specified size can be divided into several independent files.</p>
</td>
</tr>
<tr id="mrs_01_24454__row1188033394014"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p588083324019">-inline-lob-limit</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p818317333419">Sets the maximum value of an inline LOB.</p>
</td>
</tr>
<tr id="mrs_01_24454__row55323484012"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p1153113417408">-m or -num-mappers</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p65414317419">Starts <em id="mrs_01_24454__i39191538105314">n</em> (4 by default) maps to import data concurrently. The value cannot be greater than the maximum number of maps in a cluster.</p>
</td>
</tr>
<tr id="mrs_01_24454__row92281434114011"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p18228103494017">-query, -e&lt;statement&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p2615165853314">Imports data from the query result. To use this parameter, you must specify the <strong id="mrs_01_24454__b1048319459210">-target-dir</strong> and <strong id="mrs_01_24454__b2666154810214">-hive-table</strong> parameters and use the query statement containing the WHERE clause as well as $CONDITIONS.</p>
<p id="mrs_01_24454__p171631111428">Example: <strong id="mrs_01_24454__b47011342515">-query'select * from person where $CONDITIONS' -target-dir /user/hive/warehouse/person -hive-table person</strong></p>
</td>
</tr>
<tr id="mrs_01_24454__row9695323428"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p156954254220">-split-by&lt;column-name&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p11696622420">Specifies the column of a table used to split work units. Generally, the column name is followed by the primary key ID.</p>
</td>
</tr>
<tr id="mrs_01_24454__row48921429421"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p78927274210">-table &lt;table-name&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p11538925144217">Specifies the relational database table from which data is obtained.</p>
</td>
</tr>
<tr id="mrs_01_24454__row198115314429"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p98253174214">-target-dir &lt;dir&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p08216364215">Specifies the HDFS path.</p>
</td>
</tr>
<tr id="mrs_01_24454__row102567344216"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p1225612317423">-warehouse-dir &lt;dir&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p1925623104212">Specifies the directory for storing data to be imported. This parameter is applicable when data is imported to HDFS but cannot be used when you import data to Hive directories. This parameter cannot be used together with <strong id="mrs_01_24454__b1843693523015">-target-dir</strong>.</p>
</td>
</tr>
<tr id="mrs_01_24454__row154084364217"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p1440814313426">-where</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p61971518435">Specifies the WHERE clause when data is imported from a relational database, for example, <strong id="mrs_01_24454__b1920493133218">-where 'id = 2'</strong>.</p>
</td>
</tr>
<tr id="mrs_01_24454__row29405142439"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p8940121416436">-z,-compress</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p189405148437">Compresses sequence, text, and Avro data files using the GZIP compression algorithm. Data is not compressed by default.</p>
</td>
</tr>
<tr id="mrs_01_24454__row6171121518436"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p12171111544313">–compression-codec</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p01714159438">Specifies the Hadoop compression codec. GZIP is used by default.</p>
</td>
</tr>
<tr id="mrs_01_24454__row938818155432"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p6388171513431">–null-string &lt;null-string&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p978615331812">Specifies the string to be interpreted as <strong id="mrs_01_24454__b3294140144212">NULL</strong> for string columns.</p>
</td>
</tr>
<tr id="mrs_01_24454__row1062914408437"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p1562944018439">–null-non-string&lt;null-string&gt;</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p05571122921">Specifies the string to be interpreted as null for non-string columns. If this parameter is not specified, <strong id="mrs_01_24454__b098114164216">NULL</strong> will be used.</p>
</td>
</tr>
<tr id="mrs_01_24454__row263271034413"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p1633131074411">-check-column (col)</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p14633141019442">Specifies the column for checking incremental data import, for example, <strong id="mrs_01_24454__b20147142094015">id</strong>.</p>
</td>
</tr>
<tr id="mrs_01_24454__row168371910204411"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p1533542174115">-incremental (mode) append</p>
<p id="mrs_01_24454__p8837510154413">or last modified</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p37993124218">Incrementally imports data.</p>
<p id="mrs_01_24454__p1685417545416"><strong id="mrs_01_24454__b250795563818">append</strong>: appends records, for example, appending records that are greater than the value specified by <strong id="mrs_01_24454__b33662228397">last-value</strong>.</p>
<p id="mrs_01_24454__p683741014416"><strong id="mrs_01_24454__b3997226104015">lastmodified</strong>: appends data that is modified after the date specified by <strong id="mrs_01_24454__b1539641754114">last-value</strong>.</p>
</td>
</tr>
<tr id="mrs_01_24454__row71091341194419"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.4.2.3.2.2.2.3.1.1 "><p id="mrs_01_24454__p18109641154414">-last-value (value)</p>
</td>
<td class="cellrowborder" valign="top" width="70%" headers="mcps1.3.4.2.3.2.2.2.3.1.2 "><p id="mrs_01_24454__p10982452164414">Specifies the maximum value (greater than the specified value) of the column after the last import. This parameter can be set as required.</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li></ol>
</div>
<div class="section" id="mrs_01_24454__section5349195324212"><h4 class="sectiontitle">Sqoop Usage Example</h4><ul id="mrs_01_24454__ul14935173594412"><li id="mrs_01_24454__li293513357442">Importing data from MySQL to HDFS using the <strong id="mrs_01_24454__b19405195615114">sqoop import</strong> command<p id="mrs_01_24454__p83041613439"><strong id="mrs_01_24454__b1148394884115">sqoop import --connect jdbc:mysql://10.100.231.134:3306/test --username root --password </strong><em id="mrs_01_24454__i12486144819419">xxx</em><strong id="mrs_01_24454__b748454844111"> --query 'SELECT * FROM component where $CONDITIONS and component_id ="MRS 1.0_002"' --target-dir /tmp/component_test --delete-target-dir --fields-terminated-by "," -m 1 --as-textfile</strong></p>
</li><li id="mrs_01_24454__li593514356442">Exporting data from OBS to MySQL using the <strong id="mrs_01_24454__b148119251325">sqoop export</strong> command<p id="mrs_01_24454__p73865459438"><strong id="mrs_01_24454__b1254517105422">sqoop export --connect jdbc:mysql://10.100.231.134:3306/test --username root --password </strong><em id="mrs_01_24454__i254831014211">xxx</em><strong id="mrs_01_24454__b15462102428"> --table component14 -export-dir obs://obs-file-bucket/xx/part-m-00000 --fields-terminated-by ',' -m 1</strong></p>
</li><li id="mrs_01_24454__li79351835114414">Importing data from MySQL to OBS using the <strong id="mrs_01_24454__b18921031222">sqoop import</strong> command<p id="mrs_01_24454__p157011889444"><strong id="mrs_01_24454__b10967286428">sqoop import --connect jdbc:mysql://10.100.231.134:3306/test --username root --password </strong><em id="mrs_01_24454__i48701228114214">xxx</em><strong id="mrs_01_24454__b139672816424"> --table component --target-dir obs://obs-file-bucket/xx --delete-target-dir --fields-terminated-by "," -m 1 --as-textfile</strong></p>
</li><li id="mrs_01_24454__li99351735164416">Importing data from MySQL to OBS tables outside Hive<p id="mrs_01_24454__p27018854413"><a name="mrs_01_24454__li99351735164416"></a><a name="li99351735164416"></a><strong id="mrs_01_24454__b04262474424">sqoop import --connect jdbc:mysql://10.100.231.134:3306/test --username root --password </strong><em id="mrs_01_24454__i1818944812427">xxx</em><strong id="mrs_01_24454__b12426174724215"> --table component --hive-import --hive-table component_test01 --fields-terminated-by "," -m 1 --as-textfile</strong></p>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24453.html">Using Sqoop</a></div>
</div>
</div>
View Git Blame Copy Permalink