forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
68 lines
8.5 KiB
HTML
68 lines
8.5 KiB
HTML
<a name="mrs_01_24455"></a><a name="mrs_01_24455"></a>
|
|
|
|
<h1 class="topictitle1">Adapting Sqoop 1.4.7 to MRS 3.<em id="i5518260455">x</em> Clusters</h1>
|
|
<div id="body1595845458238"><p id="mrs_01_24455__p1541519517217">Sqoop is a tool designed for efficiently transmitting a large amount of data between Apache Hadoop and structured databases (such as relational databases). Customers need to use Sqoop to migrate data in MRS. However, MRS of an earlier version does not provide Sqoop. This section describes how to install and use Sqoop. In MRS 3.1.0 or later, you can select the Sqoop component during cluster creation.</p>
|
|
<div class="section" id="mrs_01_24455__section92229114613"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_24455__p17339644124415">The MRS client and the JDK environment have been installed.</p>
|
|
<p id="mrs_01_24455__p946241869"><span><img id="mrs_01_24455__image0562185233920" src="en-us_image_0000001295930368.png"></span></p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24455__section66586351184"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24455__ol189108491818"><li id="mrs_01_24455__li69105491988"><span><a href="http://archive.apache.org/dist/sqoop/1.4.7/" target="_blank" rel="noopener noreferrer">Download</a> the open-source <strong id="mrs_01_24455__b178415368508">sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz</strong> package.</span></li><li id="mrs_01_24455__li13478210243"><span>Save the downloaded package to the <strong id="mrs_01_24455__b129236595516">/opt/Bigdata/client</strong> directory on the node where the MRS client is installed and decompress it.</span><p><p id="mrs_01_24455__p92511716101617"><strong id="mrs_01_24455__b181431912168">tar zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz</strong></p>
|
|
</p></li><li id="mrs_01_24455__l945d89eadc024485b6c0bd53e78cfce4"><span>Download the MySQL JDBC driver <strong id="mrs_01_24455__b1061018285313">mysql-connector-java-<em id="mrs_01_24455__i19820144105316">xxx</em>.jar</strong> from the MySQL official website. For details about how to select the MySQL JDBC driver, see the following table.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24455__table13581236105614" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Version information</caption><thead align="left"><tr id="mrs_01_24455__row1335812363567"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.3.2.1.2.3.1.1"><p id="mrs_01_24455__p1735833618565">JDBC Driver Version</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.3.2.1.2.3.1.2"><p id="mrs_01_24455__p123587364565">MySQL Version</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24455__row535816362569"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24455__p113584368569">Connector/J 5.1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24455__p13581136125613">MySQL 4.1, MySQL 5.0, MySQL 5.1, and MySQL 6.0 alpha</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24455__row1835853615561"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24455__p1935873620566">Connector/J 5.0</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24455__p9359436145613">MySQL 4.1, MySQL 5.0 servers, and distributed transaction (XA)</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24455__row135914365569"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24455__p1359736155615">Connector/J 3.1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24455__p8359123635614">MySQL 4.1, MySQL 5.0 servers, and MySQL 5.0 except distributed transaction (XA)</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24455__row135953655619"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.1 "><p id="mrs_01_24455__p635915367566">Connector/J 3.0</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.3.2.1.2.3.1.2 "><p id="mrs_01_24455__p83591036155610">MySQL 3.x and MySQL 4.1</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_24455__li18231050111210"><span>Put the MySQL driver package in the <strong id="mrs_01_24455__b1758434413491">/opt/Bigdata/client/sqoop-1.4.7.bin__hadoop-2.6.0/lib</strong> directory of Sqoop and modify the owner group and permission of the JAR package. For details, see the owner group and permission of <strong id="mrs_01_24455__b1419017405557">omm:wheel</strong> and <strong id="mrs_01_24455__b567954320557">755</strong> in <a href="#mrs_01_24455__fig71925061212">Figure 1</a>.</span><p><div class="fignone" id="mrs_01_24455__fig71925061212"><a name="mrs_01_24455__fig71925061212"></a><a name="fig71925061212"></a><span class="figcap"><b>Figure 1 </b>Owner group and permission of the MySQL driver package</span><br><span><img id="mrs_01_24455__image1542355219111" src="en-us_image_0000001295770408.png"></span></div>
|
|
</p></li><li id="mrs_01_24455__li682643102817"><span>Replace the JAR package in the <strong id="mrs_01_24455__b79151012195710">lib</strong> directory of Sqoop with that starting with <strong id="mrs_01_24455__b168162713572">jackson</strong> in the <strong id="mrs_01_24455__b14445029105714">lib</strong> directory of Hive on the MRS client, for example, <strong id="mrs_01_24455__b2310948165717">/opt/Bigdata/client/Hive/Beeline/lib</strong>.</span><p><div class="fignone" id="mrs_01_24455__fig55818155143"><span class="figcap"><b>Figure 2 </b>JAR package starting with <strong id="mrs_01_24455__b84751910586">jackson</strong></span><br><span><img id="mrs_01_24455__image98051982142" src="en-us_image_0000001349090041.png"></span></div>
|
|
</p></li><li id="mrs_01_24455__li0711195781317"><span>Copy the <strong id="mrs_01_24455__b9182122455817">jline</strong> package from the <strong id="mrs_01_24455__b2438174275813">/opt/Bigdata/client/Hive/Beeline/lib</strong> directory of the MRS Hive client to the <strong id="mrs_01_24455__b12471185935818">lib</strong> directory of Sqoop.</span></li><li id="mrs_01_24455__li2748618156"><span>Run the <strong id="mrs_01_24455__b1034311915914">vim $JAVA_HOME/jre/lib/security/java.policy</strong> command to add the following configuration:</span><p><p id="mrs_01_24455__p186668151610"><strong id="mrs_01_24455__b937987161617">permission javax.management.MBeanTrustPermission "register";</strong></p>
|
|
</p></li><li id="mrs_01_24455__li98574221617"><span>Run the following commands to go to the <strong id="mrs_01_24455__b71011400595">conf</strong> directory of the Sqoop and add the configuration items of variables:</span><p><p id="mrs_01_24455__p153971217122412"><strong id="mrs_01_24455__b1392911287167">cd /opt/Bigdata/client/sqoop-1.4.7.bin__hadoop-2.6.0/conf</strong></p>
|
|
<p id="mrs_01_24455__p155913475169"><strong id="mrs_01_24455__b13637751111613">cp sqoop-env-template.sh sqoop-env.sh</strong></p>
|
|
</p></li><li id="mrs_01_24455__li10791351161718"><span>Run the <strong id="mrs_01_24455__b1830914198171">vim sqoop-env.sh</strong> command to set the environment variables of Sqoop. Change the Hadoop and Hive directories as required.</span><p><pre class="screen" id="mrs_01_24455__screen177355584587">export HADOOP_COMMON_HOME=/opt/Bigdata/client/HDFS/hadoop
|
|
export HADOOP_MAPRED_HOME=/opt/Bigdata/client/HDFS/hadoop
|
|
export HIVE_HOME=/opt/Bigdata/MRS_1.9.X/install/FusionInsight-Hive-<em id="mrs_01_24455__i1692723512918">3.1.0</em>/hive (Enter the actual path.)
|
|
export HIVE_CONF_DIR=/opt/Bigdata/client/Hive/config
|
|
export HCAT_HOME=/opt/Bigdata/client/Hive/HCatalog</pre>
|
|
<div class="fignone" id="mrs_01_24455__fig236714226810"><span class="figcap"><b>Figure 3 </b>Setting environment variables of Sqoop</span><br><span><img id="mrs_01_24455__image174347612820" src="en-us_image_0000001438712537.png"></span></div>
|
|
</p></li><li id="mrs_01_24455__li363433181719"><span>Build the sqoop script. For example:</span><p><pre class="screen" id="mrs_01_24455__screen5340017131819">/opt/Bigdata/FusionInsight_Current/1_19_SqoopClient/install/FusionInsight-Sqoop-1.4.7/bin/sqoop import
|
|
--connect jdbc:mysql://192.168.0.183:3306/test
|
|
--driver com.mysql.jdbc.Driver
|
|
--username 'root'
|
|
--password '<em id="mrs_01_24455__i13309113104316">xxx</em>'
|
|
--query "SELECT id, name FROM tbtest WHERE \$CONDITIONS"
|
|
--hcatalog-database default
|
|
--hcatalog-table test
|
|
--num-mappers 1</pre>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24453.html">Using Sqoop</a></div>
|
|
</div>
|
|
</div>
|
|
|