vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_1158.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

71 lines
11 KiB
HTML
Raw Blame History

<a name="mrs_01_1158"></a><a name="mrs_01_1158"></a>
<h1 class="topictitle1">loader-tool Usage Example</h1>
<div id="body8662426"><div class="section" id="mrs_01_1158__en-us_topic_0000001173789542_s53976f53098e4700b549391857c87be0"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1158__en-us_topic_0000001173789542_a46998f67d3e3498f9ca5a7169d53df79">loader-tool can be used to create, update, query, and delete a connector or job by using a job template or setting parameters.</p>
<p id="mrs_01_1158__en-us_topic_0000001173789542_a174419346b154b9ab71c897ddecf0d30">This section describes how to use loader-tool in the job template mode. The job of importing data from the SFTP server to HDFS is used as an example.</p>
</div>
<div class="section" id="mrs_01_1158__en-us_topic_0000001173789542_sd1887df48a0e4128b64b98bfe1fcab4f"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_1158__en-us_topic_0000001173789542_a4db437d058ef426fa082d789603feb45">The Loader client has been installed and configured. </p>
</div>
<div class="section" id="mrs_01_1158__en-us_topic_0000001173789542_s70a742ae01854e25b4dadcbf2c5d7207"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1158__en-us_topic_0000001173789542_o298949ec9e134702b546a50e600190ae"><li id="mrs_01_1158__en-us_topic_0000001173789542_l9686d6144a834a60a20fc07536918dcc"><span>Log in to the node where the client is located as the user who installs the client.</span></li><li id="mrs_01_1158__en-us_topic_0000001173789542_l54689f3bcf4b41dfb45fa836e36236e1"><span>Run the following command to go to the directory where loader-tool is located on the Loader client. For example, if the Loader client installation directory is <span class="filepath" id="mrs_01_1158__en-us_topic_0000001173789542_f39019d651058491495a5581565b49c42"><b>/opt/hadoopclient/Loader</b></span>, run the following command:</span><p><p id="mrs_01_1158__en-us_topic_0000001173789542_a86a6a534c0294553a7fcf392a7ced82b"><strong id="mrs_01_1158__en-us_topic_0000001173789542_a3f41f46478fb492ea6a562a588336ddc">cd /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/</strong></p>
</p></li><li id="mrs_01_1158__en-us_topic_0000001173789542_l64a7078a6f224183a33187ec11613e99"><span>Run the following command to modify the existing job template. For example, if the job template <span class="parmname" id="mrs_01_1158__en-us_topic_0000001173789542_pb8237e2b70024de9be712165245e6ecb"><b>sftp-to-hdfs.xml</b></span> already exists in the <span class="filepath" id="mrs_01_1158__en-us_topic_0000001173789542_f7be0d0e3417145378f0fdbf7114dc96a"><b>/opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/job-config/</b></span>, run the following command:</span><p><p id="mrs_01_1158__en-us_topic_0000001173789542_a72dc56791ff845808f574e610aa2442e"><strong id="mrs_01_1158__en-us_topic_0000001173789542_a0b5858ca075942fa9f1f17f88323568a">vi /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/job-config/sftp-to-hdfs.xml</strong></p>
<pre class="screen" id="mrs_01_1158__en-us_topic_0000001173789542_s4621abedc9a24f04a89ea6b28187f11e">&lt;root&gt;
&lt;!-- Database connection information --&gt;
&lt;sqoop.connection name="<strong id="mrs_01_1158__en-us_topic_0000001173789542_aa09ff867a91346dba28b0444a56d03b9">vt_sftp_test</strong>" type="sftp-connector"&gt;
&lt;connection.sftpServerIp&gt;10.96.26.111&lt;/connection.sftpServerIp&gt;
&lt;connection.sftpServerPort&gt;22&lt;/connection.sftpServerPort&gt;
&lt;connection.sftpUser&gt;root&lt;/connection.sftpUser&gt;
&lt;connection.sftpPassword&gt;d2NjX2NyeXB0ATQxNDU1MzVGNDM0MjQzOzMzMzkzOTMwMzI0NTM5MzQzOTM1Mzk0NTMwMzIzNTM4NDEzNzQ2MzIzNjQyMzMzMDM4MzMzNzQ1MzYzODQxMzQ7OzMyMzUzMDMwO0EzMTUzM0ExNTAyNDhENzE3QTRBRTlCQkRBQzlFRkFEOzYyOEE4NTlDODc2MkMyNzU7NTc0MzQzNUY0MzUyNTk1MDU0NUY0NDQ1NDY0MTU1NEM1NDVGNDQ0RjRENDE0OTRFOzMwOzMxMzQzNTM2MzMzMTMyMzgzMzMzMzIzNzMwOw&lt;/connection.sftpPassword&gt;
&lt;/sqoop.connection&gt;
&lt;!--Job name, globally unique.--&gt;
&lt;sqoop.job name="Sftp.to.Hdfs" type="IMPORT" queue="default" priority="NORMAL"&gt;
&lt;data.source connectionName="<strong id="mrs_01_1158__en-us_topic_0000001173789542_ab220380b4ad34b0f96df4bb0b67c15f7">vt_sftp_test</strong>" connectionType="sftp-connector"&gt;
&lt;file.inputPath&gt;/opt/houjt/hive/all&lt;/file.inputPath&gt;
&lt;file.splitType&gt;FILE&lt;/file.splitType&gt;
&lt;file.filterType&gt;WILDCARD&lt;/file.filterType&gt;
&lt;file.pathFilter&gt;*&lt;/file.pathFilter&gt;
&lt;file.fileFilter&gt;*&lt;/file.fileFilter&gt;
&lt;file.encodeType&gt;GBK&lt;/file.encodeType&gt;
&lt;file.suffixName&gt;&lt;/file.suffixName&gt;
&lt;file.isCompressive&gt;FALSE&lt;/file.isCompressive&gt;
&lt;/data.source&gt;
&lt;hadoop.source storageType="HDFS" &gt;
&lt;output.outputDirectory&gt;/user/loader/sftp-to-hdfs&lt;/output.outputDirectory&gt;
&lt;output.fileOprType&gt;OVERRIDE&lt;/output.fileOprType&gt;
&lt;throttling.extractors&gt;3&lt;/throttling.extractors&gt;
&lt;output.fileType&gt;TEXT_FILE&lt;/output.fileType&gt;
&lt;/hadoop.source&gt;
&lt;sqoop.job.trans.file&gt;&lt;/sqoop.job.trans.file&gt;
&lt;/sqoop.job&gt;
&lt;/root&gt;</pre>
<div class="note" id="mrs_01_1158__en-us_topic_0000001173789542_na47ff193506748738748e414dffb081d"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1158__en-us_topic_0000001173789542_aed080224ac654512bbb21d7e8d5671f0">Each Loader job needs to be associated with a connector. Connectors are used to read data from external data sources when data is imported to a cluster and used to write data into external data sources when data is exported from the cluster. In the preceding example, an SFTP data source connector is configured. To configure an SFTP and FTP data source connector, a password needs to be set and encrypted. The password encryption method is described as follows:</p>
<ol type="a" id="mrs_01_1158__en-us_topic_0000001173789542_od34f79b07427458ca3b627f34af473a1"><li id="mrs_01_1158__en-us_topic_0000001173789542_l9076609890c344609d94d82d2285258a">Run the following command to go to the <strong id="mrs_01_1158__en-us_topic_0000001173789542_b206947450391230">loader-tools-1.99.3</strong> directory. For example, if the Loader client installation directory is <span class="filepath" id="mrs_01_1158__en-us_topic_0000001173789542_fb462c867ee76443cbd969b40a0cd7c76"><b>/opt/hadoopclient/Loader</b></span>, run the following command:<p id="mrs_01_1158__en-us_topic_0000001173789542_a04a3c7d572b74e278575505d05e87378"><strong id="mrs_01_1158__en-us_topic_0000001173789542_a97af0475dada45b9a4619feb7a896f43">cd /opt/hadoopclient/Loader/loader-tools-1.99.3</strong></p>
</li><li id="mrs_01_1158__en-us_topic_0000001173789542_lf41339f2f81944878888d6585004a23f">Run the following command to encrypt the non-encrypted password:<p id="mrs_01_1158__en-us_topic_0000001173789542_a738cfe86fad747a6961f4b327f2e9b0d"><a name="mrs_01_1158__en-us_topic_0000001173789542_lf41339f2f81944878888d6585004a23f"></a><a name="en-us_topic_0000001173789542_lf41339f2f81944878888d6585004a23f"></a><strong id="mrs_01_1158__en-us_topic_0000001173789542_b142426575543">./encrypt_tool </strong><em id="mrs_01_1158__en-us_topic_0000001173789542_i116421318141315">Unencrypted password</em></p>
</li></ol>
</div></div>
</p></li><li id="mrs_01_1158__en-us_topic_0000001173789542_l09584ebe38a64b689811fa098007092f"><span>Run the following command to go to the directory where loader-tool is located:</span><p><p id="mrs_01_1158__en-us_topic_0000001173789542_ad844c71c5ade4a09857b7957d9242db9"><strong id="mrs_01_1158__en-us_topic_0000001173789542_ad02b5f74cf8b4b208b44720585a30331">cd /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool</strong></p>
</p></li><li id="mrs_01_1158__en-us_topic_0000001173789542_l7cb7420987c742e8a596238e195e36d6"><span>Run the following command to use the lt-ucc tool to create a connector:</span><p><p id="mrs_01_1158__en-us_topic_0000001173789542_aeb093b293c61496fad58983d29d009a3"><strong id="mrs_01_1158__en-us_topic_0000001173789542_a0c30c719a80c4615b67c0b7919ad95c9">./bin/lt-ucc -l /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/job-config/login-info.xml -w /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/job-config/sftp-to-hdfs.xml -a create</strong></p>
<p id="mrs_01_1158__en-us_topic_0000001173789542_a2db27da89d8f43d685292a0002017b4a">If no error is reported and the following information is displayed, the connector creation task is submitted successfully:</p>
<pre class="screen" id="mrs_01_1158__en-us_topic_0000001173789542_sf3547408b08547de8f43747be05dc03e">User login success. begin to execute task.</pre>
</p></li><li id="mrs_01_1158__en-us_topic_0000001173789542_lc00c771c54d24e5899bf38082f36e0b4"><span>Run the following command to use the lt-ucj tool to create a job:</span><p><p id="mrs_01_1158__en-us_topic_0000001173789542_a2937d8291c564678a96dbc85c57e4f1e"><strong id="mrs_01_1158__en-us_topic_0000001173789542_a4fc5eee84b624ccb8fd3bcc7d772318c">./bin/lt-ucj -l /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/job-config/login-info.xml -w /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/job-config/sftp-to-hdfs.xml -a create</strong></p>
<p id="mrs_01_1158__en-us_topic_0000001173789542_ad58c1dcfbda445eaa261269eea74a619">If no error is reported and the following information is displayed, the job creation task is submitted successfully:</p>
<pre class="screen" id="mrs_01_1158__en-us_topic_0000001173789542_s1d9b188d4ae14391909f50d9b0903a27">User login success. begin to execute task.</pre>
</p></li><li id="mrs_01_1158__en-us_topic_0000001173789542_l326aace8f05b4ef585d9ff48008c38ad"><span>Run the following command to use the lt-ctl tool to submit the job:</span><p><p id="mrs_01_1158__en-us_topic_0000001173789542_a46787d9e2b0145cbb626033909300347"><strong id="mrs_01_1158__en-us_topic_0000001173789542_aace17436f5df4a7b8d5d899cb7b80f19">./bin/lt-ctl -l /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/job-config/login-info.xml -n Sftp.to.Hdfs -a start</strong></p>
<p id="mrs_01_1158__en-us_topic_0000001173789542_a1acfeb0859074d59ba7faddad50eed5f">If the following information is displayed, the job is submitted successfully:</p>
<pre class="screen" id="mrs_01_1158__en-us_topic_0000001173789542_s472f974bf5b041c3a24ca3666e350ac9">Start job success.</pre>
</p></li><li id="mrs_01_1158__en-us_topic_0000001173789542_l9214f08e4b324e0e89f9c82245e11e0a"><span>Run the following command to view the job status:</span><p><p id="mrs_01_1158__en-us_topic_0000001173789542_af03d5752482346a3af5efd9cb265efbd"><strong id="mrs_01_1158__en-us_topic_0000001173789542_af3613551a3644727ad63e4dfd84d3269">./bin/lt-ctl -l /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/job-config/login-info.xml -n Sftp.to.Hdfs -a status</strong></p>
<pre class="screen" id="mrs_01_1158__en-us_topic_0000001173789542_s001708b454e240088caf64565b58423d">Job:Sftp.to.Hdfs
Status:RUNNING
Progress: 0.0</pre>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1155.html">Client Tool Description</a></div>
</div>
</div>
View Git Blame Copy Permalink