vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_1161.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

116 lines
19 KiB
HTML
Raw Blame History

<a name="mrs_01_1161"></a><a name="mrs_01_1161"></a>
<h1 class="topictitle1">Using loader-backup to Back Up Job Data</h1>
<div id="body8662426"><div class="section" id="mrs_01_1161__en-us_topic_0000001173630666_sf739ed620c5840659ee205a48b63ce16"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1161__en-us_topic_0000001173630666_a8e694842eb11498d962db0527c9e96ac">After a job is created using the Loader WebUI or loader-tool, use loader-backup to back up data.</p>
<div class="note" id="mrs_01_1161__en-us_topic_0000001173630666_nf6ec1b3903494afbb3b507141172705c"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1161__en-us_topic_0000001173630666_u6555491804e14681b77443281dba1bde"><li id="mrs_01_1161__en-us_topic_0000001173630666_l468c20ad49b3452fa22ef17d15e88090">Only Loader jobs of data export support data backup.</li><li id="mrs_01_1161__en-us_topic_0000001173630666_l1df94fb4c5444d4c8a3ad4b854262f71">This tool is an internal Loader interface and is invoked by the upper-layer component HBase. Only the data backup from HDFS to SFTP is supported.</li></ul>
</div></div>
</div>
<div class="section" id="mrs_01_1161__en-us_topic_0000001173630666_sf642f388aa1e4c7d85e7c2b21cc6c2d8"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_1161__en-us_topic_0000001173630666_a2d08d1d73bb74e4f8aa572d0c0537fc0">The Loader client has been installed and configured.</p>
</div>
<div class="section" id="mrs_01_1161__en-us_topic_0000001173630666_s0633f8ae8ed6466c9cc35ee6113d11a9"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1161__en-us_topic_0000001173630666_obfaf124b4df94717a4615dca24f7f730"><li id="mrs_01_1161__en-us_topic_0000001173630666_l920a19d8f3974ce484bbb02bfdc2f68c"><span>Log in to the node where the client is installed as the user who installs the client. </span></li><li id="mrs_01_1161__en-us_topic_0000001173630666_l259d9b65bac44c32b78544c714f96cfa"><span>Run the following command to go to the directory where the <span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_fb00419b3f8594d91ab7e29684f3d7d6d"><b>backup.properties</b></span> file is located. For example, if the Loader client installation directory is <span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_f5644f8c823a54eabbc02c6a1c38b0e9c"><b>/opt/hadoopclient/Loader</b></span>, run the following command:</span><p><p id="mrs_01_1161__en-us_topic_0000001173630666_a93ac9e8a8d084028ac7f412805c43508"><strong id="mrs_01_1161__en-us_topic_0000001173630666_ac858fb8259d74e859b2e15669a4ab557">cd /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-backup/conf</strong></p>
</p></li><li id="mrs_01_1161__en-us_topic_0000001173630666_l6366e616f6f944f7a1e55ddd8599a74d"><span>Run the following command to modify the configuration parameters of <span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_f8885c2f87e824c5a9a9b2d64e8e6ce11"><b>backup.properties</b></span>. For details about the parameters, see <a href="#mrs_01_1161__en-us_topic_0000001173630666_tb7ca0da0faf341c3b41cb1ef2d388137">Table 1</a>.</span><p><p id="mrs_01_1161__en-us_topic_0000001173630666_a44b69474bfe14605853f069d8fc26ed9"><strong id="mrs_01_1161__en-us_topic_0000001173630666_a703d23a4e2c44c87bf6fdb07ffe43660">vi backup.properties</strong></p>
<pre class="screen" id="mrs_01_1161__en-us_topic_0000001173630666_s73be0fc3154d444cb5156c704e45ed78">server.url = <em id="mrs_01_1161__en-us_topic_0000001173630666_a377b1fa9085043f8a47fdf956eff33d4">10.0.0.1:21351,10.0.0.2:12000</em>
authentication.type = <em id="mrs_01_1161__en-us_topic_0000001173630666_ad3aaaa46d6e6403b9d1efcd155004add">kerberos</em>
authentication.user =
authentication.password=
job.jobId = <em id="mrs_01_1161__en-us_topic_0000001173630666_a824cc5a70f884cba94ab4df8c4eea994">1</em>
use.keytab = <em id="mrs_01_1161__en-us_topic_0000001173630666_a9b0e2ee92cc949fc9641cde08e355778">true</em>
client.principal = <em id="mrs_01_1161__en-us_topic_0000001173630666_a90f7a9509ccf4e95bb3c6cc4fa3a756a">loader/hadoop</em>
client.keytab = <em id="mrs_01_1161__en-us_topic_0000001173630666_a5e26f9d80ff244729ec402b7c9f03d55">/opt/client/conf/loader.keytab</em></pre>
<div class="tablenoborder"><a name="mrs_01_1161__en-us_topic_0000001173630666_tb7ca0da0faf341c3b41cb1ef2d388137"></a><a name="en-us_topic_0000001173630666_tb7ca0da0faf341c3b41cb1ef2d388137"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1161__en-us_topic_0000001173630666_tb7ca0da0faf341c3b41cb1ef2d388137" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Configuration parameters</caption><thead align="left"><tr id="mrs_01_1161__en-us_topic_0000001173630666_r6c2de3efa4d04b3ab842ea899ffeecd0"><th align="left" class="cellrowborder" valign="top" width="31.39%" id="mcps1.3.3.2.3.2.3.2.4.1.1"><p id="mrs_01_1161__en-us_topic_0000001173630666_a8a21ab0458f34caea98278e2ebdcba38">Configuration parameters</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="44.55%" id="mcps1.3.3.2.3.2.3.2.4.1.2"><p id="mrs_01_1161__en-us_topic_0000001173630666_add76fe5c9ff54806b5d8f16851bb5784">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="24.060000000000002%" id="mcps1.3.3.2.3.2.3.2.4.1.3"><p id="mrs_01_1161__en-us_topic_0000001173630666_ad0503b34a9d04ca1b8fa8e0fce1f6a58">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1161__en-us_topic_0000001173630666_rd2f1a7f851d24eb4bef2ca8293deecd5"><td class="cellrowborder" valign="top" width="31.39%" headers="mcps1.3.3.2.3.2.3.2.4.1.1 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a5ab0c534f0394128be796de8974f3c47">server.url</p>
</td>
<td class="cellrowborder" valign="top" width="44.55%" headers="mcps1.3.3.2.3.2.3.2.4.1.2 "><p id="mrs_01_1161__en-us_topic_0000001173630666_aaccc1fde88f14289b68d9da093987135">Floating IP address and port (21351) for Loader.</p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_a9ff1ad45af3d43b1ad9dfb683d4d98a6">For compatibility, multiple IP addresses and ports can be configured and need to be separated by commas (<span class="parmvalue" id="mrs_01_1161__en-us_topic_0000001173630666_parmvalue2321164142012"><b>,</b></span>). The first IP address and port must be those of Loader (21351). The others can be configured based on service requirements.</p>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.3.2.3.2.3.2.4.1.3 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a52f879468a874df5b5a8bc00d26c7f5b">10.0.0.1:21351,10.0.0.2:12000</p>
</td>
</tr>
<tr id="mrs_01_1161__en-us_topic_0000001173630666_rf6e50be7080a4a0b84b88bb9272c2c02"><td class="cellrowborder" valign="top" width="31.39%" headers="mcps1.3.3.2.3.2.3.2.4.1.1 "><p id="mrs_01_1161__en-us_topic_0000001173630666_ad9275e13e42f4bc78d2917d464ebff90">authentication.type</p>
</td>
<td class="cellrowborder" valign="top" width="44.55%" headers="mcps1.3.3.2.3.2.3.2.4.1.2 "><p id="mrs_01_1161__en-us_topic_0000001173630666_aa2dc33d40d8a4c4a90edcc5721ac90ea">Login authentication mode.</p>
<ul id="mrs_01_1161__en-us_topic_0000001173630666_u7d0a7153b297413ca3e2b3db1cb7b33d"><li id="mrs_01_1161__en-us_topic_0000001173630666_lf7bd1fd528fa4030923333bacb8ccc2b"><span class="parmvalue" id="mrs_01_1161__en-us_topic_0000001173630666_pd309fcfc9f1b4e729a360f8a2519ec2e"><b>kerberos</b></span> indicates that the security mode is used and Kerberos authentication is performed. Kerberos authentication provides two authentication modes: the password mode and the keytab file mode.</li><li id="mrs_01_1161__en-us_topic_0000001173630666_lf0df33c204884e36a25ba4ab37f66db0"><span class="parmvalue" id="mrs_01_1161__en-us_topic_0000001173630666_p8caa1d0a77124a1ca62e61db8621aaee"><b>simple</b></span> indicates that the normal mode is used and Kerberos authentication is not performed.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.3.2.3.2.3.2.4.1.3 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a3bd41769f5d24fefb67c7ceb25483bc2">kerberos</p>
</td>
</tr>
<tr id="mrs_01_1161__en-us_topic_0000001173630666_r74a0ae088c584a9b82f94a8176a0b08c"><td class="cellrowborder" valign="top" width="31.39%" headers="mcps1.3.3.2.3.2.3.2.4.1.1 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a08c66f0f8deb4691a5bacaaf3a86cc00">authentication.user</p>
</td>
<td class="cellrowborder" valign="top" width="44.55%" headers="mcps1.3.3.2.3.2.3.2.4.1.2 "><p id="mrs_01_1161__en-us_topic_0000001173630666_aa9e45484f473457f941dcb6dc2fea721">User for login when the normal mode or password authentication is used.</p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_a41b7668e701b4eca99e111ef464d96db">In the keytab login mode, this parameter does not need to be set.</p>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.3.2.3.2.3.2.4.1.3 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a60d7275c641b405c91c72353cfe18eb0">bar</p>
</td>
</tr>
<tr id="mrs_01_1161__en-us_topic_0000001173630666_r987c986c85c248729eddb386c7920941"><td class="cellrowborder" valign="top" width="31.39%" headers="mcps1.3.3.2.3.2.3.2.4.1.1 "><p id="mrs_01_1161__en-us_topic_0000001173630666_ada1e465a7e684d219fde03b2cb671a23">authentication.password</p>
</td>
<td class="cellrowborder" valign="top" width="44.55%" headers="mcps1.3.3.2.3.2.3.2.4.1.2 "><p id="mrs_01_1161__en-us_topic_0000001173630666_afcd695fd212446bb8cbba0eae5c3f7f9">User password for login when the password authentication mode is used.</p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_a80ea5818c0e841b0a0e26c8c468d7646">In the normal mode or keytab login mode, this parameter does not need to be set.</p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_a3b2ed9e62203488787b7e3164f7837aa">The password needs to be encrypted. The encryption method is described as follows:</p>
<ol type="a" id="mrs_01_1161__en-us_topic_0000001173630666_o5438d27de79a43d1a9d5e472ba4ffecf"><li id="mrs_01_1161__en-us_topic_0000001173630666_l4e46a94cebc0459cb7fd50fc34ba8be0">Go to the directory where <span class="parmname" id="mrs_01_1161__en-us_topic_0000001173630666_p7cd6ae640d2f4d68b14fc10c72840acb"><b>encrypt_tool</b></span> is located. For example, if the Loader client installation directory is <span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_f58f52e1f6054400881b0902f3de27b13"><b>/opt/hadoopclient/Loader</b></span>, run the following command:<p id="mrs_01_1161__en-us_topic_0000001173630666_a6944bd06bded47d6939193380b2774c9"><strong id="mrs_01_1161__en-us_topic_0000001173630666_affe07b3f5e4a42a79bed4024c7bb6a3a">cd /opt/hadoopclient/Loader/loader-tools-1.99.3</strong></p>
</li><li id="mrs_01_1161__en-us_topic_0000001173630666_lbd0fa39d027842f89f4813f9bc2d4c48">Run the following command to encrypt the non-encrypted password:<p id="mrs_01_1161__en-us_topic_0000001173630666_aaabbfdd1dfc7422c9567a603d0781aa1"><a name="mrs_01_1161__en-us_topic_0000001173630666_lbd0fa39d027842f89f4813f9bc2d4c48"></a><a name="en-us_topic_0000001173630666_lbd0fa39d027842f89f4813f9bc2d4c48"></a><strong id="mrs_01_1161__en-us_topic_0000001173630666_b332234119137">./encrypt_tool </strong><em id="mrs_01_1161__en-us_topic_0000001173630666_i83281641121313">Unencrypted password</em></p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_aa027af20e9a8442b835a60cb113c3e48">The obtained encrypted password is used as the value of <span class="parmname" id="mrs_01_1161__en-us_topic_0000001173630666_parmname664817112219"><b>authentication.password</b></span>.</p>
<div class="note" id="mrs_01_1161__en-us_topic_0000001173630666_note8349540161234"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1161__en-us_topic_0000001173630666_p8036997161234">If a non-encrypted password contains special characters, the special characters must be escaped. For example, the dollar sign ($) is a special character and can be escaped using single quotation marks ('), for example, <strong id="mrs_01_1161__en-us_topic_0000001173630666_b9912104162112">'1q2w#e$r'</strong>. If a non-encrypted password contains single quotation marks, use double quotation marks to escape the single quotation marks. If a non-encrypted password contains double quotation marks, use backslashes (\) to escape the double quotation marks. For details, see the shell escape character rules.</p>
</div></div>
</li></ol>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.3.2.3.2.3.2.4.1.3 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a945fabca3a4840dd9459f95be1263ad6">43B80E33A96DF3D203ABBDFD1050C041</p>
</td>
</tr>
<tr id="mrs_01_1161__en-us_topic_0000001173630666_r73ce855969f1485785dff32a4685abe1"><td class="cellrowborder" valign="top" width="31.39%" headers="mcps1.3.3.2.3.2.3.2.4.1.1 "><p id="mrs_01_1161__en-us_topic_0000001173630666_abf7e956b8b394513bd74ede51312117b">job.jobId</p>
</td>
<td class="cellrowborder" valign="top" width="44.55%" headers="mcps1.3.3.2.3.2.3.2.4.1.2 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a6c7d7f26167a4bd48a560b4143418a17">ID of the job whose data is to be backed up.</p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_aa05ef3becc6c40ccbb5c2a4b53ad34ed">Job IDs can be viewed under created jobs on the Loader web UI.</p>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.3.2.3.2.3.2.4.1.3 "><p id="mrs_01_1161__en-us_topic_0000001173630666_adf21a0b673e64330b7a6bd54db53b8e9">1</p>
</td>
</tr>
<tr id="mrs_01_1161__en-us_topic_0000001173630666_rb208ef96d4c4451684ec3ed3f5ff7e76"><td class="cellrowborder" valign="top" width="31.39%" headers="mcps1.3.3.2.3.2.3.2.4.1.1 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a38a19895e6b54083ab8adbec4328b364">use.keytab</p>
</td>
<td class="cellrowborder" valign="top" width="44.55%" headers="mcps1.3.3.2.3.2.3.2.4.1.2 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a006ca01c8e2c4f4bb18c7d240df1e27c">Whether to use the keytab mode to log in.</p>
<ul id="mrs_01_1161__en-us_topic_0000001173630666_u719cedf1e5124327a9b853373a3ad472"><li id="mrs_01_1161__en-us_topic_0000001173630666_lca8e4643fdce411492198bc4c19b7cfa"><strong id="mrs_01_1161__en-us_topic_0000001173630666_b179014822991453">true</strong> indicates using the keytab file to log in.</li><li id="mrs_01_1161__en-us_topic_0000001173630666_lc2a7baf7cc4140b883ad1e414b183d0a"><strong id="mrs_01_1161__en-us_topic_0000001173630666_b24122881091453">false</strong> indicates using the password to log in.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.3.2.3.2.3.2.4.1.3 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a3fc3cbc793f44f3187afda3a31bc184e">true</p>
</td>
</tr>
<tr id="mrs_01_1161__en-us_topic_0000001173630666_re17bedc9419d4bab8bee27aa4cdf9d42"><td class="cellrowborder" valign="top" width="31.39%" headers="mcps1.3.3.2.3.2.3.2.4.1.1 "><p id="mrs_01_1161__en-us_topic_0000001173630666_ae09664db8124482292285bc5ae8da72c">client.principal</p>
</td>
<td class="cellrowborder" valign="top" width="44.55%" headers="mcps1.3.3.2.3.2.3.2.4.1.2 "><p id="mrs_01_1161__en-us_topic_0000001173630666_addbad074b4354758a0d8c81d5df5f5a9">User principal for accessing the Loader service when the keytab authentication mode is used.</p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_aaa91fe00c9974d259014f1456183330d">In the normal mode or password login mode, this parameter does not need to be set.</p>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.3.2.3.2.3.2.4.1.3 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a283a02b395194d0e9b72236dbfe8a680">loader/hadoop</p>
</td>
</tr>
<tr id="mrs_01_1161__en-us_topic_0000001173630666_r478f6e8f9dc841a6a3d4c960d2c34116"><td class="cellrowborder" valign="top" width="31.39%" headers="mcps1.3.3.2.3.2.3.2.4.1.1 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a6db493a9acc34ae397ff26825b51d1c2">client.keytab</p>
</td>
<td class="cellrowborder" valign="top" width="44.55%" headers="mcps1.3.3.2.3.2.3.2.4.1.2 "><p id="mrs_01_1161__en-us_topic_0000001173630666_a755fb229930248c7984988c3ea277f81">Directory where the used keytab file is located when the keytab authentication mode is used.</p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_a569e9b50017a48e28d6ecc18c6142772">In the normal mode or password login mode, this parameter does not need to be set.</p>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.3.2.3.2.3.2.4.1.3 "><p id="mrs_01_1161__en-us_topic_0000001173630666_ae313912c61bb4a4091d37a2c65fa982d">/opt/client/conf/loader.keytab</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="mrs_01_1161__en-us_topic_0000001173630666_l87279b15e1e44197a41b45fdf5cef653"><span>Run the following command to go to the directory where the backup script <span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_f635b62aff57a48f6be35e59ca0668675"><b>run.sh</b></span> is located. For example, if the Loader client installation directory is <span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_f2b81f0856aa845b6974b5909423897fa"><b>/opt/hadoopclient/Loader</b></span>, run the following command:</span><p><p id="mrs_01_1161__en-us_topic_0000001173630666_abf837f2e81e64339abb43505a25e11b5"><strong id="mrs_01_1161__en-us_topic_0000001173630666_a7faee5937ed8475eabbc7716ddc8a071">cd /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-backup</strong></p>
</p></li><li id="mrs_01_1161__en-us_topic_0000001173630666_lcb1708fa7c974db3b52842e24538b8ac"><span>Run the following command to run the backup script <span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_ffc2aa56f3c754ad5ba66ae3f20c6d46a"><b>run.sh</b></span> to back up Loader job data. The system backs up data to a directory at the same layer of the job output directory.</span><p><p id="mrs_01_1161__en-us_topic_0000001173630666_a4340b329cbe84f83aa656914840fe449"><strong id="mrs_01_1161__en-us_topic_0000001173630666_a135b0e4c7b6e47428022f52ac0746eae">./run.sh</strong> <i><span class="varname" id="mrs_01_1161__en-us_topic_0000001173630666_vbbf2c519e8c045478a2c1c88c66ab5b6">Backup data input directory</span></i></p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_af80beabee7024bdd816a0510bb112bd0">For example, the backup data input directory is <span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_f69c5354d163e411282d3a326b87c0292"><b>/user/hbase/</b></span>, and the job output directory is<span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_feb4c11d3a3b84b1083dffa4ea698e8a1"><b>/opt/client/sftp/sftp1</b></span>. <strong id="mrs_01_1161__en-us_topic_0000001173630666_b7747162121213">sftp1</strong> acts as a placeholder. Run the following command to back up data to the <span class="filepath" id="mrs_01_1161__en-us_topic_0000001173630666_f650ca2722f304e928a35379c170518f1"><b>/opt/client/sftp/hbase</b></span> directory:</p>
<p id="mrs_01_1161__en-us_topic_0000001173630666_a35d846f47d1e4b7bb25f432889ae1ce9"><strong id="mrs_01_1161__en-us_topic_0000001173630666_a1d4e6b3cd2644cc48ef7110aed7bd3dd">./run.sh /user/hbase/</strong></p>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1155.html">Client Tool Description</a></div>
</div>
</div>
View Git Blame Copy Permalink