forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
56 lines
8.8 KiB
HTML
56 lines
8.8 KiB
HTML
<a name="mrs_01_1084"></a><a name="mrs_01_1084"></a>
|
|
|
|
<h1 class="topictitle1">Using Loader from Scratch</h1>
|
|
<div id="body1590118474320"><p id="mrs_01_1084__p8060118">You can use Loader to import data from the SFTP server to HDFS.</p>
|
|
<p id="mrs_01_1084__p174796143610">This section applies to MRS clusters earlier than 3.<em id="mrs_01_1084__i3172144114616">x</em>.</p>
|
|
<div class="section" id="mrs_01_1084__sb8fe6f0415124936bb9a7810db345b17"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1084__u77b44b62f4094567bab90cb33acea37d"><li id="mrs_01_1084__l3da0b33cd6d647379b1ae998f71758af">You have prepared service data.</li><li id="mrs_01_1084__l8719dce3c103490ca6b2589ab48280fe">You have created an analysis cluster.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1084__section59245620202"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1084__ol18351433112214"><li id="mrs_01_1084__lb978fb6411794f79bba2776dab66b6b1"><span>Access the Loader page.</span><p><ol type="a" id="mrs_01_1084__o0ccdf6e190ba4491b87c2bb3d6a6e8a2"><li id="mrs_01_1084__li13806122116487">Access the cluster details page.<ul id="mrs_01_1084__ul1692910323486"><li id="mrs_01_1084__li792916322486">For versions earlier than MRS 1.9.2, log in to MRS Manager and choose <strong id="mrs_01_1084__b1112018585159">Services</strong>.</li><li id="mrs_01_1084__li979413816488">For MRS 1.9.2 or later, click the cluster name on the MRS console and choose <strong id="mrs_01_1084__b91201555151514">Components</strong>.</li></ul>
|
|
</li><li id="mrs_01_1084__le46c25d15b43451995993a826e4450f3">Choose <span class="menucascade" id="mrs_01_1084__menucascade3961175912"><b><span class="uicontrol" id="mrs_01_1084__uicontrol295525713">Hue</span></b></span>. In <span class="parmname" id="mrs_01_1084__parmname2096112515119"><b>Hue Web UI</b></span> of <span class="parmname" id="mrs_01_1084__parmname119612051116"><b>Hue Summary</b></span>, click <span class="uicontrol" id="mrs_01_1084__uicontrol696214510115"><b>Hue (Active)</b></span>. The Hue web UI is displayed.</li><li id="mrs_01_1084__l4bcf37712197447da959a6ad3f59c175">Choose <span class="menucascade" id="mrs_01_1084__menucascade221113196111"><b><span class="uicontrol" id="mrs_01_1084__uicontrol521012194117">Data Browsers</span></b> > <b><span class="uicontrol" id="mrs_01_1084__uicontrol4211101913110">Sqoop</span></b></span>.<p id="mrs_01_1084__a6335d720267a4071afc26d73e3896769">The job management tab page is displayed by default on the Loader page.</p>
|
|
</li></ol>
|
|
</p></li><li id="mrs_01_1084__li435034152811"><span>On the Loader page, click <span class="uicontrol" id="mrs_01_1084__uicontrol151611231023"><b>Manage links</b></span>.</span></li><li id="mrs_01_1084__li48883218306"><a name="mrs_01_1084__li48883218306"></a><a name="li48883218306"></a><span>Click <strong id="mrs_01_1084__b457814219410">New link</strong> and create <strong id="mrs_01_1084__b15711897415">sftp-connector</strong>. For details, see <a href="mrs_01_0402.html#mrs_01_0402__s73ada4f9d7e94890a00a2c7a90856ba6">File Server Link</a>.</span></li><li id="mrs_01_1084__li14723052103216"><a name="mrs_01_1084__li14723052103216"></a><a name="li14723052103216"></a><span>Click <strong id="mrs_01_1084__b45936511648">New link</strong>, enter the link name, select <strong id="mrs_01_1084__b10948858842">hdfs-connector</strong>, and create <strong id="mrs_01_1084__b1653277857">hdfs-connector</strong>.</span></li><li id="mrs_01_1084__li3406254193317"><span>On the Loader page, click <span class="uicontrol" id="mrs_01_1084__uicontrol4548091752"><b>Manage jobs</b></span>.</span></li><li id="mrs_01_1084__li18280558103513"><span>Click <span class="uicontrol" id="mrs_01_1084__uicontrol1667715791910"><b>New Job</b></span>.</span></li><li id="mrs_01_1084__l32f4b293c2284f68918c30de1710abaa"><span>In <span class="parmname" id="mrs_01_1084__parmname4971345956"><b>Connection</b></span>, set parameters.</span><p><ol type="a" id="mrs_01_1084__o4b938f480d724437bd0f9e8dcaa29cb7"><li id="mrs_01_1084__lad186a00ac4e4c4ab797e93d22a4f37c">In <span class="parmname" id="mrs_01_1084__parmname6841114916518"><b>Name</b></span>, enter a job name.</li><li id="mrs_01_1084__l923c7e3c49b144c285d5bc5d5348dd05">Select the source link created in <a href="#mrs_01_1084__li48883218306">3</a> and the target link created in <a href="#mrs_01_1084__li14723052103216">4</a>.</li></ol>
|
|
</p></li><li id="mrs_01_1084__l19daa8ebd2f148c1be1ae12b8db9eac7"><span>In <span class="parmname" id="mrs_01_1084__parmname163621415068"><b>From</b></span>, configure the job of the source link.</span><p><p id="mrs_01_1084__a8283878f4e024b1bb7b122f5dac47007">For details, see <a href="mrs_01_0404.html#mrs_01_0404__s033d5edc10164032b9ea23d01081beae">ftp-connector or sftp-connector</a>.</p>
|
|
</p></li><li id="mrs_01_1084__l09207ca5a8474297a6306ecb316f8676"><span>In <span class="parmname" id="mrs_01_1084__parmname19494527262"><b>To</b></span>, configure the job of the target link.</span><p><p id="mrs_01_1084__a5210f2c81ade43798514a3173de5f763">For details, see <a href="mrs_01_0405.html#mrs_01_0405__s0e7a49c2520c498aa9e3d9fa84325e2e">hdfs-connector</a>.</p>
|
|
</p></li><li id="mrs_01_1084__l8e55b33fad0b4b52b4919120d3ab7597"><span>In <span class="parmname" id="mrs_01_1084__parmname970319331797"><b>Task Config</b></span>, set job running parameters.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1084__t0fc7f46bfc1e45c6a9ec15a2e0907c24" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Loader job running properties</caption><thead align="left"><tr id="mrs_01_1084__r7c49faa79b48493a90977ee596e882bd"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.4.2.10.2.1.2.3.1.1"><p id="mrs_01_1084__a9f1a428c7a9b432398732eee31f87692"><strong id="mrs_01_1084__b18421038997">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="75%" id="mcps1.3.4.2.10.2.1.2.3.1.2"><p id="mrs_01_1084__a8d407e82957a4fb889207f43bfbd19f4"><strong id="mrs_01_1084__b8817183813917">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1084__r94d0a93bc93d49deb1dfb83600f6aec3"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.2.10.2.1.2.3.1.1 "><p id="mrs_01_1084__ae906b888e1c94499b04715dee7bec621">Extractors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.4.2.10.2.1.2.3.1.2 "><p id="mrs_01_1084__a3df14bc0588d4da496d4f2123641c2e1">Number of Map tasks</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1084__r9def9f914f9f423fbf3a7606f56f7cce"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.2.10.2.1.2.3.1.1 "><p id="mrs_01_1084__a685bd07b98584b28aaad79a89134c512">Loaders</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.4.2.10.2.1.2.3.1.2 "><p id="mrs_01_1084__a457a3a1a365e4f02bcfdc4f5d7c11752">Number of Reduce tasks</p>
|
|
<p id="mrs_01_1084__a4b34f7012ed147c0be7d3776aaa99bf0">This parameter is displayed only when the destination field is HBase or Hive.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1084__rf000788d16f84f92a7524c7e6cff5d78"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.2.10.2.1.2.3.1.1 "><p id="mrs_01_1084__a03589028c19d4130b74d813df6eb65dc">Max. Error Records in a Single Shard</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.4.2.10.2.1.2.3.1.2 "><p id="mrs_01_1084__a618938189e934288bd632e3676737d5c">Error record threshold. If the number of error records of a single Map task exceeds the threshold, the task automatically stops and the obtained data is not returned.</p>
|
|
<div class="note" id="mrs_01_1084__n07eb2914d4c046fcbcb93e4078142074"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1084__a511a2f0502ec4b63b3662653804c5bfa">Data is read and written in batches for <span class="parmname" id="mrs_01_1084__parmname168114354103"><b>MYSQL</b></span> and <span class="parmname" id="mrs_01_1084__parmname6686173512105"><b>MPPDB</b></span> of <span class="parmname" id="mrs_01_1084__parmname15686935121012"><b>generic-jdbc-connector</b></span> by default. Errors are recorded once at most for each batch of data.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1084__rcd5d2490f121446dbead7b732d0d3bee"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.4.2.10.2.1.2.3.1.1 "><p id="mrs_01_1084__a959d13ef9b0d4b9488bdf1cbba7f6e8c">Dirty Data Directory</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.4.2.10.2.1.2.3.1.2 "><p id="mrs_01_1084__a02a97de10a474db283417da620846d23">Directory for saving dirty data. If you leave this parameter blank, dirty data will not be saved.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_1084__lc08abc2a126340e89e08900a46fa1262"><span>Click <span class="uicontrol" id="mrs_01_1084__uicontrol419872019115"><b>Save</b></span>.</span></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0400.html">Using Loader</a></div>
|
|
</div>
|
|
</div>
|
|
|