forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
115 lines
17 KiB
HTML
115 lines
17 KiB
HTML
<a name="mrs_01_0406"></a><a name="mrs_01_0406"></a>
|
|
|
|
<h1 class="topictitle1">Managing Loader Jobs</h1>
|
|
<div id="body1589421630811"><div class="section" id="mrs_01_0406__s172f8b7840734cb2a186560103e0664f"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0406__aff24c5c60eda430aba547e6395403215">You can create, view, edit, and delete jobs on the Loader page.</p>
|
|
<p id="mrs_01_0406__p184721821141115">This section applies to versions earlier than MRS 3.x.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0406__sd476a7a66f3d4b549ca9050c09ba2847"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_0406__aa82074160ea940ee8d3f498c09a066ec">You have accessed the Loader page. For details, see <a href="mrs_01_0401.html#mrs_01_0401__s12f4baccf3914471bee631d0ca198278">Loader Page</a>.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0406__s1e05ed1345b8456fab83e7e58779dc25"><h4 class="sectiontitle">Creating a Job</h4><ol id="mrs_01_0406__o5d91aedfb1b844afaaae1bb647a8d376"><li id="mrs_01_0406__l1b69e3687055430b915b210f3771bf26"><span>On the Loader page, click <span class="uicontrol" id="mrs_01_0406__uicontrol33330958184943"><b>New job</b></span>.</span></li><li id="mrs_01_0406__l32f4b293c2284f68918c30de1710abaa"><span>In <span class="parmname" id="mrs_01_0406__parmname173054111511"><b>Connection</b></span>, set parameters.</span><p><ol type="a" id="mrs_01_0406__o4b938f480d724437bd0f9e8dcaa29cb7"><li id="mrs_01_0406__lad186a00ac4e4c4ab797e93d22a4f37c">In <span class="parmname" id="mrs_01_0406__parmname2179114472315"><b>Name</b></span>, enter a job name.</li><li id="mrs_01_0406__l923c7e3c49b144c285d5bc5d5348dd05">In <span class="parmname" id="mrs_01_0406__parmname49268852185014"><b>From link</b></span> and <span class="parmname" id="mrs_01_0406__parmname43017930185020"><b>To link</b></span>, select links accordingly.<p id="mrs_01_0406__en-us_topic_0071084974_p490755515134">After you select a link of a type, data is obtained from the specified source and saved to the destination.</p>
|
|
<div class="note" id="mrs_01_0406__nd094e0041acb42208e1309bd56f867ba"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0406__a17328b3e05d24cc792433a84573f9226">If no available link exists, click <span class="uicontrol" id="mrs_01_0406__uicontrol17943887185032"><b>Add a new link</b></span>.</p>
|
|
</div></div>
|
|
</li></ol>
|
|
</p></li><li id="mrs_01_0406__l19daa8ebd2f148c1be1ae12b8db9eac7"><span>In <span class="parmname" id="mrs_01_0406__parmname48271841185040"><b>From</b></span>, configure the job of the source link.</span><p><p id="mrs_01_0406__a8283878f4e024b1bb7b122f5dac47007">For details, see <a href="mrs_01_0404.html">Source Link Configurations of Loader Jobs</a>.</p>
|
|
</p></li><li id="mrs_01_0406__l09207ca5a8474297a6306ecb316f8676"><span>In <span class="parmname" id="mrs_01_0406__parmname4086350185046"><b>To</b></span>, configure the job of the destination link.</span><p><p id="mrs_01_0406__a5210f2c81ade43798514a3173de5f763">For details, see <a href="mrs_01_0405.html">Destination Link Configurations of Loader Jobs</a>.</p>
|
|
</p></li><li id="mrs_01_0406__l524319bcbfa84239bf77472fe2831bb1"><span>Check whether a database link is selected in <span class="parmname" id="mrs_01_0406__parmname39885025185056"><b>To link</b></span>.</span><p><p id="mrs_01_0406__af81e07b58e7c43089f665dce16fc45b1">Database links include:</p>
|
|
<ul id="mrs_01_0406__ub839e2eacb94491a959901357e2d1485"><li id="mrs_01_0406__en-us_topic_0071084974_li99429165022">generic-jdbc-connector</li><li id="mrs_01_0406__lc90a103523254dc0886e790b1873aac0">hbase-connector</li><li id="mrs_01_0406__l201135d3ab2040879294ddc1eef32a8d">hive-connector</li></ul>
|
|
<p id="mrs_01_0406__a5f43b6faf10b482bb4feabcf59e1e417">If you set <span class="parmname" id="mrs_01_0406__parmname167887546716436"><b>To link</b></span> to a database link, you need to configure a mapping between service data and a field in the database table.</p>
|
|
<ul id="mrs_01_0406__ufcd7027ca3114d0b87290d4232458edb"><li id="mrs_01_0406__l3d1c4f69f00d4bc7bb746987c14edc3e">If you set it to a database link, go to <a href="#mrs_01_0406__l56f38eb013c549b2b70d8f3a750ea8c4">6</a>.</li><li id="mrs_01_0406__lbdbabc479fba409f9969f600d64f252c">If you do not set it to a database link, go to <a href="#mrs_01_0406__l8e55b33fad0b4b52b4919120d3ab7597">7</a>.</li></ul>
|
|
</p></li><li id="mrs_01_0406__l56f38eb013c549b2b70d8f3a750ea8c4"><a name="mrs_01_0406__l56f38eb013c549b2b70d8f3a750ea8c4"></a><a name="l56f38eb013c549b2b70d8f3a750ea8c4"></a><span>In <span class="parmname" id="mrs_01_0406__parmname56501539185128"><b>Field Mapping</b></span>, enter a field mapping. Then proceed to <a href="#mrs_01_0406__l8e55b33fad0b4b52b4919120d3ab7597">7</a>.</span><p><p id="mrs_01_0406__a1097a7dcb3524a738a24091e41b3e384"><span class="parmname" id="mrs_01_0406__parmname7530978185132"><b>Field Mapping</b></span> specifies a mapping between each column of user data and a field in the database table.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0406__t8db35033881c4787a479889ab05f7d52" frame="border" border="1" rules="all"><caption><b>Table 1 </b><span class="parmname" id="mrs_01_0406__parmname138555504216112"><b>Field Mapping</b></span> properties</caption><thead align="left"><tr id="mrs_01_0406__r7fadf37f19d1434684d5996690c18916"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.2.6.2.2.2.3.1.1"><p id="mrs_01_0406__a0908d93254424b949f4c2351fefe4ab3">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="75%" id="mcps1.3.3.2.6.2.2.2.3.1.2"><p id="mrs_01_0406__ae87ec43a3d804018ab7d8e3ad967803a">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0406__rd9c73ea756164827a3527093bed0dc6c"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.6.2.2.2.3.1.1 "><p id="mrs_01_0406__aa2a6c847ab9646d5a6eb73dc9d502b78">Column Num</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.6.2.2.2.3.1.2 "><p id="mrs_01_0406__accd775862e054e0582fd151cb638e651">Field sequence of service data</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0406__r8a984dc3dacc4bb5953dbb009e19f620"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.6.2.2.2.3.1.1 "><p id="mrs_01_0406__ade8da1510a3340258d4f24da301e5e28">Sample</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.6.2.2.2.3.1.2 "><p id="mrs_01_0406__a3cb12030f2a14364a68e29cf159cec08">First row of sample values of service data</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0406__rb0678da8360f4015a5967eb5efa3db03"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.6.2.2.2.3.1.1 "><p id="mrs_01_0406__a4a72143b1cef4d6bb9622d751b45c604">Column Family</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.6.2.2.2.3.1.2 "><p id="mrs_01_0406__a5e550c50319247309ecc9295cf6f767c">When <span class="parmname" id="mrs_01_0406__parmname14246804392954"><b>To link</b></span> is <strong id="mrs_01_0406__b84235270617474">hbase-connector</strong>, you can select a column family for storing data.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0406__r7d44d63e62eb4df9881491ee6fd2dbc4"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.6.2.2.2.3.1.1 "><p id="mrs_01_0406__a752d5a31338f4496bf7095955325b77c">Destination Field</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.6.2.2.2.3.1.2 "><p id="mrs_01_0406__a1f4cd890f024498b92d9e8fd76560e42">Field for storing data</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0406__r3e9efdcdfd354c6b861df89a664338bf"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.6.2.2.2.3.1.1 "><p id="mrs_01_0406__a7c19e90f39654d86b793ac8d2430fc0f">Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.6.2.2.2.3.1.2 "><p id="mrs_01_0406__a8e843ce3a0384fabba2672e3c730b18e">Type of the field selected by the user</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0406__r4a97734d452944d2a446ac7300c091c5"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.6.2.2.2.3.1.1 "><p id="mrs_01_0406__a4b3b148df68141cd8b1a83002d9d7571">Row Key</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.6.2.2.2.3.1.2 "><p id="mrs_01_0406__a42f04220b0774e72ae89aa4ed50a011e">When <span class="parmname" id="mrs_01_0406__parmname2042125197161812"><b>To link</b></span> is <strong id="mrs_01_0406__b842352706174727">hbase-connector</strong>, you need to select <span class="parmname" id="mrs_01_0406__parmname1757312859161834"><b>Destination Field</b></span> as a row key.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_0406__n1f76fbf0aa794a31b6c958645553dcca"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0406__en-us_topic_0071084974_p383064159506">If the value of <strong id="mrs_01_0406__b1260115616116">From</strong> is a connector of a file type, for example, SFTP, FTP, OBS, and HDFS files, the value of <strong id="mrs_01_0406__b585117231225">Field Mapping</strong> is the first row of data in the file. Ensure that the first row of data is complete. Otherwise, the Loader job will not extract columns that are not mapped.</p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_0406__l8e55b33fad0b4b52b4919120d3ab7597"><a name="mrs_01_0406__l8e55b33fad0b4b52b4919120d3ab7597"></a><a name="l8e55b33fad0b4b52b4919120d3ab7597"></a><span>In <span class="parmname" id="mrs_01_0406__parmname36974479154331"><b>Task Config</b></span>, set job running parameters.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0406__t0fc7f46bfc1e45c6a9ec15a2e0907c24" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Loader job running properties</caption><thead align="left"><tr id="mrs_01_0406__r7c49faa79b48493a90977ee596e882bd"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.2.7.2.1.2.3.1.1"><p id="mrs_01_0406__a9f1a428c7a9b432398732eee31f87692">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="75%" id="mcps1.3.3.2.7.2.1.2.3.1.2"><p id="mrs_01_0406__a8d407e82957a4fb889207f43bfbd19f4">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0406__r94d0a93bc93d49deb1dfb83600f6aec3"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.7.2.1.2.3.1.1 "><p id="mrs_01_0406__ae906b888e1c94499b04715dee7bec621">Extractors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.7.2.1.2.3.1.2 "><p id="mrs_01_0406__a3df14bc0588d4da496d4f2123641c2e1">Number of Map tasks</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0406__r9def9f914f9f423fbf3a7606f56f7cce"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.7.2.1.2.3.1.1 "><p id="mrs_01_0406__a685bd07b98584b28aaad79a89134c512">Loaders</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.7.2.1.2.3.1.2 "><p id="mrs_01_0406__a457a3a1a365e4f02bcfdc4f5d7c11752">Number of Reduce tasks</p>
|
|
<p id="mrs_01_0406__a4b34f7012ed147c0be7d3776aaa99bf0">This parameter is displayed only when the destination field is HBase or Hive.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0406__rf000788d16f84f92a7524c7e6cff5d78"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.7.2.1.2.3.1.1 "><p id="mrs_01_0406__a03589028c19d4130b74d813df6eb65dc">Max. Error Records in a Single Shard</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.7.2.1.2.3.1.2 "><p id="mrs_01_0406__a618938189e934288bd632e3676737d5c">Error record threshold. If the number of error records of a single Map task exceeds the threshold, the task automatically stops and the obtained data is not returned.</p>
|
|
<div class="note" id="mrs_01_0406__n07eb2914d4c046fcbcb93e4078142074"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_0406__a511a2f0502ec4b63b3662653804c5bfa">Data is read and written in batches for <span class="parmname" id="mrs_01_0406__parmname201847020593051"><b>MYSQL</b></span> and <span class="parmname" id="mrs_01_0406__parmname126657715393051"><b>MPPDB</b></span> of <span class="parmname" id="mrs_01_0406__parmname157773387693133"><b>generic-jdbc-connector</b></span> by default. Errors are recorded once at most for each batch of data.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0406__rcd5d2490f121446dbead7b732d0d3bee"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.7.2.1.2.3.1.1 "><p id="mrs_01_0406__a959d13ef9b0d4b9488bdf1cbba7f6e8c">Dirty Data Directory</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.7.2.1.2.3.1.2 "><p id="mrs_01_0406__a02a97de10a474db283417da620846d23">Specifies the directory for saving dirty data. If you leave this parameter blank, dirty data will not be saved.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_0406__lc08abc2a126340e89e08900a46fa1262"><span>Click <span class="uicontrol" id="mrs_01_0406__uicontrol1533244512395"><b>Save</b></span>.</span></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0406__s4e88c9c393144afa8d8bad5fa142cf0b"><h4 class="sectiontitle">Viewing a Job</h4><ol id="mrs_01_0406__oa2155b90a2ff4c86a25019a1a3147246"><li id="mrs_01_0406__l191e304bd48f42b9836780720cb7ed66"><span>Access the Loader page. The Loader job management page is displayed by default.</span><p><ul id="mrs_01_0406__u1e2d3f03508d4b13bfe962c03b48f0f9"><li id="mrs_01_0406__l938d33e30e7a4ab58eed279acf86af31">If Kerberos authentication is enabled for the cluster, all jobs created by the current user are displayed by default and other users' jobs cannot be displayed.</li><li id="mrs_01_0406__l90bf6e29e51c4155a992bd348818979f">If Kerberos authentication is disabled for the cluster, all Loader jobs of the cluster are displayed.</li></ul>
|
|
</p></li><li id="mrs_01_0406__l43c54ae85dbd492786c3af0b5636c142"><span>In <span class="parmname" id="mrs_01_0406__parmname17979794185418"><b>Sqoop Jobs</b></span>, enter a job name to filter the job.</span></li><li id="mrs_01_0406__l3a073232f3e245928d254dcca3487895"><span>Click <span class="uicontrol" id="mrs_01_0406__uicontrol16996588185443"><b>Refresh</b></span> to obtain the latest job status.</span></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0406__s130d0dc7d09c4c21a253aba69cc208d4"><h4 class="sectiontitle">Editing a Job</h4><ol id="mrs_01_0406__o71c2d14f868244ab82c28b4388af7937"><li id="mrs_01_0406__ldf7f8b885820455da8e8dd9168390bc5"><span>Access the Loader page. The Loader job management page is displayed by default.</span></li><li id="mrs_01_0406__l5de28478c0174501a126c55461b40eea"><span>Click the job name to go to the edit page.</span></li><li id="mrs_01_0406__l4136e4d466f6489faadf3bcc1428a01b"><span>Modify the job configuration parameters based on service requirements.</span></li><li id="mrs_01_0406__l34dd079223724f879b0aad5fbc0a7548"><span>Click <span class="uicontrol" id="mrs_01_0406__uicontrol76199117417"><b>Save</b></span>.</span><p><div class="note" id="mrs_01_0406__ncc406db84a9446f7987bfead6bcc69a9"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0406__a3ba86bea550f44758338ca35e6e1d683">Basic job operations in the navigation bar on the left are <span class="uicontrol" id="mrs_01_0406__uicontrol61695106185549"><b>Run</b></span>, <span class="uicontrol" id="mrs_01_0406__uicontrol49104776185554"><b>Copy</b></span>, <span class="uicontrol" id="mrs_01_0406__uicontrol65719851185559"><b>Delete</b></span>, <span class="uicontrol" id="mrs_01_0406__uicontrol861323118569"><b>Disable</b></span>, <span class="uicontrol" id="mrs_01_0406__uicontrol39144118185624"><b>History Record</b></span>, and <span class="uicontrol" id="mrs_01_0406__uicontrol17891368185643"><b>Show Job JSON Definition</b></span>.</p>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0406__s2f01c8435f564248a0f16df2eb14ca8a"><h4 class="sectiontitle">Deleting a Job</h4><ol id="mrs_01_0406__o5d5bd9be987646d08c7945b24e44b99b"><li id="mrs_01_0406__l90f8ad4d799040d08fe2fa18bbc7aff5"><span>Access the Loader page.</span></li><li id="mrs_01_0406__l0488cc28a82f46c8b7c4bd0c6df6cabc"><span>In the row of the specified job, click <span><img id="mrs_01_0406__i60f0537b6be645cc81e00dac33c6d1c2" src="en-us_image_0000001296090328.jpg"></span>.</span><p><p id="mrs_01_0406__a2501c5411c1846f1a0a3e48a5d69128c">You can also select one or more jobs and click <strong id="mrs_01_0406__b7264615411">Delete</strong> Job in the upper right corner of the job list.</p>
|
|
</p></li><li id="mrs_01_0406__l1b68775ed52d4e57a29714bd00b6f49b"><span>In the dialog box, click <span class="uicontrol" id="mrs_01_0406__uicontrol0525124715410"><b>Yes, delete it</b></span>.</span><p><p id="mrs_01_0406__a1680a7db44054d1b8d9a4b0632bb8229">If the state of a Loader job is <span class="parmname" id="mrs_01_0406__parmname11421720185733"><b>Running</b></span>, the job fails to be deleted.</p>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0400.html">Using Loader</a></div>
|
|
</div>
|
|
</div>
|
|
|