forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
91 lines
18 KiB
HTML
91 lines
18 KiB
HTML
<a name="mrs_01_1160"></a><a name="mrs_01_1160"></a>
|
|
|
|
<h1 class="topictitle1">schedule-tool Usage Example</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_1160__en-us_topic_0000001219149041_s3b85b781ef5a4311bf3a4ebac84f2df7"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1160__en-us_topic_0000001219149041_a27ecf6c5da1c4643af8997babb7d5534">After a job is created using the Loader WebUI or Loader-tool, use schedule-tool to execute the job.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1160__en-us_topic_0000001219149041_s5252d29f170f416cb82e8d9af003b241"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_1160__en-us_topic_0000001219149041_a9e055183ecd4415ebaf2d9096c554e4d">The Loader client has been installed and configured.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1160__en-us_topic_0000001219149041_s30b1d7224cae4b609e6e0ed3768450eb"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1160__en-us_topic_0000001219149041_o2da12810a99d48d4b11ca0896492ca37"><li id="mrs_01_1160__en-us_topic_0000001219149041_la615007634824ea9af1b8e5b93c0867b"><a name="mrs_01_1160__en-us_topic_0000001219149041_la615007634824ea9af1b8e5b93c0867b"></a><a name="en-us_topic_0000001219149041_la615007634824ea9af1b8e5b93c0867b"></a><span>In the directory <span class="filepath" id="mrs_01_1160__en-us_topic_0000001219149041_filepath141287767691535"><b>/opt/houjt/test03</b></span> on the SFTP server, create multiple files with <span class="parmvalue" id="mrs_01_1160__en-us_topic_0000001219149041_parmvalue93007449691535"><b>table1</b></span> as the prefix, <span class="parmvalue" id="mrs_01_1160__en-us_topic_0000001219149041_parmvalue20982928791535"><b>.txt</b></span> as the suffix, and <strong id="mrs_01_1160__en-us_topic_0000001219149041_b156301793391535">yyyyMMdd</strong> as the date format in the middle of the file name.</span><p><div class="fignone" id="mrs_01_1160__en-us_topic_0000001219149041_fbeca649a7f394f89a37bd20d5d0d21a1"><span class="figcap"><b>Figure 1 </b>Example</span><br><span><img id="mrs_01_1160__en-us_topic_0000001219149041_i7e080b6a36b64bc09afecc2d752f4838" src="en-us_image_0000001295740224.png"></span></div>
|
|
</p></li><li id="mrs_01_1160__en-us_topic_0000001219149041_l4a977dc395bd4a429bc91f79d54632aa"><span>Create a Loader job of importing data from the SFTP server to HDFS. For details, see <a href="mrs_01_1089.html">Typical Scenario: Importing Data from an SFTP Server to HDFS or OBS</a>.</span></li><li id="mrs_01_1160__en-us_topic_0000001219149041_ld9f0bfa78edc4bb7819ab57695449add"><span>Log in to the node where the client is located as the user who installs the client.</span></li><li id="mrs_01_1160__en-us_topic_0000001219149041_l52dafe8b74f04f0697229f72101dceea"><span>Run the following command to go to the <strong id="mrs_01_1160__en-us_topic_0000001219149041_b48292582191535">conf</strong> directory of schedule-tool. For example, if the Loader client installation directory is <span class="filepath" id="mrs_01_1160__en-us_topic_0000001219149041_filepath367354710176"><b>/opt/hadoopclient/Loader</b></span>, run the following command:</span><p><p id="mrs_01_1160__en-us_topic_0000001219149041_a076bc25d32f14562ad3d9dad97d4cfcf"><strong id="mrs_01_1160__en-us_topic_0000001219149041_a62313a85bb814a92af7edce0f9b17b55">cd /opt/hadoopclient/Loader/loader-tools-1.99.3/schedule-tool/conf</strong></p>
|
|
</p></li><li id="mrs_01_1160__en-us_topic_0000001219149041_l0841cac005a14f708bac67181c8d9d57"><span>Run the following command to edit the schedule.properties file and configure the login mode:</span><p><p id="mrs_01_1160__en-us_topic_0000001219149041_af426647455b24ddba80d170a12283c4b"><strong id="mrs_01_1160__en-us_topic_0000001219149041_ac0b27705ee0247a39166db3c7878ba1a">vi schedule.properties</strong></p>
|
|
<p id="mrs_01_1160__en-us_topic_0000001219149041_af2dc3b785fd64d41822b4c09ee65b75c">schedule-tool supports two login modes. Only one mode can be selected. For parameter details, see <a href="mrs_01_1159.html">schedule-tool Usage Guide</a>.</p>
|
|
<ul id="mrs_01_1160__en-us_topic_0000001219149041_u87782671955d437db64c67cb126d6ac6"><li id="mrs_01_1160__en-us_topic_0000001219149041_l3b7c53f5488c4a709bafe98923391adf">When the password mode is used for login, the configuration information example is as follows:<pre class="screen" id="mrs_01_1160__en-us_topic_0000001219149041_s46d602462d43454f9b8f5f107bef8e5c">[server.url = 10.10.26.187:21351,127.0.0.2:21351]
|
|
[authentication.type = kerberos]
|
|
[use.keytab = false]
|
|
[authentication.user = admin]
|
|
[authentication.password= d2NjX2NyeXB0ATQxNDU1MzVGNDM0MjQzOzMwMzQzNjQ0Mzk0NTQ2NDY0MzM1MzM0NDM0NDMzMzMxNDEzMzQ1MzA0NTM0MzQ0NDQ0NDQ0NjM0MzM0MzQyNDI7OzMyMzUzMDMwOzc2NjcxMEI0M0JCRDQzQzgwQ0I4NEZGNDU3RkFDQjhBOzlCODhGNUM1RUIxQUI4QUM7NTc0MzQzNUY0MzUyNTk1MDU0NUY0NDQ1NDY0MTU1NEM1NDVGNDQ0RjRENDE0OTRFOzMwOzMxMzQzNTM2MzMzMTMyMzgzMzMzMzIzNzMwOw]</pre>
|
|
</li><li id="mrs_01_1160__en-us_topic_0000001219149041_l0d2da7b91457456ca11e7f56cde26556">When the keytab file mode is used for login, the configuration information example is as follows:<pre class="screen" id="mrs_01_1160__en-us_topic_0000001219149041_s07d1de83858d4e2ab535dfe6091e51fd">[server.url = 10.10.26.187:21351,127.0.0.2:21351]
|
|
[authentication.type = kerberos]
|
|
[use.keytab = true]
|
|
[client.principal = bar]
|
|
[client.keytab = /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/hadoop-config/user.keytab]
|
|
[krb5.conf.file = /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-tool/hadoop-config/krb5.conf]</pre>
|
|
</li></ul>
|
|
</p></li><li id="mrs_01_1160__en-us_topic_0000001219149041_l98dfbf3d724248b692bb27f879308541"><span>Run the following command to edit the job.properties file and configure job information:</span><p><p id="mrs_01_1160__en-us_topic_0000001219149041_af920f2683e9e4c1394ec4a3a67ad2b71"><strong id="mrs_01_1160__en-us_topic_0000001219149041_ae8d7af410742405ebce1b8dff833b790">vi job.properties</strong></p>
|
|
<pre class="screen" id="mrs_01_1160__en-us_topic_0000001219149041_s05e142376cb24378a06a01c7ec9aea26">#job name
|
|
job.jobName = sftp2hdfs-schedule-tool
|
|
|
|
|
|
#Whether to update the loader configuration parameters(File filter)£?This parameter is used to match the import file name.Values are true or false.
|
|
#false means update.the file name which is get by schedule tool will be updated to Loader configuration parameters (File filter).
|
|
#false means no update.the file name which is get by schedule tool will be updated to Loader configuration parameters (import path).
|
|
<strong id="mrs_01_1160__en-us_topic_0000001219149041_a966cddce278c4084b58eb2b3ec3a7a34">file.filter = false</strong>
|
|
|
|
|
|
#File name = prefix + date + suffix
|
|
#Need to import the file name prefix
|
|
<strong id="mrs_01_1160__en-us_topic_0000001219149041_abbc6a528e1aa4228bf30b40ca9db0bf2">file.fileName.prefix=table1</strong>
|
|
|
|
#Need to import the file name suffixes
|
|
<strong id="mrs_01_1160__en-us_topic_0000001219149041_a6e8a211e5d3d4221ac3e87ffccdba08b">file.fileName.posfix=.txt</strong>
|
|
|
|
#Date Days.Value is an integer.
|
|
#According to the date and number of days to get the date of the import file.
|
|
<strong id="mrs_01_1160__en-us_topic_0000001219149041_a43fc654916ba41b88a2dcd353e2f15a3">date.day = 1</strong>
|
|
|
|
#Date Format.Import file name contains the date format.Format Type£ºyyyyMMdd,yyyyMMdd HHmmss,yyyy-MM-dd,yyyy-MM-dd HH:mm:ss
|
|
<strong id="mrs_01_1160__en-us_topic_0000001219149041_a2d19aa2d55804b028d4a555bc2cc42b7">file.date.format = yyyyMMdd</strong>
|
|
|
|
#Date Format.Scheduling script execution. Enter the date format.
|
|
<strong id="mrs_01_1160__en-us_topic_0000001219149041_a41053e00d1ee4dec843721ad2a6a4afb">parameter.date.format = yyyyMMdd</strong>
|
|
|
|
|
|
#Whether the import file is a compressed format.Values ??are true or false.
|
|
#true indicates that the file is a compressed format£?Execution scheduling tool will extract the files.false indicates that the file is an uncompressed.Execution scheduling tool does not unpack.
|
|
file.format.iscompressed = false
|
|
|
|
#Hadoop storage type.Values are HDFS or HBase.
|
|
storage.type = HDFS</pre>
|
|
<p id="mrs_01_1160__en-us_topic_0000001219149041_a29d5743ee16545aa8f572cd11aded8ca">According to the data provided by <a href="#mrs_01_1160__en-us_topic_0000001219149041_la615007634824ea9af1b8e5b93c0867b">1</a>, the filtering rules are set as follows when the <strong id="mrs_01_1160__en-us_topic_0000001219149041_b102292422391535">table120160221.txt</strong> file is used as an example:</p>
|
|
<ul id="mrs_01_1160__en-us_topic_0000001219149041_u3ed3c0cda7564405aa5d924903e62ff7"><li id="mrs_01_1160__en-us_topic_0000001219149041_l3fd04b9a40fd43b7b586876c1134d323">File name prefix:<p id="mrs_01_1160__en-us_topic_0000001219149041_a6a33d2bc3b2743e9a1ecb020c6180b8b"><a name="mrs_01_1160__en-us_topic_0000001219149041_l3fd04b9a40fd43b7b586876c1134d323"></a><a name="en-us_topic_0000001219149041_l3fd04b9a40fd43b7b586876c1134d323"></a>file.fileName.prefix=table1</p>
|
|
</li><li id="mrs_01_1160__en-us_topic_0000001219149041_l9d59d52f7753487a9fe95b817a52c72f">File name suffix:<p id="mrs_01_1160__en-us_topic_0000001219149041_aed5dbdc2c7334d97b89558decf624dc8"><a name="mrs_01_1160__en-us_topic_0000001219149041_l9d59d52f7753487a9fe95b817a52c72f"></a><a name="en-us_topic_0000001219149041_l9d59d52f7753487a9fe95b817a52c72f"></a>file.fileName.posfix=.txt</p>
|
|
</li><li id="mrs_01_1160__en-us_topic_0000001219149041_l839b599022e44befbbc9ac93a73dcc8a">Date format included in the file name:<p id="mrs_01_1160__en-us_topic_0000001219149041_ab11263161a064592a2b413ec1a5b2bfc"><a name="mrs_01_1160__en-us_topic_0000001219149041_l839b599022e44befbbc9ac93a73dcc8a"></a><a name="en-us_topic_0000001219149041_l839b599022e44befbbc9ac93a73dcc8a"></a>file.date.format = yyyyMMdd</p>
|
|
</li><li id="mrs_01_1160__en-us_topic_0000001219149041_lb9db87cbd97043ea9a280d847cbcd5ba">Entered date parameter for invoking the script:<p id="mrs_01_1160__en-us_topic_0000001219149041_abfa2a269b9fb4571bebe2b826dd83d38"><a name="mrs_01_1160__en-us_topic_0000001219149041_lb9db87cbd97043ea9a280d847cbcd5ba"></a><a name="en-us_topic_0000001219149041_lb9db87cbd97043ea9a280d847cbcd5ba"></a>parameter.date.format = yyyyMMdd</p>
|
|
</li><li id="mrs_01_1160__en-us_topic_0000001219149041_l792f1fae05584dc2bafc3f9e2580169a">Number of delayed days.<p id="mrs_01_1160__en-us_topic_0000001219149041_a20b8cbfc1dd6480ca460e09c0c1dd235"><a name="mrs_01_1160__en-us_topic_0000001219149041_l792f1fae05584dc2bafc3f9e2580169a"></a><a name="en-us_topic_0000001219149041_l792f1fae05584dc2bafc3f9e2580169a"></a>date.day = 1</p>
|
|
<p id="mrs_01_1160__en-us_topic_0000001219149041_a5889bbd851ce42b6b8b38c43028bf053">For example, if the input date parameter of the script is <strong id="mrs_01_1160__en-us_topic_0000001219149041_b202598542891535">20160220</strong>, the result is <strong id="mrs_01_1160__en-us_topic_0000001219149041_b363382691535">20160221</strong> by using the addition.</p>
|
|
<div class="note" id="mrs_01_1160__en-us_topic_0000001219149041_ndff93ccd13c2484a9bebd91478604b14"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1160__en-us_topic_0000001219149041_acff90b95ea274951bccbfcad11a10e6f">If the <strong id="mrs_01_1160__en-us_topic_0000001219149041_b19208591137">./run.sh 20160220 /user/loader/schedule_01</strong> command is executed, the preceding filtering rules will be combined into a string: <strong id="mrs_01_1160__en-us_topic_0000001219149041_b103003650291535">"table1"+"20160221"+.txt = table120160221.txt</strong>.</p>
|
|
</div></div>
|
|
</li></ul>
|
|
</p></li><li id="mrs_01_1160__en-us_topic_0000001219149041_l927d8c592bd2416fba5e689b044868d8"><span>Select a filtering rule according to the value of <strong id="mrs_01_1160__en-us_topic_0000001219149041_b36810380591535">file.filter</strong>.</span><p><ul id="mrs_01_1160__en-us_topic_0000001219149041_uf45763aff86b4d5ab8609230236df86f"><li id="mrs_01_1160__en-us_topic_0000001219149041_l4fa2d626073b4817868218b4701ff5c1">If a file is to be exactly matched, go to <a href="#mrs_01_1160__en-us_topic_0000001219149041_lc22613a01864415a95c6937512f03fe6">8</a>.</li><li id="mrs_01_1160__en-us_topic_0000001219149041_l79d8193e700a48ab9734defb96d7ccf9">If a series of files are to be fuzzily matched, go to <a href="#mrs_01_1160__en-us_topic_0000001219149041_le84ccb022b5743d784c050fe68f30871">9</a>.</li></ul>
|
|
</p></li><li id="mrs_01_1160__en-us_topic_0000001219149041_lc22613a01864415a95c6937512f03fe6"><a name="mrs_01_1160__en-us_topic_0000001219149041_lc22613a01864415a95c6937512f03fe6"></a><a name="en-us_topic_0000001219149041_lc22613a01864415a95c6937512f03fe6"></a><span>Change the value of <span class="parmname" id="mrs_01_1160__en-us_topic_0000001219149041_parmname99181040791535"><b>file.filter</b></span> in the <strong id="mrs_01_1160__en-us_topic_0000001219149041_b127073864391535">job.properties</strong> file to <span class="parmvalue" id="mrs_01_1160__en-us_topic_0000001219149041_parmvalue204323121791535"><b>false</b></span>.</span><p><p id="mrs_01_1160__en-us_topic_0000001219149041_a8e448987f566420d8afdb370395a5815">Run the following commands to run the job. The task is completed.</p>
|
|
<p id="mrs_01_1160__en-us_topic_0000001219149041_ab636fe76d35540599cd976a042296124"><strong id="mrs_01_1160__en-us_topic_0000001219149041_b2465185412434">cd /opt/hadoopclient/Loader/loader-tools-1.99.3/schedule-tool</strong></p>
|
|
<p id="mrs_01_1160__en-us_topic_0000001219149041_a5147fb5cd01f4c389a191933347e46dc"><strong id="mrs_01_1160__en-us_topic_0000001219149041_a8f163658c755469388607231cc7b41ed">./run.sh </strong><i><span class="varname" id="mrs_01_1160__en-us_topic_0000001219149041_vabe9fd60ee5e4b1ca5e2a92667c68320">20160220</span></i><strong id="mrs_01_1160__en-us_topic_0000001219149041_a46b4212392d84634be51b4ad31332f6a"> </strong><i><span class="varname" id="mrs_01_1160__en-us_topic_0000001219149041_vff665df98b534eaf83579078a9d74527">/user/loader/schedule_01</span></i></p>
|
|
<p id="mrs_01_1160__en-us_topic_0000001219149041_a6258824ccc8148e5958ccee192fabcf2"><i><span class="varname" id="mrs_01_1160__en-us_topic_0000001219149041_varname158950561591535">20160220</span></i> indicates the input date, and <i><span class="varname" id="mrs_01_1160__en-us_topic_0000001219149041_varname76154621191535">/user/loader/schedule_01</span></i> indicates the output path.</p>
|
|
<div class="note" id="mrs_01_1160__en-us_topic_0000001219149041_ncaaf7b7af1474231ac4f2d1d1068d8f3"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1160__en-us_topic_0000001219149041_a225a493a8a7e4e74bf6b761038792725">The string <span class="filepath" id="mrs_01_1160__en-us_topic_0000001219149041_f836ad92a14484544b0cd24f5988cf396"><b>table120160221.txt</b></span> obtained by combining the preceding filtering rules will be used as the file name and appended to the input path of the job. Therefore, the job will only process the uniquely matched file <span class="filepath" id="mrs_01_1160__en-us_topic_0000001219149041_fa67ef19f835642cd90f23431cb16bf61"><b>table120160221.txt</b></span>.</p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_1160__en-us_topic_0000001219149041_le84ccb022b5743d784c050fe68f30871"><a name="mrs_01_1160__en-us_topic_0000001219149041_le84ccb022b5743d784c050fe68f30871"></a><a name="en-us_topic_0000001219149041_le84ccb022b5743d784c050fe68f30871"></a><span>In the <strong id="mrs_01_1160__en-us_topic_0000001219149041_b5734510091535">job.properties</strong> file, change the value of <span class="parmname" id="mrs_01_1160__en-us_topic_0000001219149041_parmname144073627291535"><b>file.filter</b></span> to <strong id="mrs_01_1160__en-us_topic_0000001219149041_b20385030591535">true</strong>, and set the value of <span class="parmname" id="mrs_01_1160__en-us_topic_0000001219149041_parmname183819893691535"><b>file.fileName.prefix</b></span> to <span class="parmvalue" id="mrs_01_1160__en-us_topic_0000001219149041_parmvalue90613760291535"><b>*</b></span>.</span><p><p id="mrs_01_1160__en-us_topic_0000001219149041_a0772d36c1f11463593fd924f0187c7b2">Run the following commands to run the job. The task is completed.</p>
|
|
<p id="mrs_01_1160__en-us_topic_0000001219149041_a6c3646a1e91a40398fbf9655f2d9f77c"><strong id="mrs_01_1160__en-us_topic_0000001219149041_b3318185512446">cd /opt/hadoopclient/Loader/loader-tools-1.99.3/schedule-tool</strong></p>
|
|
<p id="mrs_01_1160__en-us_topic_0000001219149041_afb6281ae323d46338ce5ef4a19559432"><strong id="mrs_01_1160__en-us_topic_0000001219149041_abb11de0c2402410cbacc71e9dec46e43">./run.sh </strong><i><span class="varname" id="mrs_01_1160__en-us_topic_0000001219149041_v3065cb506954405893494c4748b293e1">20160220</span></i><strong id="mrs_01_1160__en-us_topic_0000001219149041_afe4deecfe31449288febc6e65a00953f"> </strong><i><span class="varname" id="mrs_01_1160__en-us_topic_0000001219149041_v0cf81819cb5a430792cf720588c3dc83">/user/loader/schedule_01</span></i></p>
|
|
<p id="mrs_01_1160__en-us_topic_0000001219149041_aeae955059cf34eb49b4ee0363aeba03e"><i><span class="varname" id="mrs_01_1160__en-us_topic_0000001219149041_varname34215571291535">20160220</span></i> indicates the input date, and <i><span class="varname" id="mrs_01_1160__en-us_topic_0000001219149041_varname37353619691535">/user/loader/schedule_01</span></i> indicates the output path.</p>
|
|
<div class="note" id="mrs_01_1160__en-us_topic_0000001219149041_nfb33996e46e34fb0afaa4eaca859fb52"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1160__en-us_topic_0000001219149041_a999e005a8f7b48c8b29853c636767eaf">The string <span class="filepath" id="mrs_01_1160__en-us_topic_0000001219149041_filepath108849471891535"><b>*20160221.txt</b></span> obtained by combining the preceding filtering rules will be used as the fuzzy match mode of the file filter. In the input path of the job, all files matching <span class="filepath" id="mrs_01_1160__en-us_topic_0000001219149041_filepath20816634391535"><b>*20160221.txt</b></span> will be processed by the job.</p>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1155.html">Client Tool Description</a></div>
|
|
</div>
|
|
</div>
|
|
|