liangxiaobi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_1089.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

248 lines
46 KiB
HTML
Raw Blame History

<a name="mrs_01_1089"></a><a name="mrs_01_1089"></a>
<h1 class="topictitle1">Typical Scenario: Importing Data from an SFTP Server to HDFS or OBS</h1>
<div id="body8662426"><div class="section" id="mrs_01_1089__en-us_topic_0000001173630728_sac6c286724514c4386fafae64a332b96"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1089__en-us_topic_0000001173630728_a7af2d3bfa4db4a21ba2d05e158c20bfa">Use Loader to import data from an SFTP server to HDFS or OBS.</p>
</div>
<div class="section" id="mrs_01_1089__en-us_topic_0000001173630728_sec52dde8eaa949ada9e8bf641d389a70"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1089__en-us_topic_0000001173630728_ufafa70a936764eccbd3bdb6aa3107f99"><li id="mrs_01_1089__en-us_topic_0000001173630728_la22b762cc4d44411a2583dbf95a5a635">You have obtained the service username and password for creating a Loader job.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l9e8640d5b4ea4fc0a7c32cc1fd567039">You have had the permission to access the HDFS or OBS directories and data involved in job execution.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l548d19b153634716841893b1632a0cfd">You have obtained the username and password of the SFTP server as well as the read permission for the source files on the SFTP server. If file name extension needs to be added after a source file is imported, the user must have the write permission of the source file.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l95a5781dfd034042aadb6223c8a547af">No disk space alarm is reported, and the available disk space is sufficient for importing and exporting data.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l9ba7bc99292c4018a09b925cffdf72cd">When using Loader to import data from the SFTP server, the input paths and input path subdirectories of the SFTP server and the name of the files in these directories do not contain any of the special characters /"':;.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_lcc5b94be302a49bd9ed6c4ed89b978cb">If a configured task requires the Yarn queue function, the user must be authorized with related Yarn queue permission.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l17bd3194d658498e9ec9a60de5ca2ac3">The user who configures a task must obtain execution permission on the task and obtain usage permission on the related connection of the task.</li></ul>
</div>
<div class="section" id="mrs_01_1089__en-us_topic_0000001173630728_s6bdf8d3807114184a0dbf05cfccefa59"><h4 class="sectiontitle">Procedure</h4><p class="tableheading" id="mrs_01_1089__en-us_topic_0000001173630728_a68b43fe9da0e4830b35c9a99ad7045d3"><strong id="mrs_01_1089__en-us_topic_0000001173630728_b1323822710316">Setting Basic Job Information</strong></p>
<ol id="mrs_01_1089__en-us_topic_0000001173630728_o3231f47326c744ab8e329396df28e7c8"><li id="mrs_01_1089__en-us_topic_0000001173630728_l405ec938379246c887dd0dea1aacaee0"><span>Access the Loader web UI.</span><p><ol type="a" id="mrs_01_1089__en-us_topic_0000001219230551_obbc8c37dc53040efb21dc541b1dfc22c"><li id="mrs_01_1089__en-us_topic_0000001219230551_l6f0ffda40ca543d5a5660461b5e311dd">Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>.</li><li id="mrs_01_1089__en-us_topic_0000001219230551_la11b479a24ad4659a55365c2ede06015">Choose <strong id="mrs_01_1089__en-us_topic_0000001219230551_b39357591297">Cluster</strong> &gt; <em id="mrs_01_1089__en-us_topic_0000001219230551_i18941259112918">Name of the desired cluster</em> &gt; <strong id="mrs_01_1089__en-us_topic_0000001219230551_b19941115916297">Services</strong> &gt; <strong id="mrs_01_1089__en-us_topic_0000001219230551_b1894175916294">Loader</strong>.</li><li id="mrs_01_1089__en-us_topic_0000001219230551_l6601e8a2fd1b4780bf69238d6f5cc7f2">Click <strong id="mrs_01_1089__en-us_topic_0000001219230551_b13499742135213">LoaderServer(</strong><em id="mrs_01_1089__en-us_topic_0000001219230551_i187499244301">Node name</em><strong id="mrs_01_1089__en-us_topic_0000001219230551_b18777163515522">, Active)</strong>. The Loader web UI is displayed.<div class="fignone" id="mrs_01_1089__en-us_topic_0000001219230551_fe1215781f5014239823e95a7ae45a780"><span class="figcap"><b>Figure 1 </b>Loader web UI</span><br><span><img id="mrs_01_1089__image155554364292" src="en-us_image_0000001438241209.png"></span></div>
</li></ol>
</p></li><li id="mrs_01_1089__en-us_topic_0000001173630728_la3e6481520ba4aa9b7c15f711285adbf"><span>Click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol16529123943112"><b>New Job</b></span> to go to the <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname4529539113110"><b>Basic Information</b></span> page and set basic job information.</span><p><div class="fignone" id="mrs_01_1089__en-us_topic_0000001173630728_f2f323c12924544be9ba995604cdc1ef1"><span class="figcap"><b>Figure 2 </b><span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname283419421318"><b>Basic Information</b></span></span><br><span><img id="mrs_01_1089__en-us_topic_0000001173630728_iec9af8836846426eb6289aae208e46c3" src="en-us_image_0000001349139581.png"></span></div>
<ol class="subitemlist" type="a" id="mrs_01_1089__en-us_topic_0000001173630728_ofaf977ce8afa41ee86ee7ff83af3ca7b"><li id="mrs_01_1089__en-us_topic_0000001173630728_le00cd42435814cc4ac48a6e443b0076f">Set <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname165701731153320"><b>Name</b></span> to the name of the job.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l283b9e3953b448a2a59dc0182759e1e9">Set <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname648319210415"><b>Type</b></span> to <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname194881721414"><b>Import</b></span>.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_la42af73205034c2b981053e9faa1d29a">Set <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname1767512415415"><b>Group</b></span> to the group to which the job belongs. No group is created by default. You need to click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol136751243418"><b>Add</b></span> to create a group and click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol2675154164117"><b>OK</b></span> to save the created group.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l29969ff62101429e83b70e9ee1baff65">Set <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname69155918416"><b>Queue</b></span> to the Yarn queue that executes the job. The default value is <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue7645132164218"><b>root.default</b></span>.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l0b57d91b256f4d64a18a5b92d077a89d">Set <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname638384134219"><b>Priority</b></span> to the priority of the Yarn queue that executes the job. The default value is <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue518207174213"><b>NORMAL</b></span>. The options are <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue348619954212"><b>VERY_LOW</b></span>, <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue74876914428"><b>LOW</b></span>, <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue8487997420"><b>NORMAL</b></span>, <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue18487394428"><b>HIGH</b></span>, and <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue1948813924217"><b>VERY_HIGH</b></span>.</li></ol>
</p></li><li id="mrs_01_1089__en-us_topic_0000001173630728_l80ec40880c1d422eb5f1cf6f9338fbaf"><span>In the <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname1359231719422"><b>Connection</b></span> area, click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol13598141714424"><b>Add</b></span> to create a connection, set <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname2059871712425"><b>Connector</b></span> to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue1759981764214"><b>sftp-connector</b></span>, click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol55996177421"><b>Add</b></span>, set connection parameters, and click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol12600161774219"><b>Test</b></span> to verify whether the connection is available. When "Test Success" is displayed, click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol0601151764212"><b>OK</b></span>. Loader allows multiple SFTP servers to be configured. Click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol1367141094414"><b>Add</b></span> to add the configuration information of multiple SFTP servers.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1089__en-us_topic_0000001173630728_tdc3c42f270914515a45194bdf1a8e3bf" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Connection parameters</caption><thead align="left"><tr id="mrs_01_1089__en-us_topic_0000001173630728_r4a9bd3d3b07b4eb5a5b920abbadd8238"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.3.3.2.1.2.4.1.1"><p id="mrs_01_1089__en-us_topic_0000001173630728_aabd0d97fd45441fa8ba7b2835ec43c5a">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="54.32%" id="mcps1.3.3.3.3.2.1.2.4.1.2"><p id="mrs_01_1089__en-us_topic_0000001173630728_a2f38480e1a6f41068809c8a8e9ac8128">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="20.68%" id="mcps1.3.3.3.3.2.1.2.4.1.3"><p id="mrs_01_1089__en-us_topic_0000001173630728_adf4509d138f24103ac16633e00948d07">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1089__en-us_topic_0000001173630728_rf530a02ce6d44313a19101f84670ffc7"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a94358dd28df74231833ae98b2a6c1bb9">Name</p>
</td>
<td class="cellrowborder" valign="top" width="54.32%" headers="mcps1.3.3.3.3.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_ad72fcb85198848a18a767bdf92a49586">Name of the SFTP server connection</p>
</td>
<td class="cellrowborder" valign="top" width="20.68%" headers="mcps1.3.3.3.3.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a354b82aceec64bb79613f578f9595e46">sftpName</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r7cc66426e105478bb6dbae5f6c77eaf3"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a82c58d5d73e24abe946816b5181f0327">SFTP Server IP Address</p>
</td>
<td class="cellrowborder" valign="top" width="54.32%" headers="mcps1.3.3.3.3.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_acfe9bb14671a421da466fd3b10c3e503">IP address of the SFTP server</p>
</td>
<td class="cellrowborder" valign="top" width="20.68%" headers="mcps1.3.3.3.3.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_adc85699eab83434996d3f76eeb6405c4">10.16.0.1</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_rbc5a2c7af4dc4e43992e486038d306a8"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a2e23916a09cd4f74a1936379a5254579">SFTP Server Port</p>
</td>
<td class="cellrowborder" valign="top" width="54.32%" headers="mcps1.3.3.3.3.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a942c13b81bc74c789aa5b313fad3c0f5">Port number of the SFTP server</p>
</td>
<td class="cellrowborder" valign="top" width="20.68%" headers="mcps1.3.3.3.3.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a3465d96a75f64ea99c94ae29dae8df34">22</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r5067e9cf962d4e53854c6a04499278ec"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a1f091032d5b6425ba4a873fb3ab89e32">SFTP Username</p>
</td>
<td class="cellrowborder" valign="top" width="54.32%" headers="mcps1.3.3.3.3.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a364aad934c9c4690aa8502d5a8fefc15">Username for accessing the SFTP server</p>
</td>
<td class="cellrowborder" valign="top" width="20.68%" headers="mcps1.3.3.3.3.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_ae4bef44b553046cc8fd69495e8c15568">root</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r3ec48a41affb4aaa9e6a06135c8cf387"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a61328a2c3323485691bbe5bc86afb0cd">SFTP Password</p>
</td>
<td class="cellrowborder" valign="top" width="54.32%" headers="mcps1.3.3.3.3.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_adcc1555b9a6a4acb80c867351880957f">Password for accessing the SFTP server</p>
</td>
<td class="cellrowborder" valign="top" width="20.68%" headers="mcps1.3.3.3.3.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a61775ee1fad54c63918053eeae615f1e">xxxx</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_rcb5e7c31d4104a6cab455b1dc30eea80"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a0add38a0e91c4457a630482139360916">SFTP Public Key</p>
</td>
<td class="cellrowborder" valign="top" width="54.32%" headers="mcps1.3.3.3.3.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a9f96474c31dc42048f5e986ca5b5038f">Public key of the SFTP server</p>
</td>
<td class="cellrowborder" valign="top" width="20.68%" headers="mcps1.3.3.3.3.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a4270a9e7ac16480cb245487ac3baa280">OdDt/yn...etM</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="mrs_01_1089__en-us_topic_0000001173630728_na5e1891a633743aa87f659954979f3da"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1089__en-us_topic_0000001173630728_a5754bdf3b9904f42912723fe4dce7464">When multiple SFTP servers are configured, the data in the specified directories of the SFTP servers is imported to the same directory in HDFS or OBS.</p>
</div></div>
<p class="tableheading" id="mrs_01_1089__en-us_topic_0000001173630728_a43e5fbdfe8c84d559ba0cce68a55e8f9"><strong id="mrs_01_1089__en-us_topic_0000001173630728_b894513364619">Setting Data Source Information</strong></p>
</p></li><li id="mrs_01_1089__en-us_topic_0000001173630728_ld89f266a8f1147d5a82f74d0ec3640c4"><span>Click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol10652183784620"><b>Next</b></span>. On the displayed <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname665843718468"><b>From</b></span> page, set the data source information.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1089__en-us_topic_0000001173630728_te352463976e346de86ba1023e026e4ff" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1089__en-us_topic_0000001173630728_ra898f2a422f94f81aa70652c7ac5a264"><th align="left" class="cellrowborder" valign="top" width="13.530000000000003%" id="mcps1.3.3.3.4.2.1.2.4.1.1"><p id="mrs_01_1089__en-us_topic_0000001173630728_a692244a6db844f158febb1b301930ef5">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="71.62%" id="mcps1.3.3.3.4.2.1.2.4.1.2"><p id="mrs_01_1089__en-us_topic_0000001173630728_ade75ec6172ea46eeba53f7d147db254a">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="14.850000000000001%" id="mcps1.3.3.3.4.2.1.2.4.1.3"><p id="mrs_01_1089__en-us_topic_0000001173630728_a0e44dbf71a6c41aba5a085cea2190b67">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1089__en-us_topic_0000001173630728_rb3cbefc09587424bbf081554fceeb51d"><td class="cellrowborder" valign="top" width="13.530000000000003%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a8045b7f8c6c44780b4094d15c06baf15">Input Path</p>
</td>
<td class="cellrowborder" valign="top" width="71.62%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a3650246f415c46cd9c79a9626fcdde6d">Input path or name of the source file on an SFTP server. If multiple SFTP server IP addresses are configured for the connector, you can set this parameter to multiple input paths separated with semicolons (;). Ensure that the number of input paths is the same as that of SFTP servers configured for the connector.</p>
<div class="note" id="mrs_01_1089__en-us_topic_0000001173630728_n9086fdcbdd5c496995c3730a048e8855"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1089__en-us_topic_0000001173630728_a03ea6223dfea4232835c6157a7a884ce">You can use macros to define path parameters. For details, see <a href="mrs_01_1153.html">Using Macro Definitions in Configuration Items</a>.</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="14.850000000000001%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_acf303071234d416aaed38ae1cee12f4c">/opt/tempfile;/opt</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r4409ee3519a64acf8163b86567d0acc9"><td class="cellrowborder" valign="top" width="13.530000000000003%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a4b12fbc0b0e640b99ad01eb47fa0d22e">File Split Type</p>
</td>
<td class="cellrowborder" valign="top" width="71.62%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a08deb830b3584880a70bc931ebe11668">Indicates whether to split source files by file name or size. The files obtained after the splitting are used as the input files of each Map in the MapReduce task for data import.</p>
<ul id="mrs_01_1089__en-us_topic_0000001173630728_u033a4e9e104c4d1882b0f64ed22ca91b"><li id="mrs_01_1089__en-us_topic_0000001173630728_l48f824023e314869af5815817e36fca4"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue13471154416477"><b>FILE</b></span>: indicates that the source file is split by file. That is, each Map processes one or multiple complete files, the same source file cannot be allocated to different Maps, and the source file directory structure is retained after data import.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l8003bc75bffe489eaa57ae0f6d744c23"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue143614518474"><b>SIZE</b></span>: indicates that the source file is split by size. That is, each Map processes input files of a certain size, and a source file can be divided and processed by multiple Maps. After data is stored in the output directory, the number of saved files is the same as that of Maps. The file name format is <span class="filepath" id="mrs_01_1089__en-us_topic_0000001173630728_filepath174319515477"><b>import_part_</b></span><em id="mrs_01_1089__en-us_topic_0000001173630728_i159090511489">xxxx</em>, where <em id="mrs_01_1089__en-us_topic_0000001173630728_i1936555594816">xxxx</em> is a unique random number generated by the system.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="14.850000000000001%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a19a16554588e4615896625e0486a7239">FILE</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r817ed0d5f9804b798f41b716fc6071da"><td class="cellrowborder" valign="top" width="13.530000000000003%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a00159da0ac774e8185db56cb4504fd82">Filter Type</p>
</td>
<td class="cellrowborder" valign="top" width="71.62%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a132597b356ef48658258536509f97878">File filter condition. This parameter is used when <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname114014364917"><b>Path Filter</b></span> or <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue171492384918"><b>File Filter</b></span> is set.</p>
<ul id="mrs_01_1089__en-us_topic_0000001173630728_ucf03621c72c24634822643a8d77e5d10"><li id="mrs_01_1089__en-us_topic_0000001173630728_l87892eac9d574e47acb3739b88b54ee3"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue18804152574914"><b>WILDCARD</b></span>: indicates using a wildcard.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_lca28e228f76441c48da2c1571ad58d6c"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue16661112913491"><b>REGEX</b></span>: indicates using a regular expression.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l54622f0da449489e9d4c0993044b774f">If the parameter is not set, a wildcard is used by default.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="14.850000000000001%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_aba9ee515713542f4b19952ee0ddd490d">WILDCARD</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_rb11d7cc0509140f8923f4d2e75574a8e"><td class="cellrowborder" valign="top" width="13.530000000000003%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_acc4cc61fd2ef4881a9a5535e19eb009a">Path Filter</p>
</td>
<td class="cellrowborder" valign="top" width="71.62%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a961d540bcf474a91a2b46b3a9a4bfaf5">Wildcard or regular expression for filtering the directories in the input path of the source files. This parameter is used when <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname2939240144913"><b>Filter Type</b></span> is set. <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname7151558134910"><b>Input Path</b></span> is not used for filtering. Use semicolons (;) to separate the path filters on multiple servers and use commas (,) to separate the filter conditions of each server. If this parameter is left empty, directories are not filtered.</p>
<ul id="mrs_01_1089__en-us_topic_0000001173630728_ufa036572c60f4f1089f372458f31a642"><li id="mrs_01_1089__en-us_topic_0000001173630728_lbeb14024b3b24804888b88cf04e6c5f7"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue103121328175011"><b>?</b></span> matches a single character.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_leab714952e56402a91a6779d1739a1d3"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue8205161585413"><b>*</b></span> indicates multiple characters.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_la27e7a8e7b724311be8cb444b931da0a">Adding <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue6262131811544"><b>^</b></span> before the condition indicates negated filtering, that is, file filtering.</li></ul>
<p id="mrs_01_1089__en-us_topic_0000001173630728_a7f4fd440b2bf4b49a2d053712eb5e2d9">For example, when <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname1666172625413"><b>Filter type</b></span> is set to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue26771126145413"><b>WILDCARD</b></span>, set the parameter to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue068119260546"><b>*</b></span>; when <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname1668518261544"><b>Filter type</b></span> is set to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue15689202618542"><b>REGEX</b></span>, set the parameter to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue069332685412"><b>\\.*</b></span>.</p>
</td>
<td class="cellrowborder" valign="top" width="14.850000000000001%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a5650d593b32049349efee33d135f43ad">1*,2*;1*</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_rc2f2effb5e9d45d883e78d108ca13b37"><td class="cellrowborder" valign="top" width="13.530000000000003%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a633612e48203471bb79f2ef7bf229efa">File Filter</p>
</td>
<td class="cellrowborder" valign="top" width="71.62%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a0c35c49697304969a8a01a28d0774fd4">Wildcard or regular expression for filtering the file names of the source files. This parameter is used when <strong id="mrs_01_1089__en-us_topic_0000001173630728_b8526191655512">Filter Type</strong> is set. Use semicolons (;) to separate the path filters on multiple servers and use commas (,) to separate the filter conditions of each server. This parameter cannot be left blank.</p>
<ul id="mrs_01_1089__en-us_topic_0000001173630728_u143faa33365348119b786384d4976b00"><li id="mrs_01_1089__en-us_topic_0000001173630728_lff766da48e044ad6b48dc15959266770"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue97351033155516"><b>?</b></span> matches a single character.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l85a74be68ef9422ba0efb29530ae2515"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue6376153515555"><b>*</b></span> indicates multiple characters.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l61f66f2cde554cdfba40b058571127bd">Adding <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue183111937175515"><b>^</b></span> before the condition indicates negated filtering, that is, file filtering.</li></ul>
<p id="mrs_01_1089__en-us_topic_0000001173630728_ae3af4e6e68d64ca98ecdb98592a86df5">For example, when <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname1649213399550"><b>Filter type</b></span> is set to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue8497203914557"><b>WILDCARD</b></span>, set the parameter to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue650212397554"><b>*</b></span>; when <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname150653910558"><b>Filter type</b></span> is set to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue6511173965511"><b>REGEX</b></span>, set the parameter to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue3515339155510"><b>\\.*</b></span>.</p>
</td>
<td class="cellrowborder" valign="top" width="14.850000000000001%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_ac945c44c182c47f1a974e2467c58ace0">*.txt,*.csv;*.txt</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r58b0504e34484fcd825cad19198615ca"><td class="cellrowborder" valign="top" width="13.530000000000003%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a5a040e96b28848d194322067d903d067">Encoding Type</p>
</td>
<td class="cellrowborder" valign="top" width="71.62%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a9ce13cc3188d4d5b8e5ccb1b997a4a2c">Source file encoding format, for example, UTF-8 and GBK. This parameter can be set only in text file import.</p>
</td>
<td class="cellrowborder" valign="top" width="14.850000000000001%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a86a7fb8bf0c74a7b9597e0ea4c084d44">UTF-8</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r2f7c4d83fef14f8ebc2f0fa8a2086fe8"><td class="cellrowborder" valign="top" width="13.530000000000003%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_aaba6fa56b5bd4192bf060b944df24046">Suffix</p>
</td>
<td class="cellrowborder" valign="top" width="71.62%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_ac5f3f4921c674240a7307002aa6ea2e1">File name extension added to a source file after the source file is imported. If this parameter is empty, no file name extension is added to the source file. This parameter is valid only when the data source is a file system. You are advised to set this parameter in incremental data import.</p>
<p id="mrs_01_1089__en-us_topic_0000001173630728_a4f304a4c17784cdc82eeca946fe66997">For example, if the parameter is set to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue18547439185614"><b>.txt</b></span> and the source file is <span class="filepath" id="mrs_01_1089__en-us_topic_0000001173630728_filepath5558163913569"><b>test-loader.csv</b></span>, the source file name is <span class="filepath" id="mrs_01_1089__en-us_topic_0000001173630728_filepath9563153945619"><b>test-loader.csv.txt</b></span> after export.</p>
</td>
<td class="cellrowborder" valign="top" width="14.850000000000001%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a806af8daf240499181e0849650e02025">.log</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r9c1cc88f86f945c192adf92d1e57e5ce"><td class="cellrowborder" valign="top" width="13.530000000000003%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a4d7812272f564c09afb5308eff961a13">Compression</p>
</td>
<td class="cellrowborder" valign="top" width="71.62%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a1190406d82a049f38611aec7d0a2f4af">Indicates whether to enable compressed transmission when SFTP is used to export data.</p>
<ul id="mrs_01_1089__en-us_topic_0000001173630728_uf01ad28f20694ef18d1c0202f9df3a93"><li id="mrs_01_1089__en-us_topic_0000001173630728_l3e58ae78a48c466cb169d3c329cf5a52">The value <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue91112146574"><b>true</b></span> indicates that compression is enabled.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l66c85bee2ec34d4b8ba5cc4871ebb573">The value <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue1825211895720"><b>false</b></span> indicates that compression is disabled.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="14.850000000000001%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_ad2b18343e05d4f3f9f16cc4ce88e73ae">true</p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="tableheading" id="mrs_01_1089__en-us_topic_0000001173630728_a1356a618ae384581a11927243142c16b"><strong id="mrs_01_1089__en-us_topic_0000001173630728_b1981113246577">Setting Data Transformation</strong></p>
</p></li><li id="mrs_01_1089__en-us_topic_0000001173630728_lc45b822bbff14ec4a007a9d461fbc30c"><span>Click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol176563035720"><b>Next</b></span>. On the displayed <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname3786305571"><b>Transform</b></span> page, set the transformation operations in the data transformation process. For details about how to select operators and set parameters, see <a href="mrs_01_1119.html">Operator Help</a> and <a href="#mrs_01_1089__en-us_topic_0000001173630728_table895989011525">Table 3</a>.</span><p>
<div class="tablenoborder"><a name="mrs_01_1089__en-us_topic_0000001173630728_table895989011525"></a><a name="en-us_topic_0000001173630728_table895989011525"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1089__en-us_topic_0000001173630728_table895989011525" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Input and output parameters of the operator</caption><thead align="left"><tr id="mrs_01_1089__en-us_topic_0000001173630728_row1060779011525"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.3.5.2.1.2.3.1.1"><p id="mrs_01_1089__en-us_topic_0000001173630728_p1556013211525">Input Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.3.5.2.1.2.3.1.2"><p id="mrs_01_1089__en-us_topic_0000001173630728_p5241116811525">Output Type</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1089__en-us_topic_0000001173630728_row671911525"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_p54425911525">CSV File Input</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_p25487286115718">File Output</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_row42347907115217"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_p7628470115217">HTML Input</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_p13926313115217">File Output</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_row29417297115246"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_p33990823115246">Fixed File Input</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_p41907484115714">File Output</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="fignone" id="mrs_01_1089__en-us_topic_0000001173630728_f914fcaa8436241cca6bc15fa3b8a59cf"><span class="figcap"><b>Figure 3 </b>Operator operation procedure</span><br><span><img id="mrs_01_1089__en-us_topic_0000001173630728_i0363a3f3b3494e258a0eb83ee6e615a6" src="en-us_image_0000001348739893.png"></span></div>
<p class="tableheading" id="mrs_01_1089__en-us_topic_0000001173630728_a2683c206682347c681845e38bd0aa521"><strong id="mrs_01_1089__en-us_topic_0000001173630728_b173543618017">Setting Data Storage Information and Executing the Job</strong></p>
</p></li><li id="mrs_01_1089__en-us_topic_0000001173630728_lca5aef87bb944312977b434c21f6ca8b"><span>Click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol115301015015"><b>Next</b></span>. On the displayed <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname859161010010"><b>To</b></span> page, set <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname155921011011"><b>Storage type</b></span> to <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue186071016014"><b>HDFS</b></span>.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1089__en-us_topic_0000001173630728_t9852f71f74874b9c99fdf7cf9c320d5c" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1089__en-us_topic_0000001173630728_r4498fdda01c34f76b9b1f5c13dc2fd88"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.3.6.2.1.2.4.1.1"><p id="mrs_01_1089__en-us_topic_0000001173630728_a7ada0074d8294004b696e2d9f825087d">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="62.72%" id="mcps1.3.3.3.6.2.1.2.4.1.2"><p id="mrs_01_1089__en-us_topic_0000001173630728_af655d6a21210450195212d81e112c791">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="12.280000000000001%" id="mcps1.3.3.3.6.2.1.2.4.1.3"><p id="mrs_01_1089__en-us_topic_0000001173630728_abe00359eb26f4e11b0683a05b7dee218">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1089__en-us_topic_0000001173630728_r875afaafd67845fbb51391e782dff14c"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_aadec71733d1a48b491f87536dc44db2a">File Type</p>
</td>
<td class="cellrowborder" valign="top" width="62.72%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a365f122415d1483d93b8be66c4d7d005">Type of a file after the file is imported. The options are as follows:</p>
<ul id="mrs_01_1089__en-us_topic_0000001173630728_ud85a6923b22c4d8796512dcf42fb7c20"><li id="mrs_01_1089__en-us_topic_0000001173630728_lb2906683783c4f05b0eedf6a6903bc7d"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue774315401309"><b>TEXT_FILE</b></span>: imports a text file and stores it as a text file.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_le5ebf5fad44d447b8323aa23bb31c0e7"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue49925430010"><b>SEQUENCE_FILE</b></span>: imports a text file and stores it as a sequence file.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_la649c869aa0f4885b7c910afd6fe3a4c"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue9538129110"><b>BINARY_FILE</b></span>: imports files of any format by using binary streams.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="12.280000000000001%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a7062baad34f1495e9f01ca7a30000510">TEXT_FILE</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_re2d62dc09a3e4facb53bada1ec454842"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_afccbdd367ace48899acd4966f957df65">Compression Format</p>
</td>
<td class="cellrowborder" valign="top" width="62.72%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_ab5333911885346d082bc26675bd46fc9">Compression format of files imported to HDFS or OBS. Select a format from the drop-down list. If you select <strong id="mrs_01_1089__en-us_topic_0000001173630728_b4292112714111">NONE</strong> or do not set this parameter, data is not compressed.</p>
</td>
<td class="cellrowborder" valign="top" width="12.280000000000001%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a15486ccd6a0445ea8aa1c9486bdf1779">NONE</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r95d10b37ed7f4b89a8de1028ef0d95b0"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_ab7600def3b6948ce9a64ebace77b1020">Output Directory</p>
</td>
<td class="cellrowborder" valign="top" width="62.72%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a5f832474da874f26ab280408da14681f">Directory for storing data imported into HDFS or OBS.</p>
<div class="note" id="mrs_01_1089__en-us_topic_0000001173630728_n8cf89303f09041aa9bb8dda71b3527e0"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1089__en-us_topic_0000001173630728_ab3839a53667940409aab8a9349ec93d2">You can use macros to define path parameters. For details, see <a href="mrs_01_1153.html">Using Macro Definitions in Configuration Items</a>.</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="12.280000000000001%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a7999783c41314ec18c6f8a1cea5848a7">/user/test</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_rfdea601961f144b1803b48ef5adfd26c"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a505fe36bafac4f4ba1d9d23b41e38875">Operation</p>
</td>
<td class="cellrowborder" valign="top" width="62.72%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a50d3fb9fc94047e9a0caf0a959f4d667">Action during data import. When all data is to be imported from the input path to the destination path, the data is stored in a temporary directory and then copied from the temporary directory to the destination path. After the data is imported successfully, the data is deleted from the temporary directory. One of the following actions can be taken when duplicate file names exist during data transfer:</p>
<ul id="mrs_01_1089__en-us_topic_0000001173630728_u66faf1de07d64354a692891966104bcd"><li id="mrs_01_1089__en-us_topic_0000001173630728_l8e838468b8c94682834c5f1905087378"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue178761201623"><b>OVERRIDE</b></span>: overrides the old file.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l2e1fde05c4bf4bcdaecc98a7e2e8a413"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue1989912110218"><b>RENAME</b></span>: renames as new file. For a file without an extension, a string is added to the file name as the extension; for a file with an extension, a string is added to the extension. The string is unique.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l039ddcfd31e94eceab9b1d21f53f15f5"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue058111273211"><b>APPEND</b></span>: adds the content of the new file to the end of the old file. This action only adds content regardless of whether the file can be used. For example, a text file can be used after this operation, while a compressed file cannot.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_le63a100ba7d34e6c91ae328aa675881c"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue1098818362218"><b>IGNORE</b></span>: reserves the old file and does not copy the new file.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l231e0258fa7c4ad39e33117146f3e41f"><span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue112991239520"><b>ERROR</b></span>: stops the task and reports an error if duplicate file names exist. Transferred files are imported successfully, while files that have duplicate names and files that are not transferred fail to be imported.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="12.280000000000001%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a804c469fcb0c4153b0540a4eb44a59ea">OVERRIDE</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_r251bcc47f1a145a0b501765b997d6043"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a3535652458b24d5da48cae6ced7785ae">Extractors</p>
</td>
<td class="cellrowborder" valign="top" width="62.72%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a36c08d4d17844598b2dab5012d08ba66">Number of Maps that are started at the same time in a MapReduce task of a data configuration operation. This parameter cannot be set when <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname982861416315"><b>Extractor Size</b></span> is set. The value must be less than or equal to 3000. You are advised to set the parameter to the number of CPU cores on the SFTP server.</p>
<div class="note" id="mrs_01_1089__en-us_topic_0000001173630728_n345deb23c18f48eab07e9e89075ce023"><span class="notetitle"> NOTE: </span><div class="notebody"><p class="textintable" id="mrs_01_1089__en-us_topic_0000001173630728_aa589924ad3724bc4b89fd0a1875b58b1">To improve the data import speed, ensure that the following conditions are met:</p>
<ul id="mrs_01_1089__en-us_topic_0000001173630728_u559b23c662af495e9a26fe00c84aec61"><li id="mrs_01_1089__en-us_topic_0000001173630728_l6c564471dc724c938f7acc1a876ba9c2">Each Map connection is equivalent to a client connection. Therefore, you must ensure that the maximum number of connections of the SFTP server is greater than the number of Maps.</li><li id="mrs_01_1089__en-us_topic_0000001173630728_l995ce02c17f3440cbeea9f1c3e9b28d1">Ensure that the disk I/O or network bandwidth on the SFTP server does not reach the upper limit.</li></ul>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="12.280000000000001%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a77cdead92bd14bdba26d85e900368bc9">20</p>
</td>
</tr>
<tr id="mrs_01_1089__en-us_topic_0000001173630728_rfea70a52b1f047858bceb35d2aa5ecf9"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1089__en-us_topic_0000001173630728_a430d3972fe414ffd9405008dfbe373c9">Extractor Size</p>
</td>
<td class="cellrowborder" valign="top" width="62.72%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1089__en-us_topic_0000001173630728_aed17077eff2049009093ffa558f9db62">Size of data processed by Maps that are started in a MapReduce task of a data configuration operation. The unit is MB. The value must be greater than or equal to 100. The recommended value is 1000. This parameter cannot be set when <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname46966537316"><b>Extractors</b></span> is set.</p>
</td>
<td class="cellrowborder" valign="top" width="12.280000000000001%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1089__en-us_topic_0000001173630728_ab1f3ef40c35a46b7b1223febbc45617f">1000</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="mrs_01_1089__en-us_topic_0000001173630728_l116c22aeb9734568886ea867e520cc27"><span>Click <span class="uicontrol" id="mrs_01_1089__en-us_topic_0000001173630728_uicontrol1787614575317"><b>Save and run</b></span> to save and run the job.</span><p><p id="mrs_01_1089__en-us_topic_0000001173630728_a9b8253bf64b94b3eafaddd6efa121e7c"><strong id="mrs_01_1089__en-us_topic_0000001173630728_b0296759537">Checking the Job Execution Result</strong></p>
</p></li><li id="mrs_01_1089__en-us_topic_0000001173630728_l3f2748edb8f3458fa2f6fccf7bbba997"><span>Go to the Loader web UI. When <span class="parmname" id="mrs_01_1089__en-us_topic_0000001173630728_parmname9678715414"><b>Status</b></span> is <span class="parmvalue" id="mrs_01_1089__en-us_topic_0000001173630728_parmvalue46791911744"><b>Succeeded</b></span>, the job is complete.</span><p><div class="fignone" id="mrs_01_1089__en-us_topic_0000001173630728_f6c7d800e4d2a45da9f9672d8d2b33ad1"><span class="figcap"><b>Figure 4 </b>Viewing job details</span><br><span><img id="mrs_01_1089__en-us_topic_0000001173630728_ia5e94abc31a3428aaaca819fd1f345ba" src="en-us_image_0000001349259169.png"></span></div>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1086.html">Importing Data</a></div>
</div>
</div>
View Git Blame Copy Permalink