vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_1097.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

206 lines
39 KiB
HTML
Raw Blame History

<a name="mrs_01_1097"></a><a name="mrs_01_1097"></a>
<h1 class="topictitle1">Typical Scenario: Importing Data from a Relational Database to Spark</h1>
<div id="body8662426"><div class="section" id="mrs_01_1097__en-us_topic_0000001219149425_sbe1483c63a2042d8a368924b378d01ad"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1097__en-us_topic_0000001219149425_abba796e6fc314b3b9995d543ee3bac4d">Use Loader to import data from a relational database to Spark.</p>
</div>
<div class="section" id="mrs_01_1097__en-us_topic_0000001219149425_sbd04e892dd9e4f3fbd2785d33552a3c2"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1097__en-us_topic_0000001219149425_u19a0e1f8b4c548aaab0153885a93bef8"><li id="mrs_01_1097__en-us_topic_0000001219149425_le13847b1b6af47a18640e534424842c1">You have obtained the service username and password for creating a Loader job.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_lb6233b640a9146b1ae875a1b250dbd8d">You have had the permission to access the Spark tables that are used during job execution.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_lb713b76d536c4c8fa265d258ecc0e038">You have obtained the username and password of the relational database.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_ldcef94abda5148dbbafe96e8d927e114">No disk space alarm is reported, and the available disk space is sufficient for importing and exporting data.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_lf7b5e67edfeb4b19aa6eb3e4451f77e4">If a configured task requires the Yarn queue function, the user must be authorized with related Yarn queue permission.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_l64f7c2615ead487ca29e9dc1725865e0">The user who configures a task must obtain execution permission on the task and obtain usage permission on the related connection of the task.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_lf23cb247a9b1456483a385effc16792f">Before the operation, perform the following steps:<ol id="mrs_01_1097__en-us_topic_0000001219149425_ofe862618de8e4f9d947d804ca61b6cfb"><li id="mrs_01_1097__en-us_topic_0000001219149425_lfb196d87a953468fabafd0b9a792d8dc">Obtain the JAR package of the relational database driver and save it to the following directory on the active and standby Loader nodes: <strong id="mrs_01_1097__en-us_topic_0000001219149425_b174241297485">${BIGDATA_HOME}/FusionInsight_Porter_<span id="mrs_01_1097__en-us_topic_0000001219149425_text84193924817">8.1.2.2</span>/install/FusionInsight-Sqoop-1.99.3/FusionInsight-Sqoop-1.99.3/server/webapps/loader/WEB-INF/ext-lib</strong>.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_ldc4615fe37934c2f9fd6964c1119be7f">Run the following command on the active and standby nodes as user <strong id="mrs_01_1097__en-us_topic_0000001219149425_b19106311104816">root</strong> to modify the permission:<p id="mrs_01_1097__en-us_topic_0000001219149425_p49418303552"><strong id="mrs_01_1097__en-us_topic_0000001219149425_b75232536535">cd </strong><strong id="mrs_01_1097__en-us_topic_0000001219149425_b8525153175310">${BIGDATA_HOME}/FusionInsight_Porter_<span id="mrs_01_1097__en-us_topic_0000001219149425_text112131152511">8.1.2.2</span></strong><strong id="mrs_01_1097__en-us_topic_0000001219149425_b14526175315316">/install/FusionInsight-Sqoop-1.99.3/FusionInsight-Sqoop-1.99.3/server/webapps/loader/WEB-INF/ext-lib</strong></p>
<p class="litext" id="mrs_01_1097__en-us_topic_0000001219149425_a3bc5b6e4f1c6461ebef673f28d09a5f0"><i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_1097__en-us_topic_0000001219149425_cmdname08015818510">chown omm:wheel</span></b></i><strong id="mrs_01_1097__en-us_topic_0000001219149425_b18273714104817"> </strong><i><span class="varname" id="mrs_01_1097__en-us_topic_0000001219149425_varname0274414194811">JAR package name</span></i></p>
<p class="litext" id="mrs_01_1097__en-us_topic_0000001219149425_a0002a7fe4be34e3ab93c43b808855dd9"><i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_1097__en-us_topic_0000001219149425_cmdname20801281355">chmod 600</span></b></i><strong id="mrs_01_1097__en-us_topic_0000001219149425_b112617151485"> </strong><i><span class="varname" id="mrs_01_1097__en-us_topic_0000001219149425_varname162611015204816">JAR package name</span></i></p>
</li><li id="mrs_01_1097__en-us_topic_0000001219149425_l2e5ecd19fdc342ab8adef03179708936">Log in to FusionInsight Manager. Choose <span id="mrs_01_1097__en-us_topic_0000001219149425_text9587201664810"><strong id="mrs_01_1097__en-us_topic_0000001219149425_b1458791610485">Cluster</strong> &gt; <em id="mrs_01_1097__en-us_topic_0000001219149425_i9587191614488">Name of the desired cluster</em> &gt; </span><strong id="mrs_01_1097__en-us_topic_0000001219149425_b658831654817">Services </strong>&gt; <strong id="mrs_01_1097__en-us_topic_0000001219149425_b658817165487">Loader </strong>&gt; <strong id="mrs_01_1097__en-us_topic_0000001219149425_b18588151614820">More </strong>&gt; <strong id="mrs_01_1097__en-us_topic_0000001219149425_b1358810163486">Restart Service</strong>. Enter the password of the system administrator to restart the Loader service.</li></ol>
</li></ul>
</div>
<div class="section" id="mrs_01_1097__en-us_topic_0000001219149425_section98138591542"><h4 class="sectiontitle">Procedure</h4><p class="tableheading" id="mrs_01_1097__en-us_topic_0000001219149425_acb91a00803bd4943b501e38961c22fdf"><strong id="mrs_01_1097__en-us_topic_0000001219149425_b1847331918489">Setting Basic Job Information</strong></p>
<ol id="mrs_01_1097__en-us_topic_0000001219149425_o02f584aa22f04affb42e0125e44cb8ed"><li id="mrs_01_1097__en-us_topic_0000001219149425_l5ffa3dcea84b45ef8278b654f283ad28"><span>Access the Loader web UI.</span><p><ol type="a" id="mrs_01_1097__en-us_topic_0000001219230551_obbc8c37dc53040efb21dc541b1dfc22c"><li id="mrs_01_1097__en-us_topic_0000001219230551_l6f0ffda40ca543d5a5660461b5e311dd">Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>.</li><li id="mrs_01_1097__en-us_topic_0000001219230551_la11b479a24ad4659a55365c2ede06015">Choose <strong id="mrs_01_1097__en-us_topic_0000001219230551_b39357591297">Cluster</strong> &gt; <em id="mrs_01_1097__en-us_topic_0000001219230551_i18941259112918">Name of the desired cluster</em> &gt; <strong id="mrs_01_1097__en-us_topic_0000001219230551_b19941115916297">Services</strong> &gt; <strong id="mrs_01_1097__en-us_topic_0000001219230551_b1894175916294">Loader</strong>.</li><li id="mrs_01_1097__en-us_topic_0000001219230551_l6601e8a2fd1b4780bf69238d6f5cc7f2">Click <strong id="mrs_01_1097__en-us_topic_0000001219230551_b13499742135213">LoaderServer(</strong><em id="mrs_01_1097__en-us_topic_0000001219230551_i187499244301">Node name</em><strong id="mrs_01_1097__en-us_topic_0000001219230551_b18777163515522">, Active)</strong>. The Loader web UI is displayed.<div class="fignone" id="mrs_01_1097__fig1919114183218"><span class="figcap"><b>Figure 1 </b>Loader web UI</span><br><span><img id="mrs_01_1097__mrs_01_1089_image155554364292" src="en-us_image_0000001438241209.png"></span></div>
</li></ol>
</p></li><li id="mrs_01_1097__en-us_topic_0000001219149425_lbf5ca76c10954cdbb79c1d85e56af94b"><span>Click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol17471720114810"><b>New Job</b></span> to go to the <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname14472102014817"><b>Basic Information</b></span> page and set basic job information.</span><p><div class="fignone" id="mrs_01_1097__en-us_topic_0000001219149425_ff9eec34da6fd48f49b5ecd6ad50d1d09"><span class="figcap"><b>Figure 2 </b><span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname9151202284810"><b>Basic Information</b></span></span><br><span><img id="mrs_01_1097__en-us_topic_0000001219149425_iee8f72477eaa4783b62ae241bc600481" src="en-us_image_0000001349059761.png"></span></div>
<ol class="subitemlist" type="a" id="mrs_01_1097__en-us_topic_0000001219149425_oe02ea71a84344bd09145df2f6a544e94"><li id="mrs_01_1097__en-us_topic_0000001219149425_l007bd6ee25a040f39dde0cbf04b5b173">Set <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname201681139144814"><b>Name</b></span> to the name of the job.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_le7642201105d49a2837855feb3a0922c">Set <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname18172134014483"><b>Type</b></span> to <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname717214405481"><b>Import</b></span>.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_l8d24ce3011604083a09586838bccb74a">Set <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname68494117481"><b>Group</b></span> to the group to which the job belongs. No group is created by default. You need to click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol685194119489"><b>Add</b></span> to create a group and click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol12851441164817"><b>OK</b></span> to save the created group.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_l16cc1d0efa9b4322b637e64327fc9c0a">Set <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname1254194374813"><b>Queue</b></span> to the Yarn queue that executes the job. The default value is <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue171571107340"><b>root.default</b></span>.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_lcd5827d517ef4e4c9f98997638b20d8a">Set <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname1170611454487"><b>Priority</b></span> to the priority of the Yarn queue that executes the job. The default value is <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue11961347194810"><b>NORMAL</b></span>. The options are <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue13248154874813"><b>VERY_LOW</b></span>, <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue924864884818"><b>LOW</b></span>, <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue6248194819487"><b>NORMAL</b></span>, <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue624944811483"><b>HIGH</b></span>, and <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue42491948114820"><b>VERY_HIGH</b></span>.</li></ol>
</p></li><li id="mrs_01_1097__en-us_topic_0000001219149425_lafa0a56f6b994ef18580fb90cae6e488"><span>In the <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname106231449104814"><b>Connection</b></span> area, click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol3623184994814"><b>Add</b></span> to create a connection, set <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname106240492488"><b>Connector</b></span> to <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue46241749164819"><b>generic-jdbc-connector</b></span> or a dedicated database connector (oracle-connector, oracle-partition-connector, or mysql-fastpath-connector), set connection parameters, and click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol5624114913486"><b>Test</b></span> to verify whether the connection is available. When "Test Success" is displayed, click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol26251349144819"><b>OK</b></span>.</span><p><div class="note" id="mrs_01_1097__en-us_topic_0000001219149425_n9eb286cca3864ab1a9511038598ec14d"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1097__en-us_topic_0000001219149425_u3650cf4545454e51ba9acf314126037f"><li id="mrs_01_1097__en-us_topic_0000001219149425_l81e3372e79054652ae76e224876b5d6a">For connection to relational databases, general database connectors (generic-jdbc-connector) or dedicated database connectors (oracle-connector, oracle-partition-connector, and mysql-fastpath-connector) are available. However, compared with general database connectors, dedicated database connectors perform better in data import and export because they are optimized for specific database types.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_lf03fd4f537cf4d338547260bf4e9b166">When mysql-fastpath-connector is used, the <strong id="mrs_01_1097__en-us_topic_0000001219149425_b157138528488">mysqldump </strong>and <strong id="mrs_01_1097__en-us_topic_0000001219149425_b57181526488">mysqlimport</strong> commands of MySQL must be available on NodeManager nodes, and the MySQL client version to which the two commands belong must be compatible with the MySQL server version. If the two commands are unavailable or the versions are incompatible, install the MySQL client applications and tools following the instructions at <a href="http://dev.mysql.com/doc/refman/5.7/en/linux-installation-rpm.html" target="_blank" rel="noopener noreferrer">http://dev.mysql.com/doc/refman/5.7/en/linux-installation-rpm.html</a>.</li></ul>
</div></div>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1097__en-us_topic_0000001219149425_t28700deed8cb4e3eac36db33c6b0b1b7" frame="border" border="1" rules="all"><caption><b>Table 1 </b><span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue10841954184810"><b>generic-jdbc-connector</b></span> connection parameters</caption><thead align="left"><tr id="mrs_01_1097__en-us_topic_0000001219149425_r6705d0bec14f46a495d77bc87c501cbf"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.3.3.2.2.2.4.1.1"><p id="mrs_01_1097__en-us_topic_0000001219149425_ac719381587e942c88376130ff38d87eb">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="35.620000000000005%" id="mcps1.3.3.3.3.2.2.2.4.1.2"><p id="mrs_01_1097__en-us_topic_0000001219149425_a3c57a233691140d8be33efef067e9134">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="39.379999999999995%" id="mcps1.3.3.3.3.2.2.2.4.1.3"><p id="mrs_01_1097__en-us_topic_0000001219149425_aa6a4543369f64ef5884064c6f60755d5">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1097__en-us_topic_0000001219149425_rf0f9d97d1b1a4b6a8cc5e6766274df73"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a9d3b50a0ade74529b2e9ef87803d2ef5">Name</p>
</td>
<td class="cellrowborder" valign="top" width="35.620000000000005%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a956cad24ccca4511b4ff4a0ba33fc67f">Name of a relational database connection</p>
</td>
<td class="cellrowborder" valign="top" width="39.379999999999995%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_af4ff5c3cf32547e89e8edf9d62b2e102">dbName</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_reff2b6a6503045afb9d71e6065a5738e"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a598363318b734605b757252de4d2379f">JDBC Driver Class</p>
</td>
<td class="cellrowborder" valign="top" width="35.620000000000005%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a5fd25cf100c64a78987009eeb066688c">Name of a JDBC driver class</p>
</td>
<td class="cellrowborder" valign="top" width="39.379999999999995%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a2f39c00ed3c54564a3a954d0581cf856">oracle.jdbc.driver.OracleDriver</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_rc48128237652440fb751bd9fa0f38e84"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a25327c38906d46659a4ab272752a60ca">JDBC Connection String</p>
</td>
<td class="cellrowborder" valign="top" width="35.620000000000005%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a9938ddad4a7b424d9d58ab7eef04fe48">JDBC connection string</p>
</td>
<td class="cellrowborder" valign="top" width="39.379999999999995%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_aad9182b94e16405aa580dfbb356248da">jdbc:oracle:thin:@//10.16.0.1:1521/oradb</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_rd2b9fa08135743c686d9a6330cf00354"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a512f153469a3475a90518978f99a2968">Username</p>
</td>
<td class="cellrowborder" valign="top" width="35.620000000000005%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a7a24f0872fc546a6875e869fe84688e8">Username for connecting to the database</p>
</td>
<td class="cellrowborder" valign="top" width="39.379999999999995%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a9682070f467e4680a7a640faf6d29bb4">omm</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_r2805a523516c412c9120de1b86fb2379"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a139176436241470abc082b5e93e156fa">Password</p>
</td>
<td class="cellrowborder" valign="top" width="35.620000000000005%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_ab0cefa7570f544cab5faa35d47c1949f">Password for connecting to the database</p>
</td>
<td class="cellrowborder" valign="top" width="39.379999999999995%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a0abf485732004b1a8bfa807efb27b058">xxxx</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_rf679c6ee07df4167b0df2677ea3cca4f"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a49ff525f4bb042a9afa0f7393fb5e90b">JDBC Connection Properties</p>
</td>
<td class="cellrowborder" valign="top" width="35.620000000000005%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a6a793bbe561d45f5aae6c512c2437351">JDBC connection attribute. Click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol475421224919"><b>Add</b></span> to manually add the attribute.</p>
<ul id="mrs_01_1097__en-us_topic_0000001219149425_u8d76b6264b5b4ea1af143fca2ddf9f01"><li id="mrs_01_1097__en-us_topic_0000001219149425_l6341e8f730844a09acc7d2268c4eb33e">Name: connection attribute name</li><li id="mrs_01_1097__en-us_topic_0000001219149425_lae056ab6ac3d41ffa332537202b1e176">Value: connection attribute value</li></ul>
</td>
<td class="cellrowborder" valign="top" width="39.379999999999995%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><ul id="mrs_01_1097__en-us_topic_0000001219149425_ua0de27c3a14e48c9bc133c67dc522003"><li id="mrs_01_1097__en-us_topic_0000001219149425_l8d8b7d24c04c482b871a36a162c4d553">Name: <strong id="mrs_01_1097__en-us_topic_0000001219149425_b139202142498">socketTimeout</strong></li><li id="mrs_01_1097__en-us_topic_0000001219149425_l8a90ddf94cdb4be2b395b9b91a01af36">Value: <strong id="mrs_01_1097__en-us_topic_0000001219149425_b16936915184910">20</strong></li></ul>
</td>
</tr>
</tbody>
</table>
</div>
<p class="tableheading" id="mrs_01_1097__en-us_topic_0000001219149425_a564713c9172c4b2985e2544741b28d1e"><strong id="mrs_01_1097__en-us_topic_0000001219149425_b1371015164496">Setting Data Source Information</strong></p>
</p></li><li id="mrs_01_1097__en-us_topic_0000001219149425_l8847854b2ccd4ec7b452119117007a59"><span>Click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol1746818174492"><b>Next</b></span>. On the displayed <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname646810178495"><b>From</b></span> page, set the data source information.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1097__en-us_topic_0000001219149425_tea084a1870624c2fb4d4d1ea94556f98" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1097__en-us_topic_0000001219149425_r38bfee4963f74013b4a29385e02d953d"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.3.4.2.1.2.4.1.1"><p id="mrs_01_1097__en-us_topic_0000001219149425_a527b472ade944d8b89eb40836aae1343">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="49.43%" id="mcps1.3.3.3.4.2.1.2.4.1.2"><p id="mrs_01_1097__en-us_topic_0000001219149425_a5753b2a26fe04c889a13998e4a19979c">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25.569999999999997%" id="mcps1.3.3.3.4.2.1.2.4.1.3"><p id="mrs_01_1097__en-us_topic_0000001219149425_a0b21decc89784dbe8e4d80ac4e13d5f4">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1097__en-us_topic_0000001219149425_rd1b2988522884332a9acbad8826911b9"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a65a7d235ca20426ead101f38fb8d64c7">Schema Name</p>
</td>
<td class="cellrowborder" valign="top" width="49.43%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_af4044d13093b4df798b914016eafba05">Database schema name. This parameter exists in the <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname2457172294912"><b>Table name</b></span> schema.</p>
</td>
<td class="cellrowborder" valign="top" width="25.569999999999997%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_ae1e616c42f5f46199f4581bf6ebc12d9">dbo</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_r6bcf0c6566b04c26a10b436c0c5888ad"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a9c1feee03a0540b9b858a95339b5e840">Table Name</p>
</td>
<td class="cellrowborder" valign="top" width="49.43%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_aed29a76001754188894479ff9d01cc46">Database table name. This parameter exists in the <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname6158122614919"><b>Table name</b></span> schema.</p>
</td>
<td class="cellrowborder" valign="top" width="25.569999999999997%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a2b038ca6a7914ef78a45419d6a3381a0">test</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_r4d29bc597eeb42879efb94c84e7582b7"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_af8c29498b4e54f59b7b3386d8ea2331a">SQL Statement</p>
</td>
<td class="cellrowborder" valign="top" width="49.43%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_afed8277a0da743b5be5af1d0e35a7b60">SQL statement for Loader to query data to be imported in <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname1778172812491"><b>Table SQL statement</b></span> mode. The SQL statement requires the query condition <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue209407297492"><b>WHERE ${CONDITIONS}</b></span>. Without this condition, the SQL statement cannot be run properly. An example SQL statement is as follows: <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue78579302497"><b>select * from TABLE WHERE A&gt;B and ${CONDITIONS}</b></span>. If <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname138521631184915"><b>Table column names</b></span> is set, the column specified by <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname585263111499"><b>Table column names</b></span> will replace the column queried in the SQL statement. This parameter cannot be set when <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname2744932144912"><b>Schema name</b></span> or <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname474433264917"><b>Table name</b></span> is set.</p>
<div class="note" id="mrs_01_1097__en-us_topic_0000001219149425_n3e0d5e53a9d442f8a928f5d3cdba345f"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1097__en-us_topic_0000001219149425_a601216da7c2843c5890623ba9198c1be">You can use macros to define SQL Where statements. For details, see <a href="mrs_01_1153.html">Using Macro Definitions in Configuration Items</a>.</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="25.569999999999997%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a96b1e6a5a6ea4ed78b73fdb383a8491a">select * from test where ${CONDITIONS}</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_r3eb72b978f454a1aa30cd18b454d7a15"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_abe453bdaf9af4182a75e854a71488692">Table Column Names</p>
</td>
<td class="cellrowborder" valign="top" width="49.43%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a1f3e18dc6af44dccbca088705b7a9982">Table columns whose content is to be imported by Loader. Use commas (<span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue1936203817490"><b>,</b></span>) to separate multiple fields.</p>
<p id="mrs_01_1097__en-us_topic_0000001219149425_abb5f4523b4344a86b463d848924e430f">If the parameter is not set, all the columns are imported and the <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue170193984914"><b>Select *</b></span> order is used as the column location.</p>
</td>
<td class="cellrowborder" valign="top" width="25.569999999999997%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_af41dc424413b4abb88d23ac7f065f129">-</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_r5a2f49fded024fb3835890f57a903be0"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a7f5ee68baf664cc0820afec8e62e3634">Partition Column Name</p>
</td>
<td class="cellrowborder" valign="top" width="49.43%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a62b4e730515e420db048b2d6381f9f18">Database table column based on which to-be-imported data is determined. This parameter is used for partitioning in a Map job. You are advised to configure the primary key field.</p>
<div class="note" id="mrs_01_1097__en-us_topic_0000001219149425_ncc07bdf2870c4603bf0d8daf91859f75"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="mrs_01_1097__en-us_topic_0000001219149425_ude58957ee9f44e3dab15f1819fbed252"><li id="mrs_01_1097__en-us_topic_0000001219149425_l6bed632a68834504b3142752774f166d">A partition column must have an index. If no index exists, do not specify a partition column. If a partition column without an index is specified, the database server disk I/O will be busy, the access of other services to the database will be affected, and the import will take a long period.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_ld307468eeccd4f1db40b072bc9532ca3">In multiple fields with indexes, select the field that has the most discrete value as the partition column. A partition column that is not discrete may result in load imbalance when multiple MapReduce jobs are imported.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_li395643261327">The sorting rules of partition columns must be case-sensitive. Otherwise, data may be lost during data import.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_l6d3afb94526447d6bcf120d635e8826b">You are not advised to select fields of the float or double type for the partition column. Otherwise, the records containing the minimum and maximum values of the partition column may fail to be imported due to precision issues.</li></ul>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="25.569999999999997%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a701de7d40cde48d49797905e5d1e7a42">id</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_r86d58bf229f14c3598ebccfac71eff50"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_aec7d2e74ba704d3b94bc4e617863021e">Nulls in Partition Column</p>
</td>
<td class="cellrowborder" valign="top" width="49.43%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_ab9a30ab2f7d8470d94ec336599922840">Indicates whether to process records whose values are null in database table columns.</p>
<ul id="mrs_01_1097__en-us_topic_0000001219149425_u124cb2ba284346b68b0d0745a49aa265"><li id="mrs_01_1097__en-us_topic_0000001219149425_le261d10fbf4142218889ba2d32257857"><span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue18112350154911"><b>true</b></span>: Records whose values are null are processed.</li><li id="mrs_01_1097__en-us_topic_0000001219149425_l9cc79c27abd64bb88f946b436d9ef9a7"><span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue414135184919"><b>false</b></span>: Records whose values are not null are processed.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="25.569999999999997%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a6f3bc255e7fd4134b30b2c759620abb4">true</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_rf9232dcf33284c3a8f3583beb36c490c"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_afdf5a4409d434147a3c62ec33b5c74bf">Whether to Specify a Partition Column</p>
</td>
<td class="cellrowborder" valign="top" width="49.43%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a074938fad5d243399b70dcd67b8a838f">Indicates whether to specify a partition column.</p>
</td>
<td class="cellrowborder" valign="top" width="25.569999999999997%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a22c7201eea9a4ac69eb9ae799981bcd1">true</p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="tableheading" id="mrs_01_1097__en-us_topic_0000001219149425_a647050af26fd4131a3e0ea40d0a8a2b5"><strong id="mrs_01_1097__en-us_topic_0000001219149425_b18746165514493">Setting Data Transformation</strong></p>
</p></li><li id="mrs_01_1097__en-us_topic_0000001219149425_l18ddb7dc29c242498e8dced8f813833c"><span>Click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol365865654916"><b>Next</b></span>. On the displayed <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname14658356184916"><b>Transform</b></span> page, set the transformation operations in the data transformation process. For details about how to select operators and set parameters, see <a href="mrs_01_1119.html">Operator Help</a> and <a href="#mrs_01_1097__en-us_topic_0000001219149425_table895989011525">Table 3</a>.</span><p>
<div class="tablenoborder"><a name="mrs_01_1097__en-us_topic_0000001219149425_table895989011525"></a><a name="en-us_topic_0000001219149425_table895989011525"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1097__en-us_topic_0000001219149425_table895989011525" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Input and output parameters of the operator</caption><thead align="left"><tr id="mrs_01_1097__en-us_topic_0000001219149425_row1060779011525"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.3.5.2.1.2.3.1.1"><p id="mrs_01_1097__en-us_topic_0000001219149425_p1556013211525">Input Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.3.5.2.1.2.3.1.2"><p id="mrs_01_1097__en-us_topic_0000001219149425_p5241116811525">Output Type</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1097__en-us_topic_0000001219149425_row6122078811525"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_p5993680211525">Table input</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_p58185532115718">Spark output</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="fignone" id="mrs_01_1097__en-us_topic_0000001219149425_f89c899fd32af4ce0963df4d76a004a63"><span class="figcap"><b>Figure 3 </b>Operator operation procedure</span><br><span><img id="mrs_01_1097__en-us_topic_0000001219149425_iec48631789984afe87c4c6a43853b333" src="en-us_image_0000001296059912.png"></span></div>
<p class="tableheading" id="mrs_01_1097__en-us_topic_0000001219149425_a89522e4fba284ff18a79fdff1971e857"><strong id="mrs_01_1097__en-us_topic_0000001219149425_b174331857185018">Setting Data Storage Information and Executing the Job</strong></p>
</p></li><li id="mrs_01_1097__en-us_topic_0000001219149425_lbd439bad1fe34567b27a6a38304b44e1"><span>Click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol7364100115119"><b>Next</b></span>. On the displayed <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname63648095110"><b>To</b></span> page, set <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname236512055118"><b>Storage type</b></span> to <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue736513045111"><b>SPARK</b></span>.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1097__en-us_topic_0000001219149425_t26c37534a4da450c894d546208933055" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1097__en-us_topic_0000001219149425_rbc48f5844638493388583b7033062996"><th align="left" class="cellrowborder" valign="top" width="29.830000000000002%" id="mcps1.3.3.3.6.2.1.2.4.1.1"><p id="mrs_01_1097__en-us_topic_0000001219149425_a52308f6a7c2842a7ab6aeb745337e612">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="53.31%" id="mcps1.3.3.3.6.2.1.2.4.1.2"><p id="mrs_01_1097__en-us_topic_0000001219149425_a53ea65e61f9845d1a3e5da09e837cc76">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="16.86%" id="mcps1.3.3.3.6.2.1.2.4.1.3"><p id="mrs_01_1097__en-us_topic_0000001219149425_a2454ba4099ab4f6cae584bcd3b9b6b57">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1097__en-us_topic_0000001219149425_rafb343aafe5d48e2a69b93da5d39c163"><td class="cellrowborder" valign="top" width="29.830000000000002%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a6e331715f90148dd8e7e055859357c5f">Output Directory</p>
</td>
<td class="cellrowborder" valign="top" width="53.31%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_aad7804f8723b44009d075399f979b061">Directory for storing data imported into Spark.</p>
<div class="note" id="mrs_01_1097__en-us_topic_0000001219149425_ne6ea340f52b2421db8132652758bcbd2"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1097__en-us_topic_0000001219149425_a3f5cb2542c37459996a08cd3f5bb20a7">You can use macros to define path parameters. For details, see <a href="mrs_01_1153.html">Using Macro Definitions in Configuration Items</a>.</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="16.86%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a28facf66a66f453683e8362888342628">/opt/tempfile</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_r12171a689e304112af5c035b44a0c0d0"><td class="cellrowborder" valign="top" width="29.830000000000002%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a130bb05b55914f5cadc54c3b30cd36fb">Extractors</p>
</td>
<td class="cellrowborder" valign="top" width="53.31%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a905386fba1a64869a657d25283b95a27">Number of Maps that are started at the same time in a MapReduce task of a data configuration operation. The value must be less than or equal to 3000. You are advised to set the parameter to the maximum number of connections on the SFTP server.</p>
</td>
<td class="cellrowborder" valign="top" width="16.86%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a68a57b5b11f04f97b11b8d5a5dbc0313">20</p>
</td>
</tr>
<tr id="mrs_01_1097__en-us_topic_0000001219149425_rbc2ff8ad53e74706ba80f9116f03dab6"><td class="cellrowborder" valign="top" width="29.830000000000002%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a85f768497d3c4a19883516c88916f81e">Extractor Size</p>
</td>
<td class="cellrowborder" valign="top" width="53.31%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1097__en-us_topic_0000001219149425_a1b7c3f54e73f4e34890725fa9744d22e">Spark does not support this parameter. Please set <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname15541232155117"><b>Extractors</b></span>.</p>
</td>
<td class="cellrowborder" valign="top" width="16.86%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1097__en-us_topic_0000001219149425_aae92b6b92e9f4307b34862c7badcf8d0">-</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="mrs_01_1097__en-us_topic_0000001219149425_l73c29eebd475487ca51cd7acf41a980b"><span>Click <span class="uicontrol" id="mrs_01_1097__en-us_topic_0000001219149425_uicontrol18863434105114"><b>Save and run</b></span> to save and run the job.</span><p><p id="mrs_01_1097__en-us_topic_0000001219149425_a55897583c68146efbce2b281182939e7"><strong id="mrs_01_1097__en-us_topic_0000001219149425_b1867218359518">Checking the Job Execution Result</strong></p>
</p></li><li id="mrs_01_1097__en-us_topic_0000001219149425_l44bee943737f48ddb9845ebd1f1cf56b"><span>Go to the Loader web UI. When <span class="parmname" id="mrs_01_1097__en-us_topic_0000001219149425_parmname1068723611513"><b>Status</b></span> is <span class="parmvalue" id="mrs_01_1097__en-us_topic_0000001219149425_parmvalue19687193685110"><b>Succeeded</b></span>, the job is complete.</span><p><div class="fignone" id="mrs_01_1097__en-us_topic_0000001219149425_f1ee1b9b7329e4030abde34080992b346"><span class="figcap"><b>Figure 4 </b>Viewing job details</span><br><span><img id="mrs_01_1097__en-us_topic_0000001219149425_i70e785e61a774d10bd8cf84d2bb2b9cb" src="en-us_image_0000001295900076.png"></span></div>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1086.html">Importing Data</a></div>
</div>
</div>
View Git Blame Copy Permalink