forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
167 lines
32 KiB
HTML
167 lines
32 KiB
HTML
<a name="mrs_01_1110"></a><a name="mrs_01_1110"></a>
|
|
|
|
<h1 class="topictitle1">Typical Scenario: Exporting Data from Spark to a Relational Database</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_1110__en-us_topic_0000001219029079_s57f2588755ae4dc68eb8eecdec9c5ad1"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1110__en-us_topic_0000001219029079_abbed688c143d4984b94937e7491b4b2b">This section describes how to use Loader to export data from Spark to a relational database.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1110__en-us_topic_0000001219029079_s59e0d71baf5947ae95b3ea52e268de28"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1110__en-us_topic_0000001219029079_u5d9fd8eee15940f5a4c4039bddb18bb5"><li id="mrs_01_1110__en-us_topic_0000001219029079_ld33c231a3adf47b5b8093762a1b81f33">You have obtained the service username and password for creating a Loader job.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_l4d0415902345491196b7ade4e6a876f0">You have had the permission to access the Spark tables that are used during job execution.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_l006b71bfd6ad4961b5bbac7b1cfce0f8">You have obtained the username and password of the relational database.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_lf3ab9a89c83c43aebb54be997bcbe1fe">No disk space alarm is reported, and the available disk space is sufficient for importing and exporting data.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_lddaac9ecfefe452fb9394e883ad85b59">If a configured task requires the Yarn queue function, the user must be authorized with related Yarn queue permission.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_l2719080729d54b52b95a6ed0ff6ebb4b">The user who configures a task must obtain execution permission on the task and obtain usage permission on the related connection of the task.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_l4e4f9e2ee45246a0b7e477791c199939">Before the operation, perform the following steps:<ol id="mrs_01_1110__en-us_topic_0000001219029079_o89616ce16d1f4dd69eb953690e627044"><li id="mrs_01_1110__en-us_topic_0000001219029079_l92a1374a82f242d4882d606e21950f04">Obtain the JAR package of the relational database driver and save it to the following directory on the active and standby Loader nodes: <strong id="mrs_01_1110__en-us_topic_0000001219029079_b92701336152111">${BIGDATA_HOME}/FusionInsight_Porter_<span id="mrs_01_1110__en-us_topic_0000001219029079_text3270133613211">8.1.2.2</span>/install/FusionInsight-Sqoop-1.99.3/FusionInsight-Sqoop-1.99.3/server/webapps/loader/WEB-INF/ext-lib</strong>.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_le947c9f4e214447e9c2d7850abdaf8a6">Run the following command on the active and standby nodes as user root to modify the permission:<p id="mrs_01_1110__en-us_topic_0000001219029079_p87291655171216"><a name="mrs_01_1110__en-us_topic_0000001219029079_le947c9f4e214447e9c2d7850abdaf8a6"></a><a name="en-us_topic_0000001219029079_le947c9f4e214447e9c2d7850abdaf8a6"></a><strong id="mrs_01_1110__en-us_topic_0000001219029079_b2645161215135">cd ${BIGDATA_HOME}/FusionInsight_Porter_<span id="mrs_01_1110__en-us_topic_0000001219029079_text295203322820">8.1.2.2</span>/install/FusionInsight-Sqoop-1.99.3/FusionInsight-Sqoop-1.99.3/server/webapps/loader/WEB-INF/ext-lib</strong></p>
|
|
<p class="litext" id="mrs_01_1110__en-us_topic_0000001219029079_a74ad4d741023487fb1575eb6d26659ed"><b><span class="cmdname" id="mrs_01_1110__en-us_topic_0000001219029079_cmdname664524015515">chown omm:wheel</span></b><strong id="mrs_01_1110__en-us_topic_0000001219029079_ad2424186651045a0b8835f781a4ebab3"> </strong><i><span class="varname" id="mrs_01_1110__en-us_topic_0000001219029079_v376d7c20e44d412b893927d74d52d414">JAR package name</span></i></p>
|
|
<p class="litext" id="mrs_01_1110__en-us_topic_0000001219029079_aafa07b879f43462db20986ceb09b6d56"><b><span class="cmdname" id="mrs_01_1110__en-us_topic_0000001219029079_cmdname146451401858">chmod 600</span></b><strong id="mrs_01_1110__en-us_topic_0000001219029079_a517b627e1dbc466e9d5d99e0b66cb07e"> </strong><i><span class="varname" id="mrs_01_1110__en-us_topic_0000001219029079_v77f1e5ee567940159b4cb8fdf28788c9">JAR package name</span></i></p>
|
|
</li><li id="mrs_01_1110__en-us_topic_0000001219029079_lf94d7e81cea343e9a8f7ac643d3d0ed1">Log in to FusionInsight Manager. Choose <span id="mrs_01_1110__en-us_topic_0000001219029079_text0275125881615"><strong id="mrs_01_1110__en-us_topic_0000001219029079_b82741258191616">Cluster</strong> > <em id="mrs_01_1110__en-us_topic_0000001219029079_i14274115871616">Name of the desired cluster</em> > </span><strong id="mrs_01_1110__en-us_topic_0000001219029079_b16275458161612">Service </strong>> <strong id="mrs_01_1110__en-us_topic_0000001219029079_b15275175811161">Loader </strong>> <strong id="mrs_01_1110__en-us_topic_0000001219029079_b6275115811167">More </strong>> <strong id="mrs_01_1110__en-us_topic_0000001219029079_b527665831616">Restart</strong>. Enter the password of the system administrator to restart the Loader service.</li></ol>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1110__en-us_topic_0000001219029079_section1292312371052"><h4 class="sectiontitle">Procedure</h4><p class="tableheading" id="mrs_01_1110__en-us_topic_0000001219029079_a129736089d904547888921ceb4f1b43a"><strong id="mrs_01_1110__en-us_topic_0000001219029079_a8e3f1cf35ca34b688fef89f2abc4efe4">Setting Basic Job Information</strong></p>
|
|
<ol id="mrs_01_1110__en-us_topic_0000001219029079_o0b716d85bff848aca719dedbb9a8f497"><li id="mrs_01_1110__en-us_topic_0000001219029079_l0100a2c1d083462d864ba7bf4b9caec0"><span>Access the Loader web UI.</span><p><ol type="a" id="mrs_01_1110__en-us_topic_0000001219230551_obbc8c37dc53040efb21dc541b1dfc22c"><li id="mrs_01_1110__en-us_topic_0000001219230551_l6f0ffda40ca543d5a5660461b5e311dd">Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>.</li><li id="mrs_01_1110__en-us_topic_0000001219230551_la11b479a24ad4659a55365c2ede06015">Choose <strong id="mrs_01_1110__en-us_topic_0000001219230551_b39357591297">Cluster</strong> > <em id="mrs_01_1110__en-us_topic_0000001219230551_i18941259112918">Name of the desired cluster</em> > <strong id="mrs_01_1110__en-us_topic_0000001219230551_b19941115916297">Services</strong> > <strong id="mrs_01_1110__en-us_topic_0000001219230551_b1894175916294">Loader</strong>.</li><li id="mrs_01_1110__en-us_topic_0000001219230551_l6601e8a2fd1b4780bf69238d6f5cc7f2">Click <strong id="mrs_01_1110__en-us_topic_0000001219230551_b13499742135213">LoaderServer(</strong><em id="mrs_01_1110__en-us_topic_0000001219230551_i187499244301">Node name</em><strong id="mrs_01_1110__en-us_topic_0000001219230551_b18777163515522">, Active)</strong>. The Loader web UI is displayed.<div class="fignone" id="mrs_01_1110__fig10607922203218"><span class="figcap"><b>Figure 1 </b>Loader web UI</span><br><span><img id="mrs_01_1110__mrs_01_1089_image155554364292" src="en-us_image_0000001438241209.png"></span></div>
|
|
</li></ol>
|
|
</p></li><li id="mrs_01_1110__en-us_topic_0000001219029079_l24d2a03e8e7b41b68ea6b86986041cd0"><span>Click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_ufa5dd95d96b14736b4790d05221d9f2c"><b>New Job</b></span> to go to the <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_p89be3d23744c4dffa32d3fa8f4bfe182"><b>Basic Information</b></span> page and set basic job information.</span><p><div class="fignone" id="mrs_01_1110__en-us_topic_0000001219029079_f3edd564b0cfa4056b374628f10876f71"><span class="figcap"><b>Figure 2 </b><strong id="mrs_01_1110__en-us_topic_0000001219029079_b12135908279160">Basic Information</strong></span><br><span><img id="mrs_01_1110__en-us_topic_0000001219029079_i41b571e1c8354a3192986ea309073bbe" src="en-us_image_0000001349139617.png"></span></div>
|
|
<ol class="subitemlist" type="a" id="mrs_01_1110__en-us_topic_0000001219029079_oeddaa7fe985f416b8c2c511093f07d0b"><li id="mrs_01_1110__en-us_topic_0000001219029079_l8f7d5dc0fb834dd2a954d934203684e1">Set <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_peb8e26728df24570a174593ca3bb3704"><b>Name</b></span> to the name of the job.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_l8b7c6eb545924862be40c288d324b6db">Set <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_p93cd747996a443d79dd4619b3dbde98a"><b>Type</b></span> to <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_p4b57284515b04b74bfbe2554b40d5743"><b>Export</b></span>.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_ld9a2e9858c724cad8d49f4f797f20780">Set <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_parmname14509806509160"><b>Group</b></span> to the group to which the job belongs. No group is created by default. You need to click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_uicontrol485175419160"><b>Add</b></span> to create a group and click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_uicontrol10608594059160"><b>OK</b></span> to save the created group.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_l4c1f87635906408ebed4ab155ef8ff2e">Set <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_p321f8d1ad22c40c2a79865b752a40738"><b>Queue</b></span> to the Yarn queue that executes the job. The default value is <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue107501829123918"><b>root.default</b></span>.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_lc0330ab595ec411e875f9c2416cafb59">Set <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_pdf122a04830044eebc0b808b3455a34c"><b>Priority</b></span> to the priority of the Yarn queue that executes the job. The default value is <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_p4c0628a263274678a94ed89736e7f04e"><b>NORMAL</b></span>. The options are <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue18076710599160"><b>VERY_LOW</b></span>, <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue10422524659160"><b>LOW</b></span>, <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue2183069039160"><b>NORMAL</b></span>, <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue19816086759160"><b>HIGH</b></span>, and <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue17998800529160"><b>VERY_HIGH</b></span>.</li></ol>
|
|
</p></li><li id="mrs_01_1110__en-us_topic_0000001219029079_lc74cf2d05d15440aa25de75cc64578f3"><span>In the <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_parmname11731174809160"><b>Connection</b></span> area, click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_uicontrol12321749469160"><b>Add</b></span> to create a connection, set <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_parmname11956765789160"><b>Connector</b></span> to <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue1595141639160"><b>generic-jdbc-connector</b></span> or dedicated database connector (oracle-connector, oracle-partition-connector or mysql-fastpath-connector), set connection parameters, and click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_uicontrol7735726579160"><b>Test</b></span> to verify whether the connection is available. When "<span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue18257779199160"><b>Test Success</b></span>" is displayed, click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_uicontrol2585721009160"><b>OK</b></span>.</span><p><div class="note" id="mrs_01_1110__en-us_topic_0000001219029079_nbd0800c7d094422894b55135386f5192"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1110__en-us_topic_0000001219029079_u6f6713a608c1400f88209eabe636c535"><li id="mrs_01_1110__en-us_topic_0000001219029079_l1a750687ad8841dfb524da72771be8e9">For connection to relational databases, general database connectors (generic-jdbc-connector) or dedicated database connectors (oracle-connector, oracle-partition-connector, and mysql-fastpath-connector) are available. However, compared with general database connectors, dedicated database connectors perform better in data import and export because they are optimized for specific database types.</li><li id="mrs_01_1110__en-us_topic_0000001219029079_l64b15e0144e84024b4bb4a180a7dfd76">When <strong id="mrs_01_1110__en-us_topic_0000001219029079_b19792941009160">mysql-fastpath-connector</strong> is used, the <strong id="mrs_01_1110__en-us_topic_0000001219029079_b16433802959160">mysqldump </strong>and <strong id="mrs_01_1110__en-us_topic_0000001219029079_b18101059469160">mysqlimport </strong>commands of MySQL must be available on NodeManagers, and the MySQL client version to which the two commands belong must be compatible with the MySQL server version. If the two commands are unavailable or the versions are incompatible, see <a href="http://dev.mysql.com/doc/refman/5.7/en/linux-installation-rpm.html" target="_blank" rel="noopener noreferrer">http://dev.mysql.com/doc/refman/5.7/en/linux-installation-rpm.html</a>. Install the MySQL client applications and tools.</li></ul>
|
|
</div></div>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1110__en-us_topic_0000001219029079_t9ab6305d333a4276b5ce82a068c1f11e" frame="border" border="1" rules="all"><caption><b>Table 1 </b><span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_p7402d3287e5a478490ef10251c2c9855"><b>generic-jdbc-connector</b></span> connection parameters</caption><thead align="left"><tr id="mrs_01_1110__en-us_topic_0000001219029079_ra6d5e0652fa54896bf5eca0364dcf534"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.3.3.2.2.2.4.1.1"><p id="mrs_01_1110__en-us_topic_0000001219029079_af7150fd8bece4948a3c78092068eaa72">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="35.709999999999994%" id="mcps1.3.3.3.3.2.2.2.4.1.2"><p id="mrs_01_1110__en-us_topic_0000001219029079_a210e6307e9ee4e909b2a6a40e479e199">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="39.290000000000006%" id="mcps1.3.3.3.3.2.2.2.4.1.3"><p id="mrs_01_1110__en-us_topic_0000001219029079_afdaef81620ff4b909ea07d69946dc142">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1110__en-us_topic_0000001219029079_rad890588b128460bbb30404936fce80c"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_ad04206b5001d4409b025b151755c166e">Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="35.709999999999994%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a1e0fff0a33104d56ab4244436ff9c51f">Specifies the name of a relational database connection.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.290000000000006%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a1b8a0f86c471419da50afc6410074073">dbName</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1110__en-us_topic_0000001219029079_r16a36d664acd426c86503072377d0b06"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a8344a5d3521d4ba28aa8ff69776609aa">JDBC Driver Class</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="35.709999999999994%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a6df3851934e04219977fdb0090531006">Specifies the name of a Java database connectivity (JDBC) driver class.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.290000000000006%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a77d3679c05594eb49d8c26a0f21a5b97">oracle.jdbc.driver.OracleDriver</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1110__en-us_topic_0000001219029079_rb70c6376f95a4ba8b0126a33a58f26a5"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a936e0e5fa21241c695eb278775a6f505">JDBC Connection String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="35.709999999999994%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a21e9c9d02fbe48e489da361e5eb31c4a">Specifies the JDBC connection string.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.290000000000006%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_aae7bd37318c44f7e88a5c153298bb2d9">jdbc:oracle:thin:@//10.16.0.1:1521/oradb</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1110__en-us_topic_0000001219029079_r3ce80b641dc546f48734224b317cb067"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_aa42e102239f447d7ada16e5962b75a9c">Username</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="35.709999999999994%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a046f4a06905a4431a5a748b090e61da6">Specifies the username for connecting to the database.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.290000000000006%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a7e7ffbf9cc3349c89eed63011e1c9337">omm</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1110__en-us_topic_0000001219029079_r4fa84ec2d7fa46bf8e8e5a04d6fc0df6"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a93e00aa384e14ead91b71cc8d7b438e1">Password</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="35.709999999999994%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a93121f57329c4644a7dbb7831e4ee70e">Specifies the password for connecting to the database.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.290000000000006%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a1788872fa2a948f0a46fdd105bca29c9">xxxx</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1110__en-us_topic_0000001219029079_re047032e42f7482bb295ae21049710b1"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.3.2.2.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a4b84108cd35141f3ba656d5120ae7959">JDBC Connection Properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="35.709999999999994%" headers="mcps1.3.3.3.3.2.2.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_afbe1ecdfc0e54a7a9f2d83b0df4bc711">JDBC connection attribute. Click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_u4a7dc48ca0d54290a1998d9c3f8586f8"><b>Add</b></span> to manually add the attribute.</p>
|
|
<ul id="mrs_01_1110__en-us_topic_0000001219029079_u926a7f485a3249ccb8670673dbd2aabc"><li id="mrs_01_1110__en-us_topic_0000001219029079_lf79652f7fd9146d49f051a418d2a64bc">Name: connection attribute name</li><li id="mrs_01_1110__en-us_topic_0000001219029079_lbbd172b9db644a1ca2fa2ada75204c26">Value: connection attribute value</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.290000000000006%" headers="mcps1.3.3.3.3.2.2.2.4.1.3 "><ul id="mrs_01_1110__en-us_topic_0000001219029079_u0f823ce2c602477aa49b48304a8c6c48"><li id="mrs_01_1110__en-us_topic_0000001219029079_l1a6df66314f240eda1d624b9f8b18385">Name: socketTimeout</li><li id="mrs_01_1110__en-us_topic_0000001219029079_l06390182ea50485ea7b25fd676ec9092">Value: 20</li></ul>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p class="tableheading" id="mrs_01_1110__en-us_topic_0000001219029079_aed13ba3ce35a4360a2213ad1ba23eaa7"><strong id="mrs_01_1110__en-us_topic_0000001219029079_ab92c8104e8314a71bb245ad72cf19995">Setting Data Source Information</strong></p>
|
|
</p></li><li id="mrs_01_1110__en-us_topic_0000001219029079_l7037af128e294f7089a5399dd23871bb"><span>Click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_uicontrol14988956979160"><b>Next</b></span>. On the displayed <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_parmname6224067259160"><b>From</b></span> page, set <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_parmname10760855589160"><b>Source type</b></span> to <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue3628198859160"><b>SPARK</b></span>.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1110__en-us_topic_0000001219029079_tbab765954fb64e4084fdc2210549b68c" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Data source parameters</caption><thead align="left"><tr id="mrs_01_1110__en-us_topic_0000001219029079_r9f37bd5058d64846a1f710a9858381fd"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.3.4.2.1.2.4.1.1"><p id="mrs_01_1110__en-us_topic_0000001219029079_ac03d104f498d4ab49b3ce5d6614a9409">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="67.47999999999999%" id="mcps1.3.3.3.4.2.1.2.4.1.2"><p id="mrs_01_1110__en-us_topic_0000001219029079_a7abfae96a677404c93e48ba660859a36">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="7.5200000000000005%" id="mcps1.3.3.3.4.2.1.2.4.1.3"><p id="mrs_01_1110__en-us_topic_0000001219029079_addc571fd95df4f65934c52bbd6b60de1">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1110__en-us_topic_0000001219029079_r655eda05bb91420f9a3377fa250edba2"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a77413424fd0f4aacb4bdc19fbc040211">Spark instance</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="67.47999999999999%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a5dc0e9a3ea4c4b50b43e9107163d6f67">Specifies the Spark service instance that Loader selects from all available Spark service instances in the cluster. If the selected Spark service instance is not added to the cluster, the Spark job cannot be run properly.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a26071246d9464ff7af37aee216e6f7ec">spark</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1110__en-us_topic_0000001219029079_rae858e4732f44228a7b3c8060d1aabad"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.4.2.1.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a7a7d371dfe8848938f2871c6b319e68b">Quantity</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="67.47999999999999%" headers="mcps1.3.3.3.4.2.1.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a32ee2efc1f0e49f4aa9a5afb86cf3f1f">Specifies the number of maps that are started at the same time in a MapReduce job of a data configuration operation. The value must be less than or equal to 3000.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.3.4.2.1.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a20da13c51def493a9ff7d01f017dfa43">20</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p class="tableheading" id="mrs_01_1110__en-us_topic_0000001219029079_a3423205b4d1146378cb458116a50684f"><strong id="mrs_01_1110__en-us_topic_0000001219029079_a47741818c11847c8b1d050da7e76acfc">Setting Data Transformation</strong></p>
|
|
</p></li><li id="mrs_01_1110__en-us_topic_0000001219029079_l53fa12a1db0c4e0da442124aac6085e3"><span>Click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_u1f11f9a2b07b46dd94d0b7cf6760b6f3"><b>Next</b></span>. On the displayed <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_peaf819a21a8e4c4caf1a11738830ac75"><b>Transform</b></span> page, set the transformation operations in the data transformation process. For details about how to select operators and set parameters, see <a href="mrs_01_1119.html">Operator Help</a> and <a href="#mrs_01_1110__en-us_topic_0000001219029079_table895989011525">Table 3</a>.</span><p>
|
|
<div class="tablenoborder"><a name="mrs_01_1110__en-us_topic_0000001219029079_table895989011525"></a><a name="en-us_topic_0000001219029079_table895989011525"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1110__en-us_topic_0000001219029079_table895989011525" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Setting the input and output parameters of the operator</caption><thead align="left"><tr id="mrs_01_1110__en-us_topic_0000001219029079_row1060779011525"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.3.5.2.1.2.3.1.1"><p id="mrs_01_1110__en-us_topic_0000001219029079_p1556013211525">Input Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.3.5.2.1.2.3.1.2"><p id="mrs_01_1110__en-us_topic_0000001219029079_p5241116811525">Export Type</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1110__en-us_topic_0000001219029079_row6122078811525"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_p5993680211525">Spark Input</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.3.5.2.1.2.3.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_p58185532115718">Table output</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="fignone" id="mrs_01_1110__en-us_topic_0000001219029079_fa24b7c24e27d446191655b7cc0ed631e"><span class="figcap"><b>Figure 3 </b>Operator operation procedure</span><br><span><img id="mrs_01_1110__en-us_topic_0000001219029079_i2b293c97404b4abab67de10088dbf497" src="en-us_image_0000001295740096.png"></span></div>
|
|
<p class="tableheading" id="mrs_01_1110__en-us_topic_0000001219029079_a81faa462e4204f33b405854809584bdc"><strong id="mrs_01_1110__en-us_topic_0000001219029079_a307b58ef0e394b4c88c89796d24ee8f2">Setting Data Storage Information and Executing the Job</strong></p>
|
|
</p></li><li id="mrs_01_1110__en-us_topic_0000001219029079_l37419e8f66cd4823bd9d3f312cae35e6"><span>Click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_u7a14f01ecde74d4caff63697530082d0"><b>Next</b></span>. On the displayed <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_p409c80831d19441ebc2fc765ca7d83d4"><b>To</b></span> page, set the data storage mode.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1110__en-us_topic_0000001219029079_t27186482003c432db164a35ec4cda4d7" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1110__en-us_topic_0000001219029079_rd8ffe391ca1b42dba1edb544a8f81fb1"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.3.6.2.1.2.4.1.1"><p id="mrs_01_1110__en-us_topic_0000001219029079_ad14e2df2085a4c32b25681dfd4fca8c9">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="67.47999999999999%" id="mcps1.3.3.3.6.2.1.2.4.1.2"><p id="mrs_01_1110__en-us_topic_0000001219029079_a8ddbfbd4090e4caf93bcd2dce2b6d9d0">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="7.5200000000000005%" id="mcps1.3.3.3.6.2.1.2.4.1.3"><p id="mrs_01_1110__en-us_topic_0000001219029079_a124b9c6e48124a82a2ab6b52fa487603">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1110__en-us_topic_0000001219029079_r3a97faa208bd4a899290250ed7ba5437"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a34c7532208db472c8bf61cd43c1059eb">Schema name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="67.47999999999999%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_ad3684280f44e4fa7a3c41bce6e613d1c">Specifies the database schema name.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_ae9a0d793d9844ac99447261efcd6247b">dbo</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1110__en-us_topic_0000001219029079_r8c0cf171460b4774adbc134368f675cd"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a643168fa093546b19efb7c6b0acdee26">Table name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="67.47999999999999%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_aa5506df8e5e646908f0f114441b7b559">Specifies the name of a database table that is used to save the final data of the transmission.</p>
|
|
<div class="note" id="mrs_01_1110__en-us_topic_0000001219029079_ne2fbede4d8b54780ab6df964a4eae99e"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1110__en-us_topic_0000001219029079_acafe30c2643b4e0aa4eb533569f5074c">Table names can be defined using macros. For details, see <a href="mrs_01_1153.html">Using Macro Definitions in Configuration Items</a>.</p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a82943343e1c846838e313f5bc03c2a8b">test</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1110__en-us_topic_0000001219029079_r51c6ae48e06d4f88a48b319b94c440d5"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a257f7edb2b0041c78f88ce44ecd67872">Temporary table</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="67.47999999999999%" headers="mcps1.3.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_1110__en-us_topic_0000001219029079_a2128ab1ed432420bb4660681a6a649a9">Specifies the name of a temporary database table that is used to save temporary data during the transmission. The fields in the table must be the same as those in the database specified by <strong id="mrs_01_1110__en-us_topic_0000001219029079_b12533276779160">Table name</strong>.</p>
|
|
<div class="note" id="mrs_01_1110__en-us_topic_0000001219029079_note21268470212043"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1110__en-us_topic_0000001219029079_p57198506212043">A temporary table is used to prevent dirty data from being generated in the destination table when data is exported to the database. Data is migrated from the temporary table to the destination table only after all data is successfully written to the temporary table. Using temporary tables increases the job execution time.</p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_1110__en-us_topic_0000001219029079_aba6d8efa9b9642219ff1e19c31d82dbe">tmp_test</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_1110__en-us_topic_0000001219029079_l243bee1d117540608e22ece7009bdfd7"><span>Click <span class="uicontrol" id="mrs_01_1110__en-us_topic_0000001219029079_u617d84d888f9459faafeff08044970ce"><b>Save and run</b></span> to save and run the job.</span><p><p id="mrs_01_1110__en-us_topic_0000001219029079_a8a6991361106442293802ca3e3a54cf2"><strong id="mrs_01_1110__en-us_topic_0000001219029079_af86be5fad1de41088e01afa47c633249">Checking the Job Execution Result</strong></p>
|
|
</p></li><li id="mrs_01_1110__en-us_topic_0000001219029079_le4a150033e924e9895582f7b9b867f44"><span>Go to the <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_parmname9629505249160"><b>Loader WebUI</b></span>. When <span class="parmname" id="mrs_01_1110__en-us_topic_0000001219029079_parmname3301877639160"><b>Status</b></span> is <span class="parmvalue" id="mrs_01_1110__en-us_topic_0000001219029079_parmvalue20839897089160"><b>Succeeded</b></span>, the job is complete.</span><p><div class="fignone" id="mrs_01_1110__en-us_topic_0000001173630728_f6c7d800e4d2a45da9f9672d8d2b33ad1"><span class="figcap"><b>Figure 4 </b>Viewing job</span><br><span><img id="mrs_01_1110__en-us_topic_0000001173630728_ia5e94abc31a3428aaaca819fd1f345ba" src="en-us_image_0000001389147806.png"></span></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1100.html">Exporting Data</a></div>
|
|
</div>
|
|
</div>
|
|
|