vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_24179.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

188 lines
28 KiB
HTML
Raw Blame History

<a name="mrs_01_24179"></a><a name="mrs_01_24179"></a>
<h1 class="topictitle1">Interconnecting FlinkServer with Hive</h1>
<div id="body32001227"><div class="section" id="mrs_01_24179__en-us_topic_0000001173470662_section1772881163015"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24179__en-us_topic_0000001173470662_p1938161573310">Currently, FlinkServer interconnects with Hive MetaStore. Therefore, the MetaStore function must be enabled for Hive. Hive can be used as source, sink, and dimension tables.</p>
<p id="mrs_01_24179__en-us_topic_0000001173470662_p9824226141220">If your Kafka cluster is in security mode, the following example SQL statements can be used.</p>
</div>
<div class="section" id="mrs_01_24179__en-us_topic_0000001173470662_section16138104911329"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24179__en-us_topic_0000001173470662_ul1342146143311"><li id="mrs_01_24179__en-us_topic_0000001173470662_li14256183315">Services such as HDFS, Yarn, Kafka, Flink, and Hive have been installed in the cluster.</li><li id="mrs_01_24179__en-us_topic_0000001173470662_li84396173318">The client that contains the Hive service has been installed, for example, in the <strong id="mrs_01_24179__en-us_topic_0000001173470662_b3928732145315">/opt/Bigdata/client</strong> directory.</li><li id="mrs_01_24179__en-us_topic_0000001173470662_li154317610334">Flink 1.12.2 or later and Hive 3.1.0 or later are supported.</li><li id="mrs_01_24179__en-us_topic_0000001173470662_li1833255113117">You have created a user with <strong id="mrs_01_24179__en-us_topic_0000001173470662_b11566105117549">FlinkServer Admin Privilege</strong>, for example, <strong id="mrs_01_24179__en-us_topic_0000001173470662_b18122981554">flink_admin</strong>, to access the Flink web UI. For details, see <a href="mrs_01_24049.html">Authentication Based on Users and Roles</a>.</li><li id="mrs_01_24179__en-us_topic_0000001173470662_li1277033894114">You have obtained the client configuration file and credential of the user for accessing the Flink web UI. For details, see "Note" in <a href="mrs_01_24021.html#mrs_01_24021__en-us_topic_0000001173789648_section878113401693">Creating a Cluster Connection</a>.</li></ul>
</div>
<div class="section" id="mrs_01_24179__en-us_topic_0000001173470662_section5520114933317"><h4 class="sectiontitle">Procedure</h4><p id="mrs_01_24179__en-us_topic_0000001173470662_p156481254133311">The following uses the process of interconnecting a Kafka mapping table to Hive as an example.</p>
<ol id="mrs_01_24179__en-us_topic_0000001173470662_ol171559143412"><li id="mrs_01_24179__en-us_topic_0000001173470662_li17782181511106"><span>Log in to the Flink web UI as user <strong id="mrs_01_24179__en-us_topic_0000001173470662_b16284164925814">flink_admin</strong>. For details, see <a href="mrs_01_24019.html">Accessing the Flink Web UI</a>.</span></li><li id="mrs_01_24179__en-us_topic_0000001173470662_li159021532193916"><a name="mrs_01_24179__en-us_topic_0000001173470662_li159021532193916"></a><a name="en-us_topic_0000001173470662_li159021532193916"></a><span>Create a cluster connection, for example, <strong id="mrs_01_24179__en-us_topic_0000001173470662_b996210518598">flink_hive</strong>.</span><p><ol type="a" id="mrs_01_24179__en-us_topic_0000001173470662_ol1122553494717"><li id="mrs_01_24179__en-us_topic_0000001173470662_li62253345479">Choose <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1991958461114459">System Management</strong> &gt; <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1974848900114459">Cluster Connection Management</strong>. The <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1786239188114459">Cluster Connection Management</strong> page is displayed.</li><li id="mrs_01_24179__en-us_topic_0000001173470662_li3225134104717">Click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b375317310016">Create Cluster Connection</strong>. On the displayed page, enter information by referring to <a href="#mrs_01_24179__en-us_topic_0000001173470662_table134890201518">Table 1</a> and click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b2368856104">Test</strong>. After the test is successful, click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b71791757115">OK</strong>.
<div class="tablenoborder"><a name="mrs_01_24179__en-us_topic_0000001173470662_table134890201518"></a><a name="en-us_topic_0000001173470662_table134890201518"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24179__en-us_topic_0000001173470662_table134890201518" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters for creating a cluster connection</caption><thead align="left"><tr id="mrs_01_24179__en-us_topic_0000001173470662_row1491004152"><th align="left" class="cellrowborder" valign="top" width="16.970000000000002%" id="mcps1.3.3.3.2.2.1.2.5.2.4.1.1"><p id="mrs_01_24179__en-us_topic_0000001173470662_p9491904151">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="65.77000000000001%" id="mcps1.3.3.3.2.2.1.2.5.2.4.1.2"><p id="mrs_01_24179__en-us_topic_0000001173470662_p14917016153">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="17.26%" id="mcps1.3.3.3.2.2.1.2.5.2.4.1.3"><p id="mrs_01_24179__en-us_topic_0000001173470662_p10615181912355">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24179__en-us_topic_0000001173470662_row8491603157"><td class="cellrowborder" valign="top" width="16.970000000000002%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p2049120111519">Cluster Connection Name</p>
</td>
<td class="cellrowborder" valign="top" width="65.77000000000001%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p84916071517">Name of the cluster connection, which can contain a maximum of 100 characters. Only letters, digits, and underscores (_) are allowed.</p>
</td>
<td class="cellrowborder" valign="top" width="17.26%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p161510196353">flink_hive</p>
</td>
</tr>
<tr id="mrs_01_24179__en-us_topic_0000001173470662_row1749307153"><td class="cellrowborder" valign="top" width="16.970000000000002%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p174980131510">Description</p>
</td>
<td class="cellrowborder" valign="top" width="65.77000000000001%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p17491408153">Description of the cluster connection name.</p>
</td>
<td class="cellrowborder" valign="top" width="17.26%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p1261591943513">-</p>
</td>
</tr>
<tr id="mrs_01_24179__en-us_topic_0000001173470662_row174950131510"><td class="cellrowborder" valign="top" width="16.970000000000002%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p13494015152">Version</p>
</td>
<td class="cellrowborder" valign="top" width="65.77000000000001%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p45019018154">Select a cluster version.</p>
</td>
<td class="cellrowborder" valign="top" width="17.26%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p8615151914358">MRS 3</p>
</td>
</tr>
<tr id="mrs_01_24179__en-us_topic_0000001173470662_row250402157"><td class="cellrowborder" valign="top" width="16.970000000000002%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p6500051518">Secure Version</p>
</td>
<td class="cellrowborder" valign="top" width="65.77000000000001%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.2 "><ul id="mrs_01_24179__en-us_topic_0000001173470662_ul109491810171811"><li id="mrs_01_24179__en-us_topic_0000001173470662_li14949181071814">If the secure version is used, select <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1340327959114459">Yes</strong> for a security cluster. Enter the username and upload the user credential.</li><li id="mrs_01_24179__en-us_topic_0000001173470662_li1275181715181">If not, select <strong id="mrs_01_24179__en-us_topic_0000001173470662_b2057234593114459">No</strong>.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="17.26%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p12615619143512">Yes</p>
</td>
</tr>
<tr id="mrs_01_24179__en-us_topic_0000001173470662_row18501010153"><td class="cellrowborder" valign="top" width="16.970000000000002%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p6501105154">Username</p>
</td>
<td class="cellrowborder" valign="top" width="65.77000000000001%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p56819185172">The user must have the minimum permissions for accessing services in the cluster. The name can contain a maximum of 100 characters. Only letters, digits, and underscores (_) are allowed.</p>
<p id="mrs_01_24179__en-us_topic_0000001173470662_p1550100171510">This parameter is available only when <strong id="mrs_01_24179__en-us_topic_0000001173470662_b565695367114459">Secure Version</strong> is set to <strong id="mrs_01_24179__en-us_topic_0000001173470662_b647596986114459">Yes</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="17.26%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p3615191903513">flink_admin</p>
</td>
</tr>
<tr id="mrs_01_24179__en-us_topic_0000001173470662_row1350108152"><td class="cellrowborder" valign="top" width="16.970000000000002%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p7508071519">Client Profile</p>
</td>
<td class="cellrowborder" valign="top" width="65.77000000000001%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p14506011519">Client profile of the cluster, in TAR format.</p>
</td>
<td class="cellrowborder" valign="top" width="17.26%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p206154198354">-</p>
</td>
</tr>
<tr id="mrs_01_24179__en-us_topic_0000001173470662_row1450507157"><td class="cellrowborder" valign="top" width="16.970000000000002%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p55014071514">User Credential</p>
</td>
<td class="cellrowborder" valign="top" width="65.77000000000001%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p1599302613181">User authentication credential in FusionInsight Manager in TAR format.</p>
<p id="mrs_01_24179__en-us_topic_0000001173470662_p146511232111719">This parameter is available only when <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1954467687114459">Secure Version</strong> is set to <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1357925150114459">Yes</strong>.</p>
<p id="mrs_01_24179__en-us_topic_0000001173470662_p572515214116">Files can be uploaded only after the username is entered.</p>
</td>
<td class="cellrowborder" valign="top" width="17.26%" headers="mcps1.3.3.3.2.2.1.2.5.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p10615121933516">User credential of <strong id="mrs_01_24179__en-us_topic_0000001173470662_b5686107235">flink_admin</strong></p>
</td>
</tr>
</tbody>
</table>
</div>
</li></ol>
</p></li><li id="mrs_01_24179__en-us_topic_0000001173470662_li98264711467"><span>Create a Flink SQL job, for example, <strong id="mrs_01_24179__en-us_topic_0000001173470662_b134914271136">flinktest1</strong>.</span><p><ol type="a" id="mrs_01_24179__en-us_topic_0000001173470662_ol1796955124618"><li id="mrs_01_24179__en-us_topic_0000001173470662_li189791158174619">Click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1609891617114459">Job Management</strong>. The job management page is displayed.</li><li id="mrs_01_24179__en-us_topic_0000001173470662_li107961755124619">Click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b2050537483114459">Create Job</strong>. On the displayed job creation page, set parameters by referring to <a href="#mrs_01_24179__en-us_topic_0000001173470662_table25451917135812">Table 2</a> and click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1259445257114459">OK</strong>. The job development page is displayed.
<div class="tablenoborder"><a name="mrs_01_24179__en-us_topic_0000001173470662_table25451917135812"></a><a name="en-us_topic_0000001173470662_table25451917135812"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24179__en-us_topic_0000001173470662_table25451917135812" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameters for creating a job</caption><thead align="left"><tr id="mrs_01_24179__en-us_topic_0000001173470662_row15452172582"><th align="left" class="cellrowborder" valign="top" width="18.34%" id="mcps1.3.3.3.3.2.1.2.4.2.4.1.1"><p id="mrs_01_24179__en-us_topic_0000001173470662_p1654518179581">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="61.83%" id="mcps1.3.3.3.3.2.1.2.4.2.4.1.2"><p id="mrs_01_24179__en-us_topic_0000001173470662_p1854616178589">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="19.830000000000002%" id="mcps1.3.3.3.3.2.1.2.4.2.4.1.3"><p id="mrs_01_24179__en-us_topic_0000001173470662_p176084864417">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24179__en-us_topic_0000001173470662_row05461617175810"><td class="cellrowborder" valign="top" width="18.34%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p10546191715819">Type</p>
</td>
<td class="cellrowborder" valign="top" width="61.83%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p19546121765818">Job type, which can be <strong id="mrs_01_24179__en-us_topic_0000001173470662_b999571009114459">Flink SQL</strong> or <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1787732096114459">Flink Jar</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="19.830000000000002%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p207601848194416">Flink SQL</p>
</td>
</tr>
<tr id="mrs_01_24179__en-us_topic_0000001173470662_row18546117135812"><td class="cellrowborder" valign="top" width="18.34%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p1354661719588">Name</p>
</td>
<td class="cellrowborder" valign="top" width="61.83%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p135461017125815">Job name, which can contain a maximum of 64 characters. Only letters, digits, and underscores (_) are allowed.</p>
</td>
<td class="cellrowborder" valign="top" width="19.830000000000002%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p57601748154415">flinktest1</p>
</td>
</tr>
<tr id="mrs_01_24179__en-us_topic_0000001173470662_row0546111775820"><td class="cellrowborder" valign="top" width="18.34%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p20546161735819">Task Type</p>
</td>
<td class="cellrowborder" valign="top" width="61.83%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p55461817175811">Type of the job data source, which can be a stream job or a batch job.</p>
</td>
<td class="cellrowborder" valign="top" width="19.830000000000002%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p17760144814447">Stream job</p>
</td>
</tr>
<tr id="mrs_01_24179__en-us_topic_0000001173470662_row9546171710585"><td class="cellrowborder" valign="top" width="18.34%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.1 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p654615172587">Description</p>
</td>
<td class="cellrowborder" valign="top" width="61.83%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.2 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p654681775813">Job description, which can contain a maximum of 100 characters.</p>
</td>
<td class="cellrowborder" valign="top" width="19.830000000000002%" headers="mcps1.3.3.3.3.2.1.2.4.2.4.1.3 "><p id="mrs_01_24179__en-us_topic_0000001173470662_p676011487449">-</p>
</td>
</tr>
</tbody>
</table>
</div>
</li></ol>
</p></li><li id="mrs_01_24179__en-us_topic_0000001173470662_li1134538357"><span>On the job development page, enter the following statements and click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b035516411665">Check Semantic</strong> to check the input content.</span><p><pre class="screen" id="mrs_01_24179__en-us_topic_0000001173470662_screen328123795411">CREATE TABLE test_kafka (
user_id varchar,
item_id varchar,
cat_id varchar,
zw_test timestamp
) WITH (
'properties.bootstrap.servers' = '<em id="mrs_01_24179__en-us_topic_0000001173470662_i193403844914">IP address of the Kafka broker instance</em>:<em id="mrs_01_24179__en-us_topic_0000001173470662_i12941838144910">Kafka port number</em>',
'format' = 'json',
'topic' = 'zw_tset_kafka',
'connector' = 'kafka',
'scan.startup.mode' = 'latest-offset',
'properties.sasl.kerberos.service.name' = 'kafka',
'properties.security.protocol' = 'SASL_PLAINTEXT',
'properties.kerberos.domain.name' = 'hadoop.<em id="mrs_01_24179__i1620672124713">System domain name</em>'
);
CREATE CATALOG myhive WITH (
'type' = 'hive',
'hive-version' = '<em id="mrs_01_24179__en-us_topic_0000001173470662_i29751339183712">3.1.0</em>',
'default-database' = '<em id="mrs_01_24179__en-us_topic_0000001173470662_i25601154153718">default</em>',
'cluster.name' = '<em id="mrs_01_24179__en-us_topic_0000001173470662_i4130114595720">flink_hive</em>'
);
use catalog myhive;
set table.sql-dialect = hive;create table user_behavior_hive_tbl_no_partition (
user_id STRING,
item_id STRING,
cat_id STRING,
ts timestamp
) PARTITIONED BY (dy STRING, ho STRING, mi STRING) stored as textfile TBLPROPERTIES (
'partition.time-extractor.timestamp-pattern' = '$dy $ho:$mi:00',
'sink.partition-commit.trigger' = 'process-time',
'sink.partition-commit.delay' = '0S',
'sink.partition-commit.policy.kind' = 'metastore,success-file'
);
INSERT into
user_behavior_hive_tbl_no_partition
SELECT
user_id,
item_id,
cat_id,
zw_test,
DATE_FORMAT(zw_test, 'yyyy-MM-dd'),
DATE_FORMAT(zw_test, 'HH'),
DATE_FORMAT(zw_test, 'mm')
FROM
default_catalog.default_database.test_kafka;</pre>
<div class="note" id="mrs_01_24179__en-us_topic_0000001173470662_note1417125718117"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24179__en-us_topic_0000001173470662_ul171684576315"><li id="mrs_01_24179__en-us_topic_0000001173470662_li016875715316">Kafka port number<ul id="mrs_01_24179__en-us_topic_0000001299465345_ul330585217498"><li id="mrs_01_24179__en-us_topic_0000001299465345_li15431748164911">In security mode, the port number is the value of <strong id="mrs_01_24179__en-us_topic_0000001299465345_b205881224151610">sasl.port</strong> (<strong id="mrs_01_24179__en-us_topic_0000001299465345_b9304934101615">21007</strong> by default).</li><li id="mrs_01_24179__en-us_topic_0000001299465345_li175451448174913">In non-security mode, the port is the value of <strong id="mrs_01_24179__en-us_topic_0000001299465345_b149745622017">port</strong> (<strong id="mrs_01_24179__en-us_topic_0000001299465345_b1099613013203">9092</strong> by default). If the port number is set to <strong id="mrs_01_24179__en-us_topic_0000001299465345_b73621643112310">9092</strong>, set <strong id="mrs_01_24179__en-us_topic_0000001299465345_b340128102113">allow.everyone.if.no.acl.found</strong> to <strong id="mrs_01_24179__en-us_topic_0000001299465345_b841628162113">true</strong>. The procedure is as follows:<p id="mrs_01_24179__en-us_topic_0000001299465345_en-us_topic_0000001173630650_p36273255618">Log in to FusionInsight Manager and choose <strong id="mrs_01_24179__en-us_topic_0000001299465345_en-us_topic_0000001173630650_b14557164243019">Cluster</strong> &gt; <strong id="mrs_01_24179__en-us_topic_0000001299465345_en-us_topic_0000001173630650_b2122445113010">Services</strong> &gt; <strong id="mrs_01_24179__en-us_topic_0000001299465345_en-us_topic_0000001173630650_b1936517491306">Kafka</strong>. On the displayed page, click <strong id="mrs_01_24179__en-us_topic_0000001299465345_en-us_topic_0000001173630650_b1266911423116">Configurations</strong> and then <strong id="mrs_01_24179__en-us_topic_0000001299465345_en-us_topic_0000001173630650_b13479112213314">All Configurations</strong>, search for <strong id="mrs_01_24179__en-us_topic_0000001299465345_en-us_topic_0000001173630650_b12988123313112">allow.everyone.if.no.acl.found</strong>, set its value to <strong id="mrs_01_24179__en-us_topic_0000001299465345_en-us_topic_0000001173630650_b12582105613113">true</strong>, and click <strong id="mrs_01_24179__en-us_topic_0000001299465345_en-us_topic_0000001173630650_b4318185014366">Save</strong>.</p>
</li></ul>
</li><li id="mrs_01_24179__en-us_topic_0000001173470662_li1616816574316">The value of <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1292135885">'cluster.name'</strong> is the name of the cluster connection created in <a href="#mrs_01_24179__en-us_topic_0000001173470662_li159021532193916">2</a>.</li></ul>
</div></div>
</p></li><li id="mrs_01_24179__en-us_topic_0000001173470662_li932684112593"><span>After the job is developed, select <strong id="mrs_01_24179__en-us_topic_0000001173470662_b77956391090">Enable CheckPoint</strong> in <strong id="mrs_01_24179__en-us_topic_0000001173470662_b77648449912">Running Parameter</strong> and set <strong id="mrs_01_24179__en-us_topic_0000001173470662_b18619194615118">Time Interval (ms)</strong> to <strong id="mrs_01_24179__en-us_topic_0000001173470662_b9148183414109">60000</strong>.</span></li><li id="mrs_01_24179__en-us_topic_0000001173470662_li715510313513"><span>Click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1205092953114459">Submit</strong> in the upper left corner to submit the job.</span></li><li id="mrs_01_24179__en-us_topic_0000001173470662_li11575949359"><span>After the job is successfully executed, choose <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1320245231015">More</strong> &gt; <strong id="mrs_01_24179__en-us_topic_0000001173470662_b99237116114">Job Monitoring</strong> to view the job running details.</span></li><li id="mrs_01_24179__en-us_topic_0000001173470662_li20986165822912"><span>Execute the following commands to view the topic and write data to Kafka. For details, see <a href="mrs_01_0379.html">Managing Messages in Kafka Topics</a>.</span><p><p id="mrs_01_24179__en-us_topic_0000001173470662_p098645832911"><strong id="mrs_01_24179__en-us_topic_0000001173470662_b10999225195212">./kafka-topics.sh --list --zookeeper </strong><em id="mrs_01_24179__en-us_topic_0000001173470662_i50142665212">IP address of the ZooKeeper quorumpeer instance</em>:<em id="mrs_01_24179__en-us_topic_0000001173470662_i17111260526">ZooKeeper port number</em><strong id="mrs_01_24179__en-us_topic_0000001173470662_b9111263527">/kafka</strong></p>
<p id="mrs_01_24179__en-us_topic_0000001173470662_p11986758102916"><strong id="mrs_01_24179__en-us_topic_0000001173470662_b1033710910501">sh kafka-console-producer.sh --broker-list</strong> <em id="mrs_01_24179__en-us_topic_0000001173470662_i153449955011">IP address of the node where the Kafka instance locates</em><em id="mrs_01_24179__en-us_topic_0000001173470662_i13487935014">:Kafka port number</em><strong id="mrs_01_24179__en-us_topic_0000001173470662_b5350990504"> --topic</strong> <em id="mrs_01_24179__en-us_topic_0000001173470662_i23511392509">Topic name</em> --<strong id="mrs_01_24179__en-us_topic_0000001173470662_b93521895503">producer.config</strong> <em id="mrs_01_24179__en-us_topic_0000001173470662_i1635318910501">Client directory</em>/<strong id="mrs_01_24179__en-us_topic_0000001173470662_b18354189185016">Kafka/kafka/config/producer.properties</strong></p>
<p id="mrs_01_24179__en-us_topic_0000001173470662_p19986258192916">For example, if the topic name is <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1432243914536">zw_tset_kafka</strong>, the script is <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1110721235014">sh kafka-console-producer.sh --broker-list</strong> <em id="mrs_01_24179__en-us_topic_0000001173470662_i10114181255013">IP address of the node where the Kafka instance locates</em><em id="mrs_01_24179__en-us_topic_0000001173470662_i1511861215503">:Kafka port number</em> <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1211961218506">--topic zw_tset_kafka</strong> --<strong id="mrs_01_24179__en-us_topic_0000001173470662_b71201122506">producer.config </strong><strong id="mrs_01_24179__en-us_topic_0000001173470662_b112141215501">/opt/</strong><strong id="mrs_01_24179__en-us_topic_0000001173470662_b151241112165019">Bigdata/</strong><strong id="mrs_01_24179__en-us_topic_0000001173470662_b1212561285020">client</strong><strong id="mrs_01_24179__en-us_topic_0000001173470662_b20126121215011">/</strong><strong id="mrs_01_24179__en-us_topic_0000001173470662_b19128141235020">Kafka/kafka/config/producer.properties</strong>.</p>
<div class="p" id="mrs_01_24179__en-us_topic_0000001173470662_p1498618582294">Enter the message content.<pre class="screen" id="mrs_01_24179__en-us_topic_0000001173470662_screen19986195822918">{"user_id": "3","item_id":"333333","cat_id":"cat333","zw_test":"2021-09-08 09:08:01"}
{"user_id": "4","item_id":"444444","cat_id":"cat444","zw_test":"2021-09-08 09:08:01"} </pre>
</div>
<p id="mrs_01_24179__en-us_topic_0000001173470662_p1698685862916">Press <strong id="mrs_01_24179__en-us_topic_0000001173470662_b73161545419">Enter</strong> to send the message.</p>
<div class="note" id="mrs_01_24179__en-us_topic_0000001173470662_note113934313390"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24179__en-us_topic_0000001173470662_ul4278192283814"><li id="mrs_01_24179__en-us_topic_0000001173470662_li024511411817">IP address of the ZooKeeper quorumpeer instance<p id="mrs_01_24179__en-us_topic_0000001173470662_p92456140814"><a name="mrs_01_24179__en-us_topic_0000001173470662_li024511411817"></a><a name="en-us_topic_0000001173470662_li024511411817"></a>To obtain IP addresses of all ZooKeeper quorumpeer instances, log in to FusionInsight Manager and choose <strong id="mrs_01_24179__en-us_topic_0000001173470662_b0412162335411">Cluster</strong> &gt; <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1041412233543">Services</strong> &gt; <strong id="mrs_01_24179__en-us_topic_0000001173470662_b0415102365415">ZooKeeper</strong>. On the displayed page, click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b10417122315416">Instance</strong> and view the IP addresses of all the hosts where the quorumpeer instances locate.</p>
</li><li id="mrs_01_24179__en-us_topic_0000001173470662_li624515142816">Port number of the ZooKeeper client<p id="mrs_01_24179__en-us_topic_0000001173470662_p1724512142082"><a name="mrs_01_24179__en-us_topic_0000001173470662_li624515142816"></a><a name="en-us_topic_0000001173470662_li624515142816"></a>Log in to FusionInsight Manager and choose <strong id="mrs_01_24179__en-us_topic_0000001173470662_b10602113012546">Cluster</strong> &gt; <strong id="mrs_01_24179__en-us_topic_0000001173470662_b16604203015418">Service</strong> &gt; <strong id="mrs_01_24179__en-us_topic_0000001173470662_b16606193005410">ZooKeeper</strong>. On the displayed page, click <strong id="mrs_01_24179__en-us_topic_0000001173470662_b186087303541">Configurations</strong> and check the value of <strong id="mrs_01_24179__en-us_topic_0000001173470662_b86101130165411">clientPort</strong>. The default value is <strong id="mrs_01_24179__en-us_topic_0000001173470662_b1624913397545">24002</strong>.</p>
</li></ul>
</div></div>
</p></li><li id="mrs_01_24179__en-us_topic_0000001173470662_li6386217104817"><span>Run the following command to check whether data is written from the Hive table to the sink table:</span><p><p id="mrs_01_24179__en-us_topic_0000001173470662_p38001810184610"><strong id="mrs_01_24179__en-us_topic_0000001173470662_b1322721411464">beeline</strong></p>
<p id="mrs_01_24179__en-us_topic_0000001173470662_p68000107468"><strong id="mrs_01_24179__en-us_topic_0000001173470662_b16231914194615">select * from user_behavior_hive_tbl_no_partition;</strong></p>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24226.html">Interconnecting FlinkServer with External Components</a></div>
</div>
</div>
View Git Blame Copy Permalink