vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_24148.html
Yang, Tong 48706b7552 MRS COMP-LTS 320-lts.1 version 
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2024-04-12 12:51:10 +00:00

260 lines
31 KiB
HTML
Raw Blame History

<a name="mrs_01_24148"></a><a name="mrs_01_24148"></a>
<h1 class="topictitle1">Interconnecting FlinkServer with ClickHouse</h1>
<div id="body32001227"><div class="section" id="mrs_01_24148__en-us_topic_0000001173630650_section124189417421"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24148__en-us_topic_0000001173630650_p8060118">Flink interconnects with the ClickHouseBalancer instance of ClickHouse to read and write data, preventing ClickHouse traffic distribution problems.</p>
<div class="notice" id="mrs_01_24148__note12192112119446"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="mrs_01_24148__p541632524416">When "FlinkSQL" is displayed in the command output on the FlinkServer web UI in MRS 3.2.0 or later clusters, the <strong id="mrs_01_24148__b4400104013395">password</strong> field in the SQL statement is left blank to meet security requirements. Before you submit a job, manually enter the password.</p>
</div></div>
</div>
<div class="section" id="mrs_01_24148__en-us_topic_0000001173630650_section17259141064413"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24148__en-us_topic_0000001173630650_ul184973415531"><li id="mrs_01_24148__en-us_topic_0000001173630650_li349713412536">Services such as ClickHouse, HDFS, Yarn, Flink, and HBase have been installed in the cluster.</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li449716411539">The client has been installed, for example, in <strong id="mrs_01_24148__en-us_topic_0000001173630650_b556185131312">/opt/Bigdata/client</strong>.</li></ul>
</div>
<div class="section" id="mrs_01_24148__en-us_topic_0000001173630650_section667518152549"><h4 class="sectiontitle">Mapping Between Flink SQL and ClickHouse Data Types</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24148__en-us_topic_0000001173630650_table1656913419554" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_24148__en-us_topic_0000001173630650_row1571541175519"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.1"><p id="mrs_01_24148__en-us_topic_0000001173630650_p1257154175515">Flink SQL Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.2"><p id="mrs_01_24148__en-us_topic_0000001173630650_p19571104110559">ClickHouse Data Type</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_24148__en-us_topic_0000001173630650_row11571341155512"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p20571144117553">BOOLEAN</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p1057174155511">UInt8</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row357111412559"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p1457117412557">TINYINT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p5571104111556">Int8</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row15571441185510"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p19571134119552">SMALLINT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p557194155511">Int16</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row1857114415559"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p557115416555">INTEGER</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p9571104119558">Int32</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row115711341145517"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p10571174115514">BIGINT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p45717411555">Int64</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row1857144116554"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p957194111556">FLOAT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p16571541105513">Float32</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row9571134111552"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p20571124125512">DOUBLE</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p1357134155510">Float64</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row14151420105718"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p141529201572">CHAR</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p1615210206572">String</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row8791529195717"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p107928296570">VARCHAR</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p1379222965711">String</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row7792112925717"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p679252910576">VARBINARY</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p67921229185711">FixedString</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row87923295574"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p1679232917575">DATE</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p167928297576">Date</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row13793729155718"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p1793629135714">TIMESTAMP</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p14793182975719">DateTime</p>
</td>
</tr>
<tr id="mrs_01_24148__en-us_topic_0000001173630650_row179352935710"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p19793112910573">DECIMAL</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="mrs_01_24148__en-us_topic_0000001173630650_p15793122915577">Decimal</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="mrs_01_24148__en-us_topic_0000001173630650_section51341943104610"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24148__en-us_topic_0000001173630650_ol11471534103911"><li id="mrs_01_24148__en-us_topic_0000001173630650_li10269200102512"><span>Log in to the node where the client is installed as user <strong id="mrs_01_24148__en-us_topic_0000001173630650_b646251172410">root</strong>.</span></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li4269903251"><span>Run the following command to go to the client installation directory:</span><p><p id="mrs_01_24148__en-us_topic_0000001173630650_p1526916017259"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b426990122515">cd /opt/Bigdata/client</strong></p>
</p></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li172699012517"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_24148__en-us_topic_0000001173630650_p172694017259"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b16269180122520">source bigdata_env</strong></p>
</p></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li1910333418475"><span>If Kerberos authentication is enabled for the current cluster, run the following command to authenticate the current user. The current user must have the permission to create ClickHouse tables. If Kerberos authentication is disabled for the current cluster, skip this step:</span><p><p id="mrs_01_24148__en-us_topic_0000001173630650_p137512579492"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b1520521045010">kinit</strong> <em id="mrs_01_24148__en-us_topic_0000001173630650_i1133611425013">Component service user</em></p>
<p id="mrs_01_24148__en-us_topic_0000001173630650_p77201452184920">Example: <strong id="mrs_01_24148__en-us_topic_0000001173630650_b61553404501">kinit</strong> <strong id="mrs_01_24148__en-us_topic_0000001173630650_b352435194318">clickhouseuser</strong></p>
</p></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li4140746493"><span>Connect to the ClickHouse client. For details, see <a href="mrs_01_2345.html">Using ClickHouse from Scratch</a>.</span><p><ul id="mrs_01_24148__en-us_topic_0000001173630650_ul14750436195310"><li id="mrs_01_24148__en-us_topic_0000001173630650_li10750336135319">Normal mode:<p id="mrs_01_24148__en-us_topic_0000001173630650_p2068015414193"><a name="mrs_01_24148__en-us_topic_0000001173630650_li10750336135319"></a><a name="en-us_topic_0000001173630650_li10750336135319"></a><strong id="mrs_01_24148__en-us_topic_0000001173630650_b203332428164912">clickhouse client --host </strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i34944721164912">IP address of the ClickHouse instance</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b127734468164912"> --user </strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i34907564864912">Username</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b210672216864912"> --password '</strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i122175529364912">Password</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b3489143413250">' </strong><strong id="mrs_01_24148__en-us_topic_0000001173630650_b29285295764912">--port </strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i198993261964912">ClickHouse port number</em></p>
</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li376184016539">Security mode:<p id="mrs_01_24148__en-us_topic_0000001173630650_p1184311347525"><a name="mrs_01_24148__en-us_topic_0000001173630650_li376184016539"></a><a name="en-us_topic_0000001173630650_li376184016539"></a><strong id="mrs_01_24148__en-us_topic_0000001173630650_b123688351528">clickhouse client --host </strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i15368193525219">IP address of the ClickHouse instance</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b1036873518527"> --user </strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i436823511524">Username</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b4160212182615"> --password '</strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i3475312152616">Password</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b816012124263">'</strong><strong id="mrs_01_24148__en-us_topic_0000001173630650_b1436803585218">--port </strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i173681935185220">ClickHouse port number</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b2232155062415"> --secure</strong> <strong id="mrs_01_24148__en-us_topic_0000001173630650_b612794782418">--multiline</strong></p>
</li></ul>
</p></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li1372918141168"><span>Run the following statements to create a replication table and a distributed table.</span><p><ol type="a" id="mrs_01_24148__en-us_topic_0000001173630650_ol1126208205119"><li id="mrs_01_24148__en-us_topic_0000001173630650_li183932265114">Create a replication table <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1820874141618">default.test1</strong>.<pre class="screen" id="mrs_01_24148__screen1272062717553"><strong id="mrs_01_24148__b13742122765512">CREATE TABLE default.test1 on cluster default_cluster</strong>
<strong id="mrs_01_24148__b13743142745519">(</strong>
<strong id="mrs_01_24148__b8743132725511">`pid` Int8,</strong>
<strong id="mrs_01_24148__b3743227185516">`uid` UInt8,</strong>
<strong id="mrs_01_24148__b1374322755516">`Int_16` Int16,</strong>
<strong id="mrs_01_24148__b674310273556">`Int_32` Int32,</strong>
<strong id="mrs_01_24148__b5743127115515">`Int_64` Int64,</strong>
<strong id="mrs_01_24148__b1743192719552">`String_x` String,</strong>
<strong id="mrs_01_24148__b674319270559">`String_y` String,</strong>
<strong id="mrs_01_24148__b20743527205516">`float_32` Float32,</strong>
<strong id="mrs_01_24148__b474352719555">`float_64` Float64,</strong>
<strong id="mrs_01_24148__b174412719559">`Decimal_x` Decimal32(2),</strong>
<strong id="mrs_01_24148__b274442710556">`Date_x` Date,</strong>
<strong id="mrs_01_24148__b20744142745517">`DateTime_x` DateTime</strong>
<strong id="mrs_01_24148__b5744132718554">)</strong>
<strong id="mrs_01_24148__b19744102716556">ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/test1','{replica}')</strong>
<strong id="mrs_01_24148__b17744827185511">PARTITION BY pid</strong>
<strong id="mrs_01_24148__b12744727125513">ORDER BY (pid, DateTime_x);</strong></pre>
</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li172625812519">Create a distributed table <strong id="mrs_01_24148__en-us_topic_0000001173630650_b7514243170">test1_all</strong>.<pre class="screen" id="mrs_01_24148__screen573093613559"><strong id="mrs_01_24148__b7748113665513">CREATE TABLE test1_all ON CLUSTER default_cluster</strong>
<strong id="mrs_01_24148__b127481336185510">(</strong>
<strong id="mrs_01_24148__b274819366555">`pid` Int8,</strong>
<strong id="mrs_01_24148__b147481036165511">`uid` UInt8,</strong>
<strong id="mrs_01_24148__b8748193611559">`Int_16` Int16,</strong>
<strong id="mrs_01_24148__b15748163655513">`Int_32` Int32,</strong>
<strong id="mrs_01_24148__b1674893695519">`Int_64` Int64,</strong>
<strong id="mrs_01_24148__b1748143619555">`String_x` String,</strong>
<strong id="mrs_01_24148__b17748036105519">`String_y` String,</strong>
<strong id="mrs_01_24148__b1974823675510">`float_32` Float32,</strong>
<strong id="mrs_01_24148__b174820363557">`float_64` Float64,</strong>
<strong id="mrs_01_24148__b1474813361559">`Decimal_x` Decimal32(2),</strong>
<strong id="mrs_01_24148__b12748123619556">`Date_x` Date,</strong>
<strong id="mrs_01_24148__b97488364557">`DateTime_x` DateTime</strong>
<strong id="mrs_01_24148__b574843611553">)</strong>
<strong id="mrs_01_24148__b1374843616557">ENGINE = Distributed(default_cluster, default, test1, rand());</strong></pre>
</li></ol>
</p></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li12826131220399"><span>Log in to Manager and choose <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1072314350311">Cluster</strong> &gt; <strong id="mrs_01_24148__en-us_topic_0000001173630650_b157302357319">Services</strong> &gt; <strong id="mrs_01_24148__en-us_topic_0000001173630650_b67312357317">Flink</strong>. In the <strong id="mrs_01_24148__en-us_topic_0000001173630650_b173113553111">Basic Information</strong> area, click the link on the right of <strong id="mrs_01_24148__en-us_topic_0000001173630650_b167320352319">Flink WebUI</strong> to access the Flink web UI.</span></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li2023444020"><span>Create a Flink SQL job and set <strong id="mrs_01_24148__en-us_topic_0000001173630650_b149591913153818">Task Type</strong> to <strong id="mrs_01_24148__en-us_topic_0000001173630650_b917419212380">Stream job</strong>. For details, see <a href="mrs_01_24024.html#mrs_01_24024__en-us_topic_0000001173470782_section1746418521537">Creating a Job</a>. On the job development page, configure the job parameters as follows and start the job. Select <strong id="mrs_01_24148__en-us_topic_0000001173630650_b17687176144110">Enable CheckPoint</strong> in <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1210931414414">Running Parameter</strong> and set <strong id="mrs_01_24148__en-us_topic_0000001173630650_b914819181419">Time Interval (ms)</strong> to <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1794542304113">60000</strong>.</span><p><ul id="mrs_01_24148__en-us_topic_0000001173630650_ul199677462810"><li id="mrs_01_24148__en-us_topic_0000001173630650_li1796784162820">If the current MRS cluster is in security mode, perform the following operations:<pre class="screen" id="mrs_01_24148__en-us_topic_0000001173630650_screen13206102715213">create table kafkasource(
`pid` TINYINT,
`uid` BOOLEAN,
`Int_16` SMALLINT,
`Int_32` INTEGER,
`Int_64` BIGINT,
`String_x` CHAR,
`String_y` VARCHAR(10),
`float_32` FLOAT,
`float_64` DOUBLE,
`Decimal_x` DECIMAL(9,2),
`Date_x` DATE,
`DateTime_x` TIMESTAMP
) with(
'connector' = 'kafka',
'topic' = 'input',
'properties.bootstrap.servers' = '<em id="mrs_01_24148__en-us_topic_0000001173630650_i15291132212592">IP address of the Kafka broker instance</em>:<em id="mrs_01_24148__en-us_topic_0000001173630650_i113261830163415">Kafka port number</em>',
'properties.group.id' = 'group1',
'scan.startup.mode' = 'earliest-offset',
'format' = 'json',
'properties.sasl.kerberos.service.name' = 'kafka',
'properties.security.protocol' = 'SASL_PLAINTEXT',
'properties.kerberos.domain.name' = 'hadoop.<em id="mrs_01_24148__i1620672124713">System domain name</em>'
);
CREATE TABLE cksink (
`pid` TINYINT,
`uid` BOOLEAN,
`Int_16` SMALLINT,
`Int_32` INTEGER,
`Int_64` BIGINT,
`String_x` CHAR,
`String_y` VARCHAR(10),
`float_32` FLOAT,
`float_64` DOUBLE,
`Decimal_x` DECIMAL(9,2),
`Date_x` DATE,
`DateTime_x` TIMESTAMP
) WITH (
'connector' = 'jdbc',
'url' = 'jdbc:clickhouse://<em id="mrs_01_24148__en-us_topic_0000001173630650_i1845092812113">ClickHouseBalancer instance IP address:21422</em>/default?ssl=true&amp;sslmode=none',
'username' = 'ClickHouse user. For details, see the note below.',
'password' = 'ClickHouse user password. For details, see the note below.',
'table-name' = '<em id="mrs_01_24148__en-us_topic_0000001173630650_i188792414413">test1_all</em>',
'driver' = 'ru.yandex.clickhouse.ClickHouseDriver',
'sink.buffer-flush.max-rows' = '0',
'sink.buffer-flush.interval' = '60s'
);
Insert into cksink
select
*
from
kafkasource;</pre>
</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li26221516152817">If the current MRS cluster is in normal mode, perform the following operations:<pre class="screen" id="mrs_01_24148__en-us_topic_0000001173630650_screen6169267013">create table kafkasource(
`pid` TINYINT,
`uid` BOOLEAN,
`Int_16` SMALLINT,
`Int_32` INTEGER,
`Int_64` BIGINT,
`String_x` CHAR,
`String_y` VARCHAR(10),
`float_32` FLOAT,
`float_64` DOUBLE,
`Decimal_x` DECIMAL(9,2),
`Date_x` DATE,
`DateTime_x` TIMESTAMP
) with(
'connector' = 'kafka',
'topic' = 'kinput',
'properties.bootstrap.servers' = '<em id="mrs_01_24148__en-us_topic_0000001173630650_i4352113320110">IP address of the Kafka broker instance</em>:<em id="mrs_01_24148__en-us_topic_0000001173630650_i135317332119">Kafka port number</em>',
'properties.group.id' = 'kafka_test',
'scan.startup.mode' = 'earliest-offset',
'format' = 'json'
);
CREATE TABLE cksink (
`pid` TINYINT,
`uid` BOOLEAN,
`Int_16` SMALLINT,
`Int_32` INTEGER,
`Int_64` BIGINT,
`String_x` CHAR,
`String_y` VARCHAR(10),
`float_32` FLOAT,
`float_64` DOUBLE,
`Decimal_x` DECIMAL(9,2),
`Date_x` DATE,
`DateTime_x` TIMESTAMP
) WITH (
'connector' = 'jdbc',
'url' = 'jdbc:clickhouse://<em id="mrs_01_24148__en-us_topic_0000001173630650_i217152619010">ClickHouseBalancer instance IP address:21425</em>/default',
'table-name' = '<em id="mrs_01_24148__en-us_topic_0000001173630650_i131719266015">test1_all</em>',
'driver' = 'ru.yandex.clickhouse.ClickHouseDriver',
'sink.buffer-flush.max-rows' = '0',
'sink.buffer-flush.interval' = '60s'
);
Insert into cksink
select
*
from
kafkasource;</pre>
</li></ul>
<div class="note" id="mrs_01_24148__en-us_topic_0000001173630650_note14613113320511"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24148__en-us_topic_0000001173630650_ul1861353325113"><li id="mrs_01_24148__en-us_topic_0000001173630650_li961373385115">If an MRS cluster is in the security mode, the user in the <strong id="mrs_01_24148__en-us_topic_0000001173630650_b16771185971810">cksink</strong> table must have related permissions on the ClickHouse tables. For details, see <a href="mrs_01_24057.html">ClickHouse User and Permission Management</a>.</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li20295195815519">Kafka port number<ul id="mrs_01_24148__ul330585217498"><li id="mrs_01_24148__li15431748164911">In security mode, the port number is the value of <strong id="mrs_01_24148__b205881224151610">sasl.port</strong> (<strong id="mrs_01_24148__b9304934101615">21007</strong> by default).</li><li id="mrs_01_24148__li175451448174913">In non-security mode, the port is the value of <strong id="mrs_01_24148__b149745622017">port</strong> (<strong id="mrs_01_24148__b1099613013203">9092</strong> by default). If the port number is set to <strong id="mrs_01_24148__b73621643112310">9092</strong>, set <strong id="mrs_01_24148__b340128102113">allow.everyone.if.no.acl.found</strong> to <strong id="mrs_01_24148__b841628162113">true</strong>. The procedure is as follows:<p id="mrs_01_24148__en-us_topic_0000001173630650_p36273255618">Log in to FusionInsight Manager and choose <strong id="mrs_01_24148__en-us_topic_0000001173630650_b14557164243019">Cluster</strong> &gt; <strong id="mrs_01_24148__en-us_topic_0000001173630650_b2122445113010">Services</strong> &gt; <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1936517491306">Kafka</strong>. On the displayed page, click <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1266911423116">Configurations</strong> and then <strong id="mrs_01_24148__en-us_topic_0000001173630650_b13479112213314">All Configurations</strong>, search for <strong id="mrs_01_24148__en-us_topic_0000001173630650_b12988123313112">allow.everyone.if.no.acl.found</strong>, set its value to <strong id="mrs_01_24148__en-us_topic_0000001173630650_b12582105613113">true</strong>, and click <strong id="mrs_01_24148__en-us_topic_0000001173630650_b4318185014366">Save</strong>.</p>
</li></ul>
</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li1961353335119"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b18417141711365">21422</strong>: HTTPS port number of the ClickHouseBalancer instance IP address.</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li9613633185115"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b452417284374">21425</strong>: HTTP port number of the ClickHouseBalancer instance IP address.</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li1761318332516">Parameters for batch write: Flink stores data in the memory and then flushes the data to the database table when the trigger condition is met. The configurations are as follows:<p id="mrs_01_24148__en-us_topic_0000001173630650_p19613133365120"><a name="mrs_01_24148__en-us_topic_0000001173630650_li1761318332516"></a><a name="en-us_topic_0000001173630650_li1761318332516"></a><strong id="mrs_01_24148__en-us_topic_0000001173630650_b16441941102414">sink.buffer-flush.max-rows</strong>: Number of rows written to ClickHouse. The default value is <strong id="mrs_01_24148__en-us_topic_0000001173630650_b127905212253">100</strong>.</p>
<p id="mrs_01_24148__en-us_topic_0000001173630650_p15613183385113"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b728842317244">sink.buffer-flush.interval</strong>: Interval for batch write. The default value is <strong id="mrs_01_24148__en-us_topic_0000001173630650_b15232111734811">1s</strong>.</p>
<p id="mrs_01_24148__en-us_topic_0000001173630650_p1661493310519">If either of the two conditions is met, a sink operation is triggered. That is, data will be flushed to the database table.</p>
<ul id="mrs_01_24148__en-us_topic_0000001173630650_ul7856112131110"><li id="mrs_01_24148__en-us_topic_0000001173630650_li14261685114">Scenario 1: sink every 60s<p id="mrs_01_24148__en-us_topic_0000001173630650_p1461493314511"><a name="mrs_01_24148__en-us_topic_0000001173630650_li14261685114"></a><a name="en-us_topic_0000001173630650_li14261685114"></a>'sink.buffer-flush.max-rows' = '0',</p>
<p id="mrs_01_24148__en-us_topic_0000001173630650_p17614133375113">'sink.buffer-flush.interval' = '60s'</p>
</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li999111691119">Scenario 2: sink every 100 rows<p id="mrs_01_24148__en-us_topic_0000001173630650_p1961419333515"><a name="mrs_01_24148__en-us_topic_0000001173630650_li999111691119"></a><a name="en-us_topic_0000001173630650_li999111691119"></a>'sink.buffer-flush.max-rows' = '100',</p>
<p id="mrs_01_24148__en-us_topic_0000001173630650_p136141333175111">'sink.buffer-flush.interval' = '0s'</p>
</li><li id="mrs_01_24148__en-us_topic_0000001173630650_li5517142181117">Scenario 3: no sink<p id="mrs_01_24148__en-us_topic_0000001173630650_p8614153315118"><a name="mrs_01_24148__en-us_topic_0000001173630650_li5517142181117"></a><a name="en-us_topic_0000001173630650_li5517142181117"></a>'sink.buffer-flush.max-rows' = '0',</p>
<p id="mrs_01_24148__en-us_topic_0000001173630650_p18614533195117">'sink.buffer-flush.interval' = '0s'</p>
</li></ul>
</li></ul>
</div></div>
</p></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li15985194854015"><span>On the job management page, check whether the job status is <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1660812893112">Running</strong>.</span></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li5414141215307"><span>Execute the following script to write data to Kafka. For details, see <a href="mrs_01_0379.html">Managing Messages in Kafka Topics</a>.</span><p><p id="mrs_01_24148__en-us_topic_0000001173630650_p19743153203810"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b136457202402">sh kafka-console-producer.sh --broker-list</strong> <em id="mrs_01_24148__en-us_topic_0000001173630650_ad4cb407ba2c54021864adb859619ccd3">IP address of the node where the Kafka instance locates</em><em id="mrs_01_24148__en-us_topic_0000001173630650_i1025175217472">:Kafka port number</em> <em id="mrs_01_24148__en-us_topic_0000001173630650_i51841516152514"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b7184191610251">--</strong></em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b9188141619255">topic</strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i7186131642517">Topic name</em> <strong id="mrs_01_24148__en-us_topic_0000001173630650_b64831959154510">--producer.config ../config/producer.properties</strong></p>
<p id="mrs_01_24148__en-us_topic_0000001173630650_p174351156414">For example, if the topic name is <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1047112419563">kinput</strong>, the script is <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1868584319427">sh kafka-console-producer.sh --broker-list</strong> <em id="mrs_01_24148__en-us_topic_0000001173630650_i9144137134213">IP address of the node where the Kafka instance locates</em><em id="mrs_01_24148__en-us_topic_0000001173630650_i03131843124710">:Kafka port number</em> <strong id="mrs_01_24148__en-us_topic_0000001173630650_b1449134754211">--topic kinput</strong> <strong id="mrs_01_24148__en-us_topic_0000001173630650_b193851165463">--producer.config ../config/producer.properties</strong>.</p>
<div class="p" id="mrs_01_24148__en-us_topic_0000001173630650_p835285214424">Enter the message content.<pre class="screen" id="mrs_01_24148__en-us_topic_0000001173630650_screen1869616424319">{"pid": "3","uid":false,"Int_16": "6533","Int_32": "429496294","Int_64": "1844674407370955614","String_x": "abc1","String_y": "abc1defghi","float_32": "0.1234","float_64": "95.1","Decimal_x": "0.451236414","Date_x": "2021-05-29","DateTime_x": "2021-05-21 10:05:10"},
{"pid": "4","uid":false,"Int_16": "6533","Int_32": "429496294","Int_64": "1844674407370955614","String_x": "abc1","String_y": "abc1defghi","float_32": "0.1234","float_64": "95.1","Decimal_x": "0.4512314","Date_x": "2021-05-29","DateTime_x": "2021-05-21 10:05:10"}</pre>
</div>
<p id="mrs_01_24148__en-us_topic_0000001173630650_adbfec31492e84dec85245866152e21be">Press <strong id="mrs_01_24148__en-us_topic_0000001173630650_b9625103216914">Enter</strong> to send the message.</p>
</p></li><li id="mrs_01_24148__en-us_topic_0000001173630650_li184902034134420"><span>Interconnect with ClickHouse to query the table data.</span><p><p id="mrs_01_24148__en-us_topic_0000001173630650_p194752022174512"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b819722317129">clickhouse client --host </strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i11204192315125">IP address of the ClickHouse instance</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b620632351216"> --user </strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i14207172311125">Username</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b16208192318122"> --password '</strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i152104239128">Password</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b1821112231126">'</strong><strong id="mrs_01_24148__en-us_topic_0000001173630650_b3213223101219">--port </strong><em id="mrs_01_24148__en-us_topic_0000001173630650_i1221472311219">ClickHouse port number</em><strong id="mrs_01_24148__en-us_topic_0000001173630650_b1121542321215"> --secure</strong> <strong id="mrs_01_24148__en-us_topic_0000001173630650_b421710239123">--multiline</strong></p>
<p id="mrs_01_24148__en-us_topic_0000001173630650_p1572165315457">Run the following command to check whether data is written to a specified ClickHouse table, for example, <strong id="mrs_01_24148__en-us_topic_0000001173630650_b156560218139">test1_all</strong>.</p>
<p id="mrs_01_24148__en-us_topic_0000001173630650_p89951781474"><strong id="mrs_01_24148__en-us_topic_0000001173630650_b107953274473">select * from test1_all;</strong></p>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24226.html">Interconnecting FlinkServer with External Components</a></div>
</div>
</div>
View Git Blame Copy Permalink