forked from docs/doc-exports
Su, Xiaomeng
04d4597cf3
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
225 lines
26 KiB
HTML
225 lines
26 KiB
HTML
<a name="dli_08_0401"></a><a name="dli_08_0401"></a>
|
|
|
|
<h1 class="topictitle1">Upsert Kafka Result Table</h1>
|
|
<div id="body8662426"><div class="section" id="dli_08_0401__en-us_topic_0000001310015809_section77915506517"><h4 class="sectiontitle">Function</h4><p id="dli_08_0401__en-us_topic_0000001310015809_p1370804594310">Apache Kafka is a fast, scalable, and fault-tolerant distributed message publishing and subscription system. It delivers high throughput and built-in partitions and provides data replicas and fault tolerance. Apache Kafka is applicable to scenarios of handling massive messages. DLI outputs the Flink job output data to Kafka in upsert mode.</p>
|
|
<p id="dli_08_0401__en-us_topic_0000001310015809_p661816320442">The Upsert Kafka connector allows for reading data from and writing data into Kafka topics in the upsert fashion.</p>
|
|
<p id="dli_08_0401__en-us_topic_0000001310015809_p111651030194311">As a sink, the Upsert Kafka connector can consume a changelog stream. It will write INSERT/UPDATE_AFTER data as normal Kafka messages value, and write DELETE data as Kafka messages with null values (indicate tombstone for the key). Flink will guarantee the message ordering on the primary key by partition data on the values of the primary key columns, so the UPDATE/DELETE messages on the same key will fall into the same partition.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0401__en-us_topic_0000001310015809_dli_08_0256_en-us_topic_0132788972_section2579142713429"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0401__en-us_topic_0000001310015809_ul578517503176"><li id="dli_08_0401__en-us_topic_0000001310015809_li2785750141711">You have created a Kafka cluster.</li><li id="dli_08_0401__en-us_topic_0000001310015809_li1627119813515">An enhanced datasource connection has been created for DLI to connect to Kafka clusters, so that jobs can run on the dedicated queue of DLI and you can set the security group rules as required.
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0401__en-us_topic_0000001310015809_section1230618441125"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0401__en-us_topic_0000001310015809_ul181711754173415"><li id="dli_08_0401__en-us_topic_0000001310015809_li21711354143416">When creating a Flink OpenSource SQL job, you need to set <strong id="dli_08_0401__en-us_topic_0000001310015809_b19555174892515">Flink Version</strong> to <strong id="dli_08_0401__en-us_topic_0000001310015809_b4556194852516">1.12</strong> on the <strong id="dli_08_0401__en-us_topic_0000001310015809_b12557134882512">Running Parameters</strong> tab of the job editing page, select <strong id="dli_08_0401__en-us_topic_0000001310015809_b14558194802519">Save Job Log</strong>, and set the OBS bucket for saving job logs.</li><li id="dli_08_0401__en-us_topic_0000001310015809_li289283494718">For details about how to use data types, see section <a href="dli_08_0407.html#dli_08_0407">Format</a>.</li><li id="dli_08_0401__en-us_topic_0000001310015809_li4188135519185">The Upsert Kafka always works in the upsert fashion and requires to define the primary key in the DDL.</li><li id="dli_08_0401__en-us_topic_0000001310015809_li858312295191">By default, an Upsert Kafka sink ingests data with at-least-once guarantees into a Kafka topic if the query is executed with checkpointing enabled. This means that Flink may write duplicate records with the same key into the Kafka topic. Therefore, the Upsert Kafka connector achieves idempotent writes.</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0401__en-us_topic_0000001310015809_section892013201167"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0401__en-us_topic_0000001310015809_dli_08_0256_screen1461215294716"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">kafkaSource</span><span class="p">(</span>
|
|
<span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="p">,</span><span class="w"> </span><span class="p">...)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">ENFORCED</span><span class="p">)</span>
|
|
<span class="p">)</span>
|
|
<span class="k">with</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="s1">'connector'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'upsert-kafka'</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'topic'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'properties.bootstrap.servers'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'key.format'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="s1">'value.format'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span>
|
|
<span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0401__en-us_topic_0000001310015809_section893519312815"><h4 class="sectiontitle">Parameters</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0401__en-us_topic_0000001310015809_table82421231587" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0401__en-us_topic_0000001310015809_row13242831486"><th align="left" class="cellrowborder" valign="top" width="16.3%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0401__en-us_topic_0000001310015809_p524213118810">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="11.12%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0401__en-us_topic_0000001310015809_p1124218311813">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="11.98%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0401__en-us_topic_0000001310015809_p8335102131613">Default Value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="10.48%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0401__en-us_topic_0000001310015809_p12962019151617">Data Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50.12%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0401__en-us_topic_0000001310015809_p15242163119815">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0401__en-us_topic_0000001310015809_row15242113115811"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0401__en-us_topic_0000001310015809_p02421431888">connector</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.12%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0401__en-us_topic_0000001310015809_p112425311817">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.98%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0401__en-us_topic_0000001310015809_p133632161617">(none)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.48%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0401__en-us_topic_0000001310015809_p229631913168">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.12%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0401__en-us_topic_0000001310015809_p11242431287">Connector to be used. Set this parameter to <strong id="dli_08_0401__en-us_topic_0000001310015809_b59726382336">upsert-kafka</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0401__en-us_topic_0000001310015809_row82421531587"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0401__en-us_topic_0000001310015809_p2024317318813">topic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.12%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0401__en-us_topic_0000001310015809_p12431831982">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.98%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0401__en-us_topic_0000001310015809_p133616291619">(none)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.48%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0401__en-us_topic_0000001310015809_p1429741971613">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.12%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0401__en-us_topic_0000001310015809_p12432311483">Kafka topic name.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0401__en-us_topic_0000001310015809_row724317317820"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0401__en-us_topic_0000001310015809_p924303118810">properties.bootstrap.servers</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.12%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0401__en-us_topic_0000001310015809_p32435314815">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.98%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0401__en-us_topic_0000001310015809_p1333611214161">(none)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.48%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0401__en-us_topic_0000001310015809_p162972194169">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.12%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0401__en-us_topic_0000001310015809_p024319311282">Comma separated list of Kafka brokers.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0401__en-us_topic_0000001310015809_row17243203117811"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0401__en-us_topic_0000001310015809_p112431031983">key.format</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.12%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0401__en-us_topic_0000001310015809_p124317311385">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.98%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0401__en-us_topic_0000001310015809_p183361526165">(none)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.48%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0401__en-us_topic_0000001310015809_p9297171920164">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.12%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0401__en-us_topic_0000001310015809_p9603137143311">Format used to deserialize and serialize the key part of Kafka messages. The key fields are specified by the <strong id="dli_08_0401__en-us_topic_0000001310015809_b1582959143320">PRIMARY KEY</strong> syntax. The following formats are supported:</p>
|
|
<ul id="dli_08_0401__en-us_topic_0000001310015809_ul6122242153320"><li id="dli_08_0401__en-us_topic_0000001310015809_li19328194414334">csv</li><li id="dli_08_0401__en-us_topic_0000001310015809_li1156344893319">json</li><li id="dli_08_0401__en-us_topic_0000001310015809_li6675158452">avro</li></ul>
|
|
<p id="dli_08_0401__en-us_topic_0000001310015809_p13812342331">Refer to <a href="dli_08_0407.html#dli_08_0407">Format</a> for more details and format parameters.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0401__en-us_topic_0000001310015809_row1912194914110"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0401__en-us_topic_0000001310015809_p179131249144115">key.fields-prefix</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.12%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0401__en-us_topic_0000001310015809_p4913149114111">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.98%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0401__en-us_topic_0000001310015809_p99133497414">(none)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.48%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0401__en-us_topic_0000001310015809_p16913114917417">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.12%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0401__en-us_topic_0000001310015809_p789918504320">Defines a custom prefix for all fields of the key format to avoid name clashes with fields of the value format.</p>
|
|
<p id="dli_08_0401__en-us_topic_0000001310015809_p39131349154111">By default, the prefix is empty. If a custom prefix is defined, both the table schema and <strong id="dli_08_0401__en-us_topic_0000001310015809_b195710175346">key.fields</strong> will work with prefixed names. When constructing the data type of the key format, the prefix will be removed and the non-prefixed names will be used within the key format. Note that this option requires that <strong id="dli_08_0401__en-us_topic_0000001310015809_b428382417341">value.fields-include</strong> must be set to <strong id="dli_08_0401__en-us_topic_0000001310015809_b9284132413344">EXCEPT_KEY</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0401__en-us_topic_0000001310015809_row750314113432"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0401__en-us_topic_0000001310015809_p145042011104317">value.format</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.12%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0401__en-us_topic_0000001310015809_p85041711124316">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.98%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0401__en-us_topic_0000001310015809_p18504191154313">(none)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.48%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0401__en-us_topic_0000001310015809_p10504151120439">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.12%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0401__en-us_topic_0000001310015809_p666811347347">Format used to deserialize and serialize the value part of Kafka messages. The following formats are supported:</p>
|
|
<ul id="dli_08_0401__en-us_topic_0000001310015809_ul19198153716341"><li id="dli_08_0401__en-us_topic_0000001310015809_li18422154073420">csv</li><li id="dli_08_0401__en-us_topic_0000001310015809_li19501446349">json</li><li id="dli_08_0401__en-us_topic_0000001310015809_li17198153714342">avro</li></ul>
|
|
<p id="dli_08_0401__en-us_topic_0000001310015809_p6207691445">Refer to <a href="dli_08_0407.html#dli_08_0407">Format</a> for more details and format parameters.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0401__en-us_topic_0000001310015809_row4575174020442"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0401__en-us_topic_0000001310015809_p0575154004417">value.fields-include</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.12%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0401__en-us_topic_0000001310015809_p4576940134417">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.98%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0401__en-us_topic_0000001310015809_p195761240124419">'ALL'</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.48%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0401__en-us_topic_0000001310015809_p1957634016440">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.12%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0401__en-us_topic_0000001310015809_p1576154015440">Controls which fields should appear in the value part. Options:</p>
|
|
<ul id="dli_08_0401__en-us_topic_0000001310015809_ul198451237165110"><li id="dli_08_0401__en-us_topic_0000001310015809_li484553712516"><strong id="dli_08_0401__en-us_topic_0000001310015809_b1581101013511">ALL</strong>: All fields in the schema, including the primary key field, are included in the value part.</li><li id="dli_08_0401__en-us_topic_0000001310015809_li16282847523"><strong id="dli_08_0401__en-us_topic_0000001310015809_b16562316103513">EXCEPT_KEY</strong>: All the fields of the table schema are included, except the primary key field.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0401__en-us_topic_0000001310015809_row424319311581"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0401__en-us_topic_0000001310015809_p7721553467">sink.parallelism</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.12%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0401__en-us_topic_0000001310015809_p42431315820">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.98%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0401__en-us_topic_0000001310015809_p1933602171612">(none)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.48%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0401__en-us_topic_0000001310015809_p122972019181615">Interger</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.12%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0401__en-us_topic_0000001310015809_p922053312465">Defines the parallelism of the Upsert Kafka sink operator. By default, the parallelism is determined by the framework using the same parallelism of the upstream chained operator.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0401__en-us_topic_0000001310015809_row56454611484"><td class="cellrowborder" valign="top" width="16.3%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0401__en-us_topic_0000001310015809_p18929131412486">properties.*</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.12%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0401__en-us_topic_0000001310015809_p793014149489">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="11.98%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0401__en-us_topic_0000001310015809_p1493061417486">(none)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.48%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0401__en-us_topic_0000001310015809_p1293061484819">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.12%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0401__en-us_topic_0000001310015809_p2194199519">This option can set and pass arbitrary Kafka configurations.</p>
|
|
<p id="dli_08_0401__en-us_topic_0000001310015809_p4776162717512">The suffix of this parameter must match the parameter defined in <a href="https://kafka.apache.org/documentation/#configuration" target="_blank" rel="noopener noreferrer">Kafka Configuration documentation</a>. Flink will remove the <strong id="dli_08_0401__en-us_topic_0000001310015809_b10462153312369">properties.</strong> key prefix and pass the transformed key and value to the underlying KafkaClient.</p>
|
|
<p id="dli_08_0401__en-us_topic_0000001310015809_p12930141413480">For example, you can disable automatic topic creation via <strong id="dli_08_0401__en-us_topic_0000001310015809_b2315163953613">'properties.allow.auto.create.topics' = 'false'</strong>. But there are some configurations that do not support to set, because Flink will override them, for example, <strong id="dli_08_0401__en-us_topic_0000001310015809_b3518114213363">'key.deserializer'</strong> and <strong id="dli_08_0401__en-us_topic_0000001310015809_b9519442123617">'value.deserializer'</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0401__en-us_topic_0000001310015809_section12707752101211"><h4 class="sectiontitle">Example</h4><p id="dli_08_0401__en-us_topic_0000001310015809_p131929412711">In this example, Kafka source topic data is read from the Kafka source table and written to the Kafka sink topic through the Upsert Kafka result table.</p>
|
|
<ol id="dli_08_0401__en-us_topic_0000001310015809_ol194356442062"><li id="dli_08_0401__en-us_topic_0000001310015809_li6127741153217">Create an enhanced datasource connection in the VPC and subnet where Kafka locates, and bind the connection to the required Flink elastic resource pool.</li><li id="dli_08_0401__en-us_topic_0000001310015809_li1233002211514">Set Kafka security groups and add inbound rules to allow access from the Flink queue. Test the connectivity using the Kafka address. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</li><li id="dli_08_0401__en-us_topic_0000001310015809_li5712045101217">Create a Flink OpenSource SQL job. Enter the following job script and submit the job.<div class="p" id="dli_08_0401__en-us_topic_0000001310015809_p1111612171319"><a name="dli_08_0401__en-us_topic_0000001310015809_li5712045101217"></a><a name="en-us_topic_0000001310015809_li5712045101217"></a>When you create a job, set <strong id="dli_08_0401__en-us_topic_0000001310015809_b4609212103913">Flink Version</strong> to <strong id="dli_08_0401__en-us_topic_0000001310015809_b261001283913">1.12</strong> on the <strong id="dli_08_0401__en-us_topic_0000001310015809_b14610712113913">Running Parameters</strong> tab. Select <strong id="dli_08_0401__en-us_topic_0000001310015809_b361171215399">Save Job Log</strong>, and specify the OBS bucket for saving job logs. <strong id="dli_08_0401__en-us_topic_0000001310015809_b137771152398">Change the values of the parameters in bold as needed in the following script.</strong><pre class="screen" id="dli_08_0401__en-us_topic_0000001310015809_screen15061439184715">CREATE TABLE orders (
|
|
order_id string,
|
|
order_channel string,
|
|
order_time string,
|
|
pay_amount double,
|
|
real_pay double,
|
|
pay_time string,
|
|
user_id string,
|
|
user_name string,
|
|
area_id string
|
|
) WITH (
|
|
'connector' = 'kafka',
|
|
'topic' = '<em id="dli_08_0401__en-us_topic_0000001310015809_i5663552125810"><strong id="dli_08_0401__en-us_topic_0000001310015809_b544510556581">KafkaTopic</strong></em>',
|
|
'properties.bootstrap.servers' = '<em id="dli_08_0401__en-us_topic_0000001310015809_i1477713010594"><strong id="dli_08_0401__en-us_topic_0000001310015809_b14777309596">Kafka</strong></em><em id="dli_08_0401__en-us_topic_0000001310015809_i15795191714599"><strong id="dli_08_0401__en-us_topic_0000001310015809_b1454541665919">Address1:KafkaPort,KafkAddress2:KafkaPort</strong></em>',
|
|
'properties.group.id' = '<em id="dli_08_0401__en-us_topic_0000001310015809_i48641216140"><strong id="dli_08_0401__en-us_topic_0000001310015809_b1258711171413">GroupId</strong></em>',
|
|
'scan.startup.mode' = 'latest-offset',
|
|
"format" = "json"
|
|
);
|
|
CREATE TABLE UPSERTKAFKASINK (
|
|
order_id string,
|
|
order_channel string,
|
|
order_time string,
|
|
pay_amount double,
|
|
real_pay double,
|
|
pay_time string,
|
|
user_id string,
|
|
user_name string,
|
|
area_id string,
|
|
PRIMARY KEY (order_id) NOT ENFORCED
|
|
) WITH (
|
|
'connector' = 'upsert-kafka',
|
|
'topic' = '<em id="dli_08_0401__en-us_topic_0000001310015809_i375271191411"><strong id="dli_08_0401__en-us_topic_0000001310015809_b1875351112147">KafkaTopic</strong></em>',
|
|
'properties.bootstrap.servers' = '<em id="dli_08_0401__en-us_topic_0000001310015809_i10367120181416"><strong id="dli_08_0401__en-us_topic_0000001310015809_b7367112011141">Kafka</strong></em><em id="dli_08_0401__en-us_topic_0000001310015809_i15367102051410"><strong id="dli_08_0401__en-us_topic_0000001310015809_b3367112018143">Address1:KafkaPort,KafkAddress2:KafkaPort</strong></em>',
|
|
'key.format' = 'json',
|
|
'value.format' = 'json'
|
|
);
|
|
insert into UPSERTKAFKASINK
|
|
select * from orders;</pre>
|
|
</div>
|
|
</li><li id="dli_08_0401__en-us_topic_0000001310015809_li34350440610">Connect to the Kafka cluster and send the following test data to the Kafka source topic:<pre class="screen" id="dli_08_0401__en-us_topic_0000001310015809_screen985814717447">{"order_id":"202103251202020001", "order_channel":"miniAppShop", "order_time":"2021-03-25 12:02:02", "pay_amount":"60.00", "real_pay":"60.00", "pay_time":"2021-03-25 12:03:00", "user_id":"0002", "user_name":"Bob", "area_id":"330110"}
|
|
|
|
{"order_id":"202103251505050001", "order_channel":"qqShop", "order_time":"2021-03-25 15:05:05", "pay_amount":"500.00", "real_pay":"400.00", "pay_time":"2021-03-25 15:10:00", "user_id":"0003", "user_name":"Cindy", "area_id":"330108"}
|
|
|
|
{"order_id":"202103251202020001", "order_channel":"miniAppShop", "order_time":"2021-03-25 12:02:02", "pay_amount":"60.00", "real_pay":"60.00", "pay_time":"2021-03-25 12:03:00", "user_id":"0002", "user_name":"Bob", "area_id":"330110"}</pre>
|
|
</li><li id="dli_08_0401__en-us_topic_0000001310015809_li176961346201518">Connect to the Kafka cluster and read data from the Kafka sink topic. The result is as follows:<pre class="screen" id="dli_08_0401__en-us_topic_0000001310015809_screen4147849184516">{"order_id":"202103251202020001", "order_channel":"miniAppShop", "order_time":"2021-03-25 12:02:02", "pay_amount":"60.00", "real_pay":"60.00", "pay_time":"2021-03-25 12:03:00", "user_id":"0002", "user_name":"Bob", "area_id":"330110"}
|
|
|
|
{"order_id":"202103251505050001", "order_channel":"qqShop", "order_time":"2021-03-25 15:05:05", "pay_amount":"500.00", "real_pay":"400.00", "pay_time":"2021-03-25 15:10:00", "user_id":"0003", "user_name":"Cindy", "area_id":"330108"}
|
|
|
|
{"order_id":"202103251202020001", "order_channel":"miniAppShop", "order_time":"2021-03-25 12:02:02", "pay_amount":"60.00", "real_pay":"60.00", "pay_time":"2021-03-25 12:03:00", "user_id":"0002", "user_name":"Bob", "area_id":"330110"}</pre>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_08_0401__en-us_topic_0000001310015809_section4902740144614"><h4 class="sectiontitle">FAQ</h4><p id="dli_08_0401__en-us_topic_0000001310015809_p1328805554611">None</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0391.html">Creating Result Tables</a></div>
|
|
</div>
|
|
</div>
|
|
|