vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0308.html
Su, Xiaomeng 04d4597cf3 dli_sqlreference_0511_version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2023-11-02 14:34:08 +00:00

158 lines
19 KiB
HTML
Raw Blame History

<a name="dli_08_0308"></a><a name="dli_08_0308"></a>
<h1 class="topictitle1">Kafka Result Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_dli_08_0256_en-us_topic_0132788972_section108631122164917"><h4 class="sectiontitle">Function</h4><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p661816320442">DLI exports the output data of the Flink job to Kafka.</p>
<p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p2618163211448">Apache Kafka is a fast, scalable, and fault-tolerant distributed message publishing and subscription system. It delivers high throughput and built-in partitions and provides data replicas and fault tolerance. Apache Kafka is applicable to scenarios of handling massive messages.</p>
</div>
<div class="section" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_dli_08_0256_en-us_topic_0132788972_section2579142713429"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_dli_08_0237_p1038754112542">Kafka is an offline cluster. You have built an enhanced datasource connection to connect Flink jobs to Kafka. You have set security group rules as required.</p>
</div>
<div class="section" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_section13191959213"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p247314912217">SASL_SSL cannot be enabled for the interconnected Kafka cluster.</p>
</div>
<div class="section" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_dli_08_0256_en-us_topic_0132788972_section3126105364419"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_dli_08_0256_screen1461215294716"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">kafkaSource</span><span class="p">(</span>
<span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="p">,</span><span class="w"> </span><span class="p">...)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">ENFORCED</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'kafka'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.version'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.topic'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.properties.bootstrap.servers'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'format.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_dli_08_0256_section4299113491"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_table11617424154613" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row146177242466"><th align="left" class="cellrowborder" valign="top" width="20.99209920992099%" id="mcps1.3.5.2.2.4.1.1"><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p1361712418461">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="11.391139113911391%" id="mcps1.3.5.2.2.4.1.2"><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p176171424114615">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="67.61676167616763%" id="mcps1.3.5.2.2.4.1.3"><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p261712247467">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row136171242461"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p361702424615">connector.type</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p1761716242465">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p5617112404614">Connector type. Set this parameter to <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b138211026184615">kafka</strong>.</p>
</td>
</tr>
<tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row1961742414462"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p461719249469">connector.version</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p12617192413467">No</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p1461712414613">Kafka version. The value can be '<strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b7851144163520">0.10'</strong> or <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b685744110355">'0.11'</strong>, which corresponds to Kafka 2.11 to 2.4.0 and other historical versions, respectively.</p>
</td>
</tr>
<tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row1761802415461"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p4618142424610">format.type</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p961814244460">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p56183243463">Data serialization format. The value can be <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b1485316558464">csv</strong>, <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b31735574468">json</strong>, or <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b13432195810464">avro</strong>.</p>
</td>
</tr>
<tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row476917301215"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p1919231813244">format.field-delimiter</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p17192151882416">No</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p31922184244">Attribute delimiter. You can customize the attribute delimiter only when the encoding format is CSV. The default delimiter is a comma (,).</p>
</td>
</tr>
<tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row1261892474618"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p7618112414465">connector.topic</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p9618124174619">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p16181324114615">Kafka topic name.</p>
</td>
</tr>
<tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row15618102474612"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p661814243464">connector.properties.bootstrap.servers</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p18618182410467">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p166188243464">Kafka broker addresses. Use commas (,) to separated them.</p>
</td>
</tr>
<tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row1561842464619"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p20618122424610">connector.sink-partitioner</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p961852420461">No</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p14618824124619">Partitioner type. The value can be <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b82831050144717">fixed</strong>, <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b10962155254718">round-robin</strong>, or <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b226355420475">custom</strong>.</p>
</td>
</tr>
<tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row16618102434612"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p13618192454615">connector.sink-partitioner-class</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p2618102417465">No</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p76181124104612">Custom partitioner. This parameter is mandatory when <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b75281927194818">sink-partitioner</strong> is <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b47823613480">custom</strong>, for example, <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b2105452488">org.mycompany.MyPartitioner</strong>.</p>
</td>
</tr>
<tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row126181924134616"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p1961815245463">update-mode</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p0618624174610">No</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p1561872494611">Data update mode. Three write modes are supported: <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b13300112215497">append</strong>, <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b13107541204919">retract</strong>, and <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b1624854354913">upsert</strong>.</p>
</td>
</tr>
<tr id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_row967410521254"><td class="cellrowborder" valign="top" width="20.99209920992099%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p124303110813">connector.properties.*</p>
</td>
<td class="cellrowborder" valign="top" width="11.391139113911391%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p924317312813">No</p>
</td>
<td class="cellrowborder" valign="top" width="67.61676167616763%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p1243143120814">Native properties of Kafka</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_dli_08_0256_en-us_topic_0132788972_section19670175612466"><h4 class="sectiontitle">Example</h4><div class="p" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_p1326131081119">Output the data in <strong id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_b158861655165017">kafkaSink</strong> to Kafka.<div class="codecoloring" codetype="Sql" id="dli_08_0308__en-us_topic_0000001119232082_en-us_topic_0000001081419146_dli_08_0256_en-us_topic_0132788972_screen539510164159"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">kafkaSink</span><span class="p">(</span>
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_speed</span><span class="w"> </span><span class="nb">INT</span><span class="p">)</span>
<span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'kafka'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.version'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'0.10'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.topic'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'test-topic'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.properties.bootstrap.servers'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xx.xx.xx.xx:9092'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'connector.sink-partitioner'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'round-robin'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'format.type'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'csv'</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0307.html">Creating a Result Table</a></div>
</div>
</div>
View Git Blame Copy Permalink