forked from docs/doc-exports
Su, Xiaomeng
76a5b1ee83
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
221 lines
26 KiB
HTML
221 lines
26 KiB
HTML
<a name="dli_08_0239"></a><a name="dli_08_0239"></a>
|
|
|
|
<h1 class="topictitle1">Open-Source Kafka Source Stream</h1>
|
|
<div id="body1574393070651"><div class="section" id="dli_08_0239__en-us_topic_0113887276_en-us_topic_0111499973_section13926185035213"><h4 class="sectiontitle">Function</h4><p id="dli_08_0239__en-us_topic_0111499973_p18963184212118">Create a source stream to obtain data from Kafka as input data for jobs.</p>
|
|
<p id="dli_08_0239__en-us_topic_0111499973_p433435316341">Apache Kafka is a fast, scalable, and fault-tolerant distributed message publishing and subscription system. It delivers high throughput and built-in partitions and provides data replicas and fault tolerance. Apache Kafka is applicable to scenarios of handling massive messages.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0239__en-us_topic_0113887276_en-us_topic_0111499973_section19566131381513"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_ul19567726133916"><li id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_li729542114118">If the Kafka server listens on the port using hostname, you need to add the mapping between the hostname and IP address of the Kafka Broker node to the DLI queue. Contact the Kafka service deployment personnel to obtain the hostname and IP address of the Kafka Broker node. For details about how to add an IP-domain mapping, see <strong id="dli_08_0239__dli_08_0238_b1189481225013">Enhanced Datasource Connections</strong> > <strong id="dli_08_0239__dli_08_0238_b689517126505">Modifying the Host Information</strong> in the <em id="dli_08_0239__dli_08_0238_i1589518121509">Data Lake Insight User Guide</em>.</li><li id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_li2741672136">Kafka is an offline cluster. You need to use the enhanced datasource connection function to connect Flink jobs to Kafka. You can also set security group rules as required.<p id="dli_08_0239__dli_08_0238_p94091618103916"><a name="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_li2741672136"></a><a name="dli_08_0238_en-us_topic_0113887276_en-us_topic_0111562554_li2741672136"></a>For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_08_0239__dli_08_0238_dli_08_0249_en-us_topic_0132966325_filepath13726182511815"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_08_0239__dli_08_0238_dli_08_0249_cite534516e564224b13bad1de4471da3139181527">Data Lake Insight User Guide</cite></i>.</p>
|
|
<p id="dli_08_0239__dli_08_0238_p17953337436">For details about how to configure security group rules, see <span class="filepath" id="dli_08_0239__dli_08_0238_dli_08_0249_filepath78235157113"><b>Security Group</b></span> in the <em id="dli_08_0239__dli_08_0238_dli_08_0249_i082519153111">Virtual Private Cloud User Guide</em>.</p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0239__en-us_topic_0113887276_en-us_topic_0111499973_section1810912715521"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0239__screen29519537283"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span>
|
|
<span class="normal">8</span>
|
|
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">SOURCE</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">stream_id</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"kafka"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_bootstrap_servers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_group_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_topic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"json"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">json_config</span><span class="o">=</span><span class="ss">""</span>
|
|
<span class="w"> </span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0239__section1594243973712"><h4 class="sectiontitle">Keywords</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_table15955155035214" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row29690509522"><th align="left" class="cellrowborder" valign="top" width="22.12%" id="mcps1.3.4.2.2.4.1.1"><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p697520507529">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="12.7%" id="mcps1.3.4.2.2.4.1.2"><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p997925010528">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="65.18%" id="mcps1.3.4.2.2.4.1.3"><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p9983150185215">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row398835017524"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p6993185085210">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p12996185095215">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p22751105212">Data source type. <span class="parmvalue" id="dli_08_0239__parmvalue3288456165910"><b>Kafka</b></span> indicates that the data source is Kafka.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row210135125214"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p1014115195212">kafka_bootstrap_servers</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p171745175210">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p112245114525">Port that connects DLI to Kafka. Use enhanced datasource connections to connect DLI queues with Kafka clusters.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row124195117528"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p1029151195216">kafka_group_id</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p153375118522">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p23725115214">Group ID.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row15401751115211"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p20447517528">kafka_topic</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p1550185145218">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p105418515527">Kafka topic to be read. Currently, only one topic can be read at a time.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_row9567519522"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p1409195216203">encode</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p66618519523">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_en-us_topic_0111499973_p1870145119522">Data encoding format. The value can be <span class="parmvalue" id="dli_08_0239__parmvalue73249297283"><b>csv</b></span>, <span class="parmvalue" id="dli_08_0239__parmvalue13241329132819"><b>json</b></span>, <span class="parmvalue" id="dli_08_0239__parmvalue8324729112815"><b>blob</b></span>, or <span class="parmvalue" id="dli_08_0239__parmvalue14324152912280"><b>user_defined</b></span>.</p>
|
|
<ul id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_ul1234754115422"><li id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_li5347114174217"><span class="parmvalue" id="dli_08_0239__parmvalue61231630142817"><b>field_delimiter</b></span> must be specified if this parameter is set to <span class="parmvalue" id="dli_08_0239__parmvalue11123830132819"><b>csv</b></span>.</li><li id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_li734784174215"><span class="parmvalue" id="dli_08_0239__parmvalue1440743510287"><b>json_config</b></span> must be specified if this parameter is set to <span class="parmvalue" id="dli_08_0239__parmvalue10407135152810"><b>json</b></span>.</li><li id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_li15776193716382">If this parameter is set to <strong id="dli_08_0239__b19917174016283">blob</strong>, the received data will not be parsed, and only one Array[TINYINT] field exists in the table.</li><li id="dli_08_0239__dli_08_0238_li45109622213"><span class="parmname" id="dli_08_0239__parmname17726124632818"><b>encode_class_name</b></span> and <span class="parmname" id="dli_08_0239__parmname1873274618282"><b>encode_class_parameter</b></span> must be specified if this parameter is set to <span class="parmvalue" id="dli_08_0239__parmvalue373216468283"><b>user_defined</b></span>.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_row19540101012412"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0111501791_p1588112121610">encode_class_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0111501791_p2881182101617">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0111501791_p68811521201617">If <strong id="dli_08_0239__b63811354143018">encode</strong> is set to <strong id="dli_08_0239__b12387135413015">user_defined</strong>, you need to set this parameter to the name of the user-defined decoding class (including the complete package path). The class must inherit the <strong id="dli_08_0239__b438855413010">DeserializationSchema</strong> class.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_row1467311517246"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0111501791_p8881321191617">encode_class_parameter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0111501791_p1988120218166">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0111501791_p198821021151620">If <strong id="dli_08_0239__b560611793111">encode</strong> is set to <strong id="dli_08_0239__b2612127203114">user_defined</strong>, you can set this parameter to specify the input parameter of the user-defined decoding class. Only one parameter of the string type is supported.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_row106903351133"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p12812938161811">json_config</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p885216280195">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__p11444152182617">If <strong id="dli_08_0239__b1643293710524">encode</strong> is set to <strong id="dli_08_0239__b11438173735219">json</strong>, you can use this parameter to specify the mapping between JSON fields and stream attributes. </p>
|
|
<p id="dli_08_0239__p112713374368">The format is field1=json_field1;field2=json_field2.</p>
|
|
<p id="dli_08_0239__p1985719473364"><strong id="dli_08_0239__b1265211142113">field1</strong> and <strong id="dli_08_0239__b12652101119217">field2</strong> indicate the names of the created table fields. <strong id="dli_08_0239__b17371645132115">json_field1</strong> and <strong id="dli_08_0239__b113734514217">json_field2</strong> are key fields of the JSON strings in the Kafka input data.</p>
|
|
<p id="dli_08_0239__p1759319793714">For details, see <a href="#dli_08_0239__en-us_topic_0113887276_en-us_topic_0111499973_section444391525220">Example</a>.</p>
|
|
<div class="note" id="dli_08_0239__note1988311238262"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_0239__p10509115814149">If the attribute names in the source stream are the same as those in JSON fields, you do not need to set this parameter.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_row19610134014133"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p163688362615">field_delimiter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p113701136269">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p103708364611">If <strong id="dli_08_0239__b10350165063119">encode</strong> is set to <strong id="dli_08_0239__b33563505314">csv</strong>, you can use this parameter to specify the separator between CSV fields. By default, the comma (,) is used.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_row34014525134"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p14728918132620">quote</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p8730131818266">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p397121416310">Quoted symbol in a data format. The attribute delimiters between two quoted symbols are treated as common characters.</p>
|
|
<ul id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_ul18631537631"><li id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_li10631173715314">If double quotation marks are used as the quoted symbol, set this parameter to <strong id="dli_08_0239__b1225093014323">\u005c\u0022</strong> for character conversion.</li><li id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_li1963112378318">If a single quotation mark is used as the quoted symbol, set this parameter to a single quotation mark (').</li></ul>
|
|
<div class="note" id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_note1377013361642"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0239__dli_08_0238_ul17578193311169"><li id="dli_08_0239__dli_08_0238_li13578123361611">Currently, only the CSV format is supported.</li><li id="dli_08_0239__dli_08_0238_li155791333201616">After this parameter is specified, ensure that each field does not contain quoted symbols or contains an even number of quoted symbols. Otherwise, parsing will fail.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_row181247017287"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p151265010289">start_time</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p1012712012285">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p14681221163019">Start time when Kafka data is ingested.</p>
|
|
<p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p171271404288">If this parameter is specified, DLI reads data read from the specified time. The format is <strong id="dli_08_0239__b1746403823519">yyyy-MM-dd HH:mm:ss</strong>. Ensure that the value of <strong id="dli_08_0239__b376425810352">start_time</strong> is not later than the current time. Otherwise, no data will be obtained.</p>
|
|
<p id="dli_08_0239__p866145217339">If you set this parameter, only the data generated after the specified time for the Kafka topic will be read.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_row687701112365"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p17878511163619">kafka_properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p1087801193613">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_en-us_topic_0113887276_p158789118366">Native properties of Kafka. The format is <strong id="dli_08_0239__b39591529192515">key1=value1;key2=value2</strong>. For details about the property values, see the description in <a href="https://kafka.apache.org/documentation/#configuration" target="_blank" rel="noopener noreferrer">Apache Kafka</a>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0239__dli_08_0238_row464612616345"><td class="cellrowborder" valign="top" width="22.12%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0239__dli_08_0238_p1564718693412">kafka_certificate_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="12.7%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0239__dli_08_0238_p2648766342">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.18%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0239__dli_08_0238_p8648761341">Name of the datasource authentication information. This parameter is valid only when the datasource authentication type is set to <strong id="dli_08_0239__b775855915362">Kafka_SSL</strong>.</p>
|
|
<div class="note" id="dli_08_0239__dli_08_0238_note2261630114619"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0239__dli_08_0238_ul137206345480"><li id="dli_08_0239__dli_08_0238_li2720123464812">If this parameter is specified, the service loads only the specified file and password under the authentication. The system automatically sets this parameter to <strong id="dli_08_0239__b822912410374">kafka_properties</strong>.</li><li id="dli_08_0239__dli_08_0238_li18720143413484">Other configuration information required for Kafka SSL authentication needs to be manually configured in the <strong id="dli_08_0239__b8781513133715">kafka_properties</strong> attribute.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0239__section1041914561381"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0239__dli_08_0238_p11761959195610">When creating a source stream, you can specify a time model for subsequent calculation. Currently, DLI supports two time models: Processing Time and Event Time. For details about the syntax, see <a href="dli_08_0107.html">Configuring Time Models</a>.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0239__en-us_topic_0113887276_en-us_topic_0111499973_section444391525220"><a name="dli_08_0239__en-us_topic_0113887276_en-us_topic_0111499973_section444391525220"></a><a name="en-us_topic_0113887276_en-us_topic_0111499973_section444391525220"></a><h4 class="sectiontitle">Example</h4><ul id="dli_08_0239__ul7999422195817"><li id="dli_08_0239__li1299922214587">Read Kafka topic <strong id="dli_08_0239__b33001521132118">test</strong>. The data encoding format is non-nested JSON, for example, {"attr1": "lilei", "attr2": 18}.<div class="codecoloring" codetype="Sql" id="dli_08_0239__screen8298555912"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span>
|
|
<span class="normal">8</span>
|
|
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">SOURCE</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">kafka_source</span><span class="w"> </span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span><span class="w"> </span><span class="n">age</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span>
|
|
<span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"kafka"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">kafka_bootstrap_servers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"ip1:port1,ip2:port2"</span><span class="p">,</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="n">kafka_group_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"sourcegroup1"</span><span class="p">,</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="n">kafka_topic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"test"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"json"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">json_config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"name=attr1;age=attr2"</span>
|
|
<span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_08_0239__li182916515910">Read Kafka topic <strong id="dli_08_0239__b2072918246212">test</strong>. The data is encoded in JSON format and nested. This example uses the complex data type ROW. For details about the syntax of ROW, see <a href="dli_08_0207.html">Data Type</a>.<p id="dli_08_0239__p8106173717013">The test data is as follows:</p>
|
|
<pre class="screen" id="dli_08_0239__screen8223320015">{
|
|
"id":"1",
|
|
"type2":"online",
|
|
"data":{
|
|
"patient_id":1234,
|
|
"name":"bob1234"
|
|
}
|
|
}</pre>
|
|
<div class="p" id="dli_08_0239__p6996123814015">An example of the table creation statements is as follows:<pre class="screen" id="dli_08_0239__screen170712713110">CREATE SOURCE STREAM kafka_source
|
|
(
|
|
id STRING,
|
|
type2 STRING,
|
|
<strong id="dli_08_0239__b19717153364815"> data ROW<</strong>
|
|
<strong id="dli_08_0239__b9719113324818"> patient_id STRING, </strong>
|
|
<strong id="dli_08_0239__b1872023364818"> name STRING></strong>
|
|
)
|
|
WITH (
|
|
type = "kafka",
|
|
kafka_bootstrap_servers = "ip1:port1,ip2:port2",
|
|
kafka_group_id = "sourcegroup1",
|
|
kafka_topic = "test",
|
|
<strong id="dli_08_0239__b117643812482">encode = "json"</strong>
|
|
);
|
|
|
|
CREATE SINK STREAM kafka_sink
|
|
(
|
|
id STRING,
|
|
type2 STRING,
|
|
patient_id STRING,
|
|
name STRING
|
|
)
|
|
WITH (
|
|
type="kafka",
|
|
kafka_bootstrap_servers = "ip1:port1,ip2:port2",
|
|
kafka_topic = "testsink",
|
|
encode = "csv"
|
|
);
|
|
|
|
INSERT INTO kafka_sink select id, type2, data.patient_id, data.name from kafka_source;</pre>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0234.html">Creating a Source Stream</a></div>
|
|
</div>
|
|
</div>
|
|
|