forked from docs/doc-exports
Su, Xiaomeng
76a5b1ee83
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
178 lines
21 KiB
HTML
178 lines
21 KiB
HTML
<a name="dli_08_0252"></a><a name="dli_08_0252"></a>
|
|
|
|
<h1 class="topictitle1">CSS Elasticsearch Sink Stream</h1>
|
|
<div id="body1574393112904"><div class="section" id="dli_08_0252__en-us_topic_0111499976_section108631122164917"><h4 class="sectiontitle">Function</h4><p id="dli_08_0252__en-us_topic_0111499976_p8918102973511">DLI exports Flink job output data to Elasticsearch of Cloud Search Service (CSS). Elasticsearch is a popular enterprise-class Lucene-powered search server and provides the distributed multi-user capabilities. It delivers multiple functions, including full-text retrieval, structured search, analytics, aggregation, and highlighting. With Elasticsearch, you can achieve stable, reliable, real-time search. Elasticsearch applies to diversified scenarios, such as log analysis and site search.</p>
|
|
<p id="dli_08_0252__p1179007173419">CSS is a fully managed, distributed search service. It is fully compatible with open-source Elasticsearch and provides DLI with structured and unstructured data search, statistics, and report capabilities.</p>
|
|
<p id="dli_08_0252__en-us_topic_0111499976_p15813145563517"></p>
|
|
<p id="dli_08_0252__en-us_topic_0111499976_p6926133821315">For more information about CSS, see the <em id="dli_08_0252__en-us_topic_0111499976_i84235269710351">Cloud Search Service User Guide</em>.</p>
|
|
<div class="note" id="dli_08_0252__note1876217311841"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0252__p976216311449">If the security mode is enabled when you create a CSS cluster, it cannot be undone.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="dli_08_0252__en-us_topic_0111499976_section2579142713429"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0252__en-us_topic_0111499976_ul917413169425"><li id="dli_08_0252__en-us_topic_0111499976_li17192372456">Ensure that you have created a cluster on CSS using your account. For details about how to create a cluster on CSS, see <span class="filepath" id="dli_08_0252__en-us_topic_0111499976_filepath2020317864516"><b>Creating a Cluster</b></span> in the <i><cite id="dli_08_0252__cite1f3f835c3e9c4d1ba2ce036b50e688c7153342">Cloud Search Service User Guide</cite></i>.</li><li id="dli_08_0252__en-us_topic_0111499976_li141761416194218">In this scenario, jobs must run on the dedicated queue of DLI. Therefore, DLI must interconnect with the enhanced datasource connection that has been connected with CSS. You can also set the security group rules as required.<p id="dli_08_0252__p94091618103916"><a name="dli_08_0252__en-us_topic_0111499976_li141761416194218"></a><a name="en-us_topic_0111499976_li141761416194218"></a>For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_08_0252__dli_08_0249_en-us_topic_0132966325_filepath13726182511815"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_08_0252__dli_08_0249_cite534516e564224b13bad1de4471da3139181527">Data Lake Insight User Guide</cite></i>.</p>
|
|
<p id="dli_08_0252__p459216501155">For details about how to configure security group rules, see <span class="filepath" id="dli_08_0252__dli_08_0249_filepath78235157113"><b>Security Group</b></span> in the <em id="dli_08_0252__dli_08_0249_i082519153111">Virtual Private Cloud User Guide</em>.</p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0252__en-us_topic_0111499976_section3126105364419"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0252__screen1653016163010"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">stream_id</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"es"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">cluster_address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">es_index</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">es_type</span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">es_fields</span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">batch_insert_data_num</span><span class="o">=</span><span class="w"> </span><span class="ss">""</span>
|
|
<span class="w"> </span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
<p id="dli_08_0252__en-us_topic_0111499976_p332713141253"></p>
|
|
</div>
|
|
<div class="section" id="dli_08_0252__section538311333216"><h4 class="sectiontitle">Keywords</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0252__en-us_topic_0111499976_table73321814102516" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0252__en-us_topic_0111499976_row1934731413256"><th align="left" class="cellrowborder" valign="top" width="20.565656565656564%" id="mcps1.3.4.2.2.4.1.1"><p id="dli_08_0252__en-us_topic_0111499976_p163529142259">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="10.464646464646464%" id="mcps1.3.4.2.2.4.1.2"><p id="dli_08_0252__en-us_topic_0111499976_p17357514162518">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="68.96969696969697%" id="mcps1.3.4.2.2.4.1.3"><p id="dli_08_0252__en-us_topic_0111499976_p936111422510">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0252__en-us_topic_0111499976_row636719141253"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p4371161482515">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p7377121413259">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__en-us_topic_0111499976_p113827145254">Output channel type. <strong id="dli_08_0252__en-us_topic_0111499976_b842352706194212">es</strong> indicates that data is exported to CSS.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__en-us_topic_0111499976_row17383101422514"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p3389914152519">region</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p11394101472515">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__en-us_topic_0111499976_p17398414172520">Region where CSS is located.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__en-us_topic_0111499976_row9401214162511"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p640621410256">cluster_address</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p9412414172510">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__en-us_topic_0111499976_p114181414122512">Private access address of the CSS cluster, for example: <strong id="dli_08_0252__en-us_topic_0111499976_b5654132215220">x.x.x.x:x</strong>. Use commas (,) to separate multiple addresses.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__en-us_topic_0111499976_row1141921417251"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p342315149255">es_index</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p84296147257">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__p12502033103610">Index of the data to be inserted. This parameter corresponds to CSS index.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__en-us_topic_0111499976_row1643713148256"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p2044101412510">es_type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p24461714112511">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__en-us_topic_0111499976_p86361910994">Type of the document to which data is to be inserted. This parameter corresponds to the CSS type.</p>
|
|
<p id="dli_08_0252__p873844217223">If the Elasticsearch version is 6.x, the value cannot start with an underscore (_).</p>
|
|
<p id="dli_08_0252__p151463107238">If the Elasticsearch version is 7.x and the type of CSS is preset, the value must be <strong id="dli_08_0252__b1310483294714">_doc</strong>. Otherwise, the value must comply with CSS specifications.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__en-us_topic_0111499976_row134850141257"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p1490314132517">es_fields</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p10494111411256">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__en-us_topic_0111499976_p28045291899">Key of the data field to be inserted. The format is <strong id="dli_08_0252__b81381432164715">id,f1,f2,f3,f4</strong>. Ensure that the key corresponds to the data column in the sink. If a random attribute field instead of a key is used, the keyword <strong id="dli_08_0252__b5832644134711">id</strong> does not need to be used, for example, <strong id="dli_08_0252__b8796164817475">f1,f2,f3,f4,f5</strong>. This parameter corresponds to the CSS filed.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__en-us_topic_0111499976_row6438076811"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p4439371789">batch_insert_data_num</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p412192013816">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__en-us_topic_0111499976_p13440476817">Amount of data to be written in batches at a time. The value must be a positive integer. The unit is 10 records. The maximum value allowed is <strong id="dli_08_0252__b881075895419">65536</strong>, and the default value is <strong id="dli_08_0252__b3996101365519">10</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__en-us_topic_0111499976_row990612555417"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p390719256542">action</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p990722519546">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__en-us_topic_0111499976_p890717257543">If the value is <strong id="dli_08_0252__b141625234226">add</strong>, data is forcibly overwritten when the same ID is encountered. If the value is <strong id="dli_08_0252__b1916872372219">upsert</strong>, data is updated when the same ID is encountered. (If <strong id="dli_08_0252__b91681523162217">upsert</strong> is selected, <strong id="dli_08_0252__b116820230222">id</strong> in the <strong id="dli_08_0252__b18169122342214">es_fields</strong> field must be specified.) The default value is <strong id="dli_08_0252__b416942317228">add</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__en-us_topic_0111499976_row88749394151"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p487523912155">enable_output_null</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p14875139141519">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__en-us_topic_0111499976_p108761939131515">This parameter is used to configure whether to generate an empty field. If this parameter is set to <strong id="dli_08_0252__en-us_topic_0111501792_b12646129162811">true</strong>, an empty field (the value is <strong id="dli_08_0252__en-us_topic_0111501792_b866311115280">null</strong>) is generated. If set to <strong id="dli_08_0252__en-us_topic_0111501792_b9553184122814">false</strong>, no empty field is generated. The default value is <strong id="dli_08_0252__b17448103119226">false</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__en-us_topic_0111499976_row382415463343"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__en-us_topic_0111499976_p38255466342">max_record_num_cache</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__en-us_topic_0111499976_p7825194623414">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__en-us_topic_0111499976_p13826114617345">Maximum number of records that can be cached.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0252__row1791073612348"><td class="cellrowborder" valign="top" width="20.565656565656564%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0252__p99111936133419">es_certificate_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.464646464646464%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0252__p16911143663415">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="68.96969696969697%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0252__p8404109143917">Name of the datasource authentication information</p>
|
|
<p id="dli_08_0252__p571319488275">If the security mode is enabled and HTTPS is used by the Elasticsearch cluster, the certificate is required for access. In this case, set the datasource authentication type to <strong id="dli_08_0252__b101824717217">CSS</strong>.</p>
|
|
<p id="dli_08_0252__p4911143610347">If the security mode is enabled for the Elasticsearch cluster but HTTPS is disabled, the certificate and username and password are required for access. In this case, set the datasource authentication type to <strong id="dli_08_0252__b111751840104616">Password</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0252__section1485175513219"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0252__en-us_topic_0111499976_p277413624614">If a configuration item can be specified through parameter configurations, one or more columns in the record can be used as part of the configuration item. For example, if the configuration item is set to <strong id="dli_08_0252__b1940174382220">car_$ {car_brand}</strong> and the value of <strong id="dli_08_0252__b18416436225">car_brand</strong> in a record is <strong id="dli_08_0252__b941164317223">BMW</strong>, the value of this configuration item is <strong id="dli_08_0252__b1641164311225">car_BMW</strong> in the record.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0252__section085022413338"><h4 class="sectiontitle">Example</h4><div class="p" id="dli_08_0252__en-us_topic_0111499976_p152162533235">Data of stream <strong id="dli_08_0252__en-us_topic_0111499976_b842352706141932">qualified_cars</strong> is exported to the cluster on CSS.<div class="codecoloring" codetype="Sql" id="dli_08_0252__en-us_topic_0111499976_screen17542814142516"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span>
|
|
<span class="normal">16</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">qualified_cars</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_age</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">average_speed</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">total_miles</span><span class="w"> </span><span class="nb">INT</span>
|
|
<span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"es"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"xxx"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">cluster_address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"192.168.0.212:9200"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">es_index</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"car"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">es_type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"information"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">es_fields</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"id,owner,age,speed,miles"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">batch_insert_data_num</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"10"</span>
|
|
<span class="p">);</span><span class="w"> </span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0240.html">Creating a Sink Stream</a></div>
|
|
</div>
|
|
</div>
|
|
|