vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0201.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

131 lines
18 KiB
HTML
Raw Blame History

<a name="dli_08_0201"></a><a name="dli_08_0201"></a>
<h1 class="topictitle1">Creating a DLI Table and Associating It with CSS</h1>
<div id="body1578361312450"><div class="section" id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section540354616207"><h4 class="sectiontitle">Function</h4><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p13312652114">This statement is used to create a DLI table and associate it with an existing CSS table.</p>
<div class="note" id="dli_08_0201__note443995614391"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0201__dli_08_0119_p964618104214">In Spark cross-source development scenarios, there is a risk of password leakage if datasource authentication information is directly configured. You are advised to use the datasource authentication provided by DLI.</p>
</div></div>
</div>
<div class="section" id="dli_08_0201__section1886921516407"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_08_0201__p1976716256406">Before creating a DLI table and associating it with CSS, you need to create a datasource connection. For details about operations on the management console, see </p>
</div>
<div class="section" id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section20884135882017"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0201__en-us_topic_0151059555_screen13181194105218"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="p">[</span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="p">]</span><span class="w"> </span><span class="k">TABLE_NAME</span><span class="p">(</span>
<span class="w"> </span><span class="n">FIELDNAME1</span><span class="w"> </span><span class="n">FIELDTYPE1</span><span class="p">,</span>
<span class="w"> </span><span class="n">FIELDNAME2</span><span class="w"> </span><span class="n">FIELDTYPE2</span><span class="p">)</span>
<span class="w"> </span><span class="k">USING</span><span class="w"> </span><span class="n">CSS</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'es.nodes'</span><span class="o">=</span><span class="s1">'xx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'resource'</span><span class="o">=</span><span class="s1">'type_path_in_CSS'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'pushdown'</span><span class="o">=</span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'strict'</span><span class="o">=</span><span class="s1">'false'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'batch.size.entries'</span><span class="o">=</span><span class="w"> </span><span class="s1">'1000'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'batch.size.bytes'</span><span class="o">=</span><span class="w"> </span><span class="s1">'1mb'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'es.nodes.wan.only'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'es.mapping.id'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'FIELDNAME'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section784510261216"><h4 class="sectiontitle">Keywords</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_t2fa8597e0dde4884bd6412ac207ebea8" frame="border" border="1" rules="all"><caption><b>Table 1 </b>CREATE TABLE keywords</caption><thead align="left"><tr id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_r16c75fd8fdbf41c79f9aee82b032351c"><th align="left" class="cellrowborder" valign="top" width="16.88%" id="mcps1.3.4.2.2.3.1.1"><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_a9dd69379a3f54904a9c510fb817659ad"><strong id="dli_08_0201__b1772284402810">Parameter</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="83.12%" id="mcps1.3.4.2.2.3.1.2"><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_a7ede0d62539542baa7e367cf902bf2b2"><strong id="dli_08_0201__en-us_topic_0093946771_en-us_topic_0053447306_en-us_topic_0039551470_b6335010717346">Description</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row9789172211455"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p1783227111576">es.nodes</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_p73919172215">Before obtaining the CSS IP address, you need to create a datasource connection first.. </p>
<p id="dli_08_0201__en-us_topic_0151059555_p14832153644513">If you have created an enhanced datasource connection, you can use the internal IP address provided by CSS. The format is <strong id="dli_08_0201__b15809175412285"><em id="dli_08_0201__i3807145414285">IP1</em>:<em id="dli_08_0201__i9808854162814">PORT1</em>,<em id="dli_08_0201__i8808854122811">IP2</em>:<em id="dli_08_0201__i138081754112811">PORT2</em></strong>.</p>
</td>
</tr>
<tr id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row166137253465"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p584237211576">resource</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p5902520811576">The <strong id="dli_08_0201__b265814573283">resource</strong> is used to specify the CSS datasource connection name. You can use <strong id="dli_08_0201__b86582573283">/index/type</strong> to specify the resource location (for easier understanding, the <strong id="dli_08_0201__b10658185712819">index</strong> can be seen as <strong id="dli_08_0201__b8659165711281">database</strong> and <strong id="dli_08_0201__b1065955772814">type</strong> as <strong id="dli_08_0201__b18660165722812">table</strong>).</p>
<div class="note" id="dli_08_0201__en-us_topic_0151059555_note2975123311388"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0201__en-us_topic_0151059555_ul7415352143219"><li id="dli_08_0201__en-us_topic_0151059555_li24168526326">In ES 6.X, a single index supports only one type, and the type name can be customized.</li><li id="dli_08_0201__en-us_topic_0151059555_li104169520329">In ES 7.X, a single index uses <strong id="dli_08_0201__b8483152518294">_doc</strong> as the type name and cannot be customized. To access ES 7.X, set this parameter to <strong id="dli_08_0201__b18200627172914">index</strong>.</li></ul>
</div></div>
</td>
</tr>
<tr id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row193609262164"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p247311352164">pushdown</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p570649132164">Indicates whether the press function of CSS is enabled. The default value is set to <strong id="dli_08_0201__b2083352819290">true</strong>. If there are a large number of I/O transfer tables, the <strong id="dli_08_0201__b1223843022911">pushdown</strong> can be enabled to reduce I/Os when the <strong id="dli_08_0201__b1623803012296">where</strong> filtering conditions are met.</p>
</td>
</tr>
<tr id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row469853952167"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p477206762167">strict</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p401695862167">Indicates whether the CSS <strong id="dli_08_0201__b084373222918">pushdown</strong> is strict. The default value is set to <strong id="dli_08_0201__b28441432132917">false</strong>. In exact match scenarios, more I/Os are reduced than <strong id="dli_08_0201__b132875344291">pushdown</strong>.</p>
</td>
</tr>
<tr id="dli_08_0201__en-us_topic_0151059555_row760162013214"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_p760311201523">batch.size.entries</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_p26038205218">Maximum number of entries that can be inserted to a batch processing. The default value is <strong id="dli_08_0201__b19555113618297">1000</strong>. If the size of a single data record is so large that the number of data records in the bulk storage reaches the upper limit of the data amount of a single batch processing, the system stops storing data and submits the data based on the <strong id="dli_08_0201__b033793917295">batch.size.bytes</strong>.</p>
</td>
</tr>
<tr id="dli_08_0201__en-us_topic_0151059555_row196031220821"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_p560319206215">batch.size.bytes</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_p960392015212">Maximum amount of data in a single batch processing. The default value is 1 MB. If the size of a single data record is so small that the number of data records in the bulk storage reaches the upper limit of the data amount of a single batch processing, the system stops storing data and submits the data based on the <strong id="dli_08_0201__b196741643152918">batch.size.entries</strong>.</p>
</td>
</tr>
<tr id="dli_08_0201__en-us_topic_0151059555_row33941161710"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_p23951216179">es.nodes.wan.only</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_p18395916773">Indicates whether to access the Elasticsearch node using only the domain name. The default value is <strong id="dli_08_0201__b11877194511291">false</strong>. If the original internal IP address provided by CSS is used as the <strong id="dli_08_0201__b7341248162917">es.nodes</strong>, you do not need to set this parameter or set it to <strong id="dli_08_0201__b53421948142920">false</strong>.</p>
</td>
</tr>
<tr id="dli_08_0201__en-us_topic_0151059555_row55101634143410"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_p1251117346341">es.mapping.id</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_p135116347348">Specifies a field whose value is used as the document ID in the Elasticsearch node.</p>
<div class="note" id="dli_08_0201__en-us_topic_0151059555_note20720112316195"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0201__en-us_topic_0151059555_ul72517457221"><li id="dli_08_0201__en-us_topic_0151059555_li172518457227">The document ID in the same <strong id="dli_08_0201__b13960151102913">/index/type</strong> is unique. If a field that functions as a document ID has duplicate values, the document with the duplicate ID will be overwritten when the ES is inserted.</li><li id="dli_08_0201__en-us_topic_0151059555_li1525164572214">This feature can be used as a fault tolerance solution. When data is being inserted, the DLI job fails and some data has been inserted into Elasticsearch. The data is redundant. If Document id is set, the last redundant data will be overwritten when the DLI job is executed again.</li></ul>
</div></div>
</td>
</tr>
<tr id="dli_08_0201__en-us_topic_0151059555_row76173411451"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_p1362113494517">es.net.ssl</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_p10621334194517">Whether to connect to the secure CSS cluster. The default value is <strong id="dli_08_0201__b139341468302">false</strong>.</p>
</td>
</tr>
<tr id="dli_08_0201__en-us_topic_0151059555_row46262372457"><td class="cellrowborder" valign="top" width="16.88%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0201__en-us_topic_0151059555_p13626153754515">es.certificate.name</p>
</td>
<td class="cellrowborder" valign="top" width="83.12%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0201__en-us_topic_0151059555_p562616372455">Name of the datasource authentication used to connect to the secure CSS cluster. For details about how to create datasource authentication, see Datasource Authentication in the <em id="dli_08_0201__i13279433310">Data Lake Insight User Guide</em>.</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="dli_08_0201__en-us_topic_0151059555_note625154520182"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0201__en-us_topic_0151059555_p172521456189"><strong id="dli_08_0201__b1714671520309">batch.size.entries</strong> and <strong id="dli_08_0201__b11146615153012">batch.size.bytes</strong> limit the number of data records and data volume respectively.</p>
</div></div>
</div>
<div class="section" id="dli_08_0201__en-us_topic_0151059555_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section9892337112213"><h4 class="sectiontitle">Example</h4><div class="codecoloring" codetype="Sql" id="dli_08_0201__en-us_topic_0151059555_screen1630185635410"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="n">dli_to_css</span><span class="w"> </span><span class="p">(</span><span class="n">doc_id</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">age</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span>
<span class="w"> </span><span class="k">USING</span><span class="w"> </span><span class="n">CSS</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">nodes</span><span class="w"> </span><span class="s1">'to-css-1174404703-LzwpJEyx.datasource.com:9200'</span><span class="p">,</span>
<span class="w"> </span><span class="n">resource</span><span class="w"> </span><span class="s1">'/dli_index/dli_type'</span><span class="p">,</span>
<span class="w"> </span><span class="n">pushdown</span><span class="w"> </span><span class="s1">'false'</span><span class="p">,</span>
<span class="w"> </span><span class="k">strict</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">nodes</span><span class="p">.</span><span class="n">wan</span><span class="p">.</span><span class="k">only</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">mapping</span><span class="p">.</span><span class="n">id</span><span class="w"> </span><span class="s1">'doc_id'</span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0200.html">Creating a Datasource Connection with a CSS Table</a></div>
</div>
</div>
View Git Blame Copy Permalink