vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0226.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

183 lines
21 KiB
HTML
Raw Blame History

<a name="dli_08_0226"></a><a name="dli_08_0226"></a>
<h1 class="topictitle1">Creating a DLI Table and Associating It with DCS</h1>
<div id="body1578361312450"><div class="section" id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section540354616207"><h4 class="sectiontitle">Function</h4><p id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p13312652114">This statement is used to create a DLI table and associate it with an existing DCS key.</p>
<div class="note" id="dli_08_0226__note443995614391"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0226__dli_08_0119_p964618104214">In Spark cross-source development scenarios, there is a risk of password leakage if datasource authentication information is directly configured. You are advised to use the datasource authentication provided by DLI.</p>
</div></div>
</div>
<div class="section" id="dli_08_0226__section1886921516407"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_08_0226__p1976716256406">Before creating a DLI table and associating it with DCS, you need to create a datasource connection and bind it to a queue. For details about operations on the management console, see </p>
</div>
<div class="section" id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section20884135882017"><h4 class="sectiontitle">Syntax</h4><ul id="dli_08_0226__en-us_topic_0204105365_ul1667132219188"><li id="dli_08_0226__en-us_topic_0204105365_li11667142211813">Specified key<div class="codecoloring" codetype="Sql" id="dli_08_0226__en-us_topic_0204105365_screen13181194105218"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="p">[</span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="p">]</span><span class="w"> </span><span class="k">TABLE_NAME</span><span class="p">(</span>
<span class="w"> </span><span class="n">FIELDNAME1</span><span class="w"> </span><span class="n">FIELDTYPE1</span><span class="p">,</span>
<span class="w"> </span><span class="n">FIELDNAME2</span><span class="w"> </span><span class="n">FIELDTYPE2</span><span class="p">)</span>
<span class="w"> </span><span class="k">USING</span><span class="w"> </span><span class="n">REDIS</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'host'</span><span class="o">=</span><span class="s1">'xx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'port'</span><span class="o">=</span><span class="s1">'xx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'passwdauth'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xxx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'encryption'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'table'</span><span class="o">=</span><span class="s1">'namespace_in_redis:key_in_redis'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'key.column'</span><span class="o">=</span><span class="w"> </span><span class="s1">'FIELDNAME1'</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_08_0226__li996404212355">Wildcard key<div class="codecoloring" codetype="Sql" id="dli_08_0226__en-us_topic_0204105365_screen5917193011711"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="p">[</span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="p">]</span><span class="w"> </span><span class="k">TABLE_NAME</span><span class="p">(</span>
<span class="w"> </span><span class="n">FIELDNAME1</span><span class="w"> </span><span class="n">FIELDTYPE1</span><span class="p">,</span>
<span class="w"> </span><span class="n">FIELDNAME2</span><span class="w"> </span><span class="n">FIELDTYPE2</span><span class="p">)</span>
<span class="w"> </span><span class="k">USING</span><span class="w"> </span><span class="n">REDIS</span><span class="w"> </span><span class="k">OPTIONS</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'host'</span><span class="o">=</span><span class="s1">'xx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'port'</span><span class="o">=</span><span class="s1">'xx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'passwdauth'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xxx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'encryption'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'keys.pattern'</span><span class="o">=</span><span class="s1">'key*:*'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'key.column'</span><span class="o">=</span><span class="w"> </span><span class="s1">'FIELDNAME1'</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</div>
<div class="section" id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section784510261216"><h4 class="sectiontitle">Keywords</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_t2fa8597e0dde4884bd6412ac207ebea8" frame="border" border="1" rules="all"><caption><b>Table 1 </b>CREATE TABLE keywords</caption><thead align="left"><tr id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_r16c75fd8fdbf41c79f9aee82b032351c"><th align="left" class="cellrowborder" valign="top" width="18.75%" id="mcps1.3.4.2.2.3.1.1"><p id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_a9dd69379a3f54904a9c510fb817659ad"><strong id="dli_08_0226__b10443111204619">Parameter</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="81.25%" id="mcps1.3.4.2.2.3.1.2"><p id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_a7ede0d62539542baa7e367cf902bf2b2"><strong id="dli_08_0226__en-us_topic_0093946771_en-us_topic_0053447306_en-us_topic_0039551470_b6335010717346">Description</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row9789172211455"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p1783227111576">host</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_p73919172215">To connect to DCS, you need to create a datasource connection first. </p>
<p id="dli_08_0226__en-us_topic_0204105365_p14832153644513">After creating an enhanced datasource connection, use the connection address provided by DCS. If there are multiple connection addresses, select one of them. </p>
<div class="note" id="dli_08_0226__en-us_topic_0204105365_note4577403322"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_0226__en-us_topic_0204105365_p15851802326">Currently, only enhanced datasource is supported.</p>
</div></div>
</td>
</tr>
<tr id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row166137253465"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p584237211576">port</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p5902520811576">DCS connection port, for example, 6379.</p>
</td>
</tr>
<tr id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row193609262164"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p247311352164">password</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p570649132164">Password entered during DCS cluster creation. You do not need to set this parameter when accessing a non-secure Redis cluster.</p>
</td>
</tr>
<tr id="dli_08_0226__row2059017393302"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__p367053022018">passwdauth</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0151059555_p562616372455">Datasource password authentication name. For details about how to create datasource authentication, see Datasource Authentication in the <em id="dli_08_0226__i196971245163314">Data Lake Insight User Guide</em>.</p>
</td>
</tr>
<tr id="dli_08_0226__row6444134453015"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__p1073983512205">encryption</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__p13739113518204">Set this parameter to <span class="parmvalue" id="dli_08_0226__parmvalue68629349580"><b>true</b></span> when datasource password authentication is used.</p>
</td>
</tr>
<tr id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_row469853952167"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_p477206762167">table</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_p3413914173920">The key or hash key in Redis.</p>
<ul id="dli_08_0226__en-us_topic_0204105365_ul1647472215396"><li id="dli_08_0226__en-us_topic_0204105365_li1647415222396">This parameter is mandatory when Redis data is inserted.</li><li id="dli_08_0226__en-us_topic_0204105365_li5474622103915">Either this parameter or the <strong id="dli_08_0226__b1978622762310">keys.pattern</strong> parameter when Redis data is queried.</li></ul>
</td>
</tr>
<tr id="dli_08_0226__en-us_topic_0204105365_row463110345346"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_p1263223414340">keys.pattern</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_p19522133713392">Use a regular expression to match multiple keys or hash keys. This parameter is used only for query. Either this parameter or <strong id="dli_08_0226__b1983114334233">table</strong> is used to query Redis data.</p>
</td>
</tr>
<tr id="dli_08_0226__en-us_topic_0204105365_row760162013214"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_p760311201523">key.column</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_p26038205218">(Optional) Specifies a field in the schema as the key ID in Redis. This parameter is used together with the <strong id="dli_08_0226__b294715493238">table</strong> parameter when data is inserted.</p>
</td>
</tr>
<tr id="dli_08_0226__en-us_topic_0204105365_row10510329194412"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_p1051082954410">partitions.number</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_p4511122924416">Number of concurrent tasks during data reading.</p>
</td>
</tr>
<tr id="dli_08_0226__en-us_topic_0204105365_row17631032144411"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_p776343211447">scan.count</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_p6763232104413">Number of data records read in each batch. The default value is <strong id="dli_08_0226__b18349954122313">100</strong>. If the CPU usage of the Redis cluster still needs to be improved during data reading, increase the value of this parameter.</p>
</td>
</tr>
<tr id="dli_08_0226__en-us_topic_0204105365_row26964357449"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_p12696183564418">iterator.grouping.size</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_p469663534418">Number of data records inserted in each batch. The default value is <strong id="dli_08_0226__b187649598235">100</strong>. If the CPU usage of the Redis cluster still needs to be improved during the insertion, increase the value of this parameter.</p>
</td>
</tr>
<tr id="dli_08_0226__en-us_topic_0204105365_row13744175515298"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.3.1.1 "><p id="dli_08_0226__en-us_topic_0204105365_p474525520294">timeout</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.2.3.1.2 "><p id="dli_08_0226__en-us_topic_0204105365_p16745165592919">Timeout interval for connecting to the Redis, in milliseconds. The default value is <strong id="dli_08_0226__b17802039243">2000</strong> (2 seconds).</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="dli_08_0226__en-us_topic_0204105365_note731815316450"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0226__en-us_topic_0204105365_p1331916311457">When connecting to DCS, complex data types such as Array, Struct, and Map are not supported.</p>
<p id="dli_08_0226__en-us_topic_0204105365_p56435714611">The following methods can be used to process complex data:</p>
<ul id="dli_08_0226__en-us_topic_0204105365_ul9682134111498"><li id="dli_08_0226__en-us_topic_0204105365_li1268284116496">Place the fields of the next level in the Schema field of the same level.</li><li id="dli_08_0226__en-us_topic_0204105365_li10682154124916">Write and read data in binary mode, and encode and decode it using user-defined functions.</li></ul>
</div></div>
</div>
<div class="section" id="dli_08_0226__en-us_topic_0204105365_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_section9892337112213"><h4 class="sectiontitle">Example</h4><ul id="dli_08_0226__en-us_topic_0204105365_ul19160175410424"><li id="dli_08_0226__en-us_topic_0204105365_li13160195410429">Specifying a table</li></ul>
<div class="codecoloring" codetype="Sql" id="dli_08_0226__en-us_topic_0204105365_screen1630185635410"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">test_redis</span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">age</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span><span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">redis</span><span class="w"> </span><span class="k">options</span><span class="p">(</span>
<span class="w"> </span><span class="s1">'host'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'192.168.4.199'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'port'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'6379'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'passwdauth'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xxx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'encryption'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'table'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'person'</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<ul id="dli_08_0226__en-us_topic_0204105365_ul6387121612436"><li id="dli_08_0226__en-us_topic_0204105365_li03875163436">Wildcarding the table name</li></ul>
<div class="codecoloring" codetype="Sql" id="dli_08_0226__en-us_topic_0204105365_screen103961738164315"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">test_redis_keys_patten</span><span class="p">(</span><span class="n">id</span><span class="w"> </span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">age</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span><span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">redis</span><span class="w"> </span><span class="k">options</span><span class="p">(</span>
<span class="w"> </span><span class="s1">'host'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'192.168.4.199'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'port'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'6379'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'passwdauth'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'xxx'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'encryption'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'true'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'keys.pattern'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'p*:*'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'key.column'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'id'</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0225.html">Creating a Datasource Connection with a DCS Table</a></div>
</div>
</div>
View Git Blame Copy Permalink