lboka/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity

dli_umn_20240227_version

Browse Source 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
This commit is contained in:
Su, Xiaomeng 2024-03-01 10:53:24 +00:00 committed by zuul
parent 82093fcb23
commit 2dbe7904be
69 changed files with 3634 additions and 502 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

105
docs/dli/umn/ALL_META.TXT.json
View File

File diff suppressed because it is too large Load Diff

42
docs/dli/umn/CLASS.TXT.json
View File

File diff suppressed because it is too large Load Diff

16
docs/dli/umn/dli_01_00006.html
View File

@ -8,7 +8,21 @@
</th>
</tr>
</thead>
<tbody><tr id="dli_01_00006__row1569106191216"><td class="cellrowborder" valign="top" width="33.67%" headers="mcps1.3.1.1.3.1.1 "><p id="dli_01_00006__p36566919122">2023-10-08</p>
<tbody><tr id="dli_01_00006__row4248101042"><td class="cellrowborder" valign="top" width="33.67%" headers="mcps1.3.1.1.3.1.1 "><p id="dli_01_00006__p14248407419">2024-02-27</p>
</td>
<td class="cellrowborder" valign="top" width="66.33%" headers="mcps1.3.1.1.3.1.2 "><p id="dli_01_00006__p752534712233">Added the following content:</p>
<ul id="dli_01_00006__ul1052694717232"><li id="dli_01_00006__li45262474232">Added Spark 3.1.1 dependencies to <a href="dli_01_0397.html">Built-in Dependencies</a>.</li></ul>
<p id="dli_01_00006__p593013411147">Modified the following section:</p>
<p id="dli_01_00006__p132481401644">Changed <strong id="dli_01_00006__b129917574313">DDS MongoDB</strong> to <strong id="dli_01_00006__b22823511410">DDS</strong> in <a href="dli_01_0457.html">Creating a Flink Jar Job</a>.</p>
</td>
</tr>
<tr id="dli_01_00006__row6758122510313"><td class="cellrowborder" valign="top" width="33.67%" headers="mcps1.3.1.1.3.1.1 "><p id="dli_01_00006__p462652810312">2023-11-01</p>
</td>
<td class="cellrowborder" valign="top" width="66.33%" headers="mcps1.3.1.1.3.1.2 "><p id="dli_01_00006__p462614287310">Modified the following content:</p>
<ul id="dli_01_00006__ul13626728339"><li id="dli_01_00006__li186261028531">Modified the link for obtaining service support during quota application in <a href="dli_07_0009.html">Quotas</a>.</li><li id="dli_01_00006__li26261928732">Added the link to <em id="dli_01_00006__i1661311132615">Data Lake Insight API Reference</em> to <a href="dli_01_0378.html">What Is Data Lake Insight?</a>.</li><li id="dli_01_00006__li14959163416518">Modified the method of obtaining host information in <a href="dli_01_0013.html">Modifying Host Information</a>.</li></ul>
</td>
</tr>
<tr id="dli_01_00006__row1569106191216"><td class="cellrowborder" valign="top" width="33.67%" headers="mcps1.3.1.1.3.1.1 "><p id="dli_01_00006__p36566919122">2023-10-08</p>
</td>
<td class="cellrowborder" valign="top" width="66.33%" headers="mcps1.3.1.1.3.1.2 "><p id="dli_01_00006__p465618951211">Modified the following content:</p>
<ul id="dli_01_00006__ul82951936181419"><li id="dli_01_00006__li129511362149">Modified the example format in <a href="dli_01_0384.html#dli_01_0384__table17326112613420">Table 2</a>.</li><li id="dli_01_00006__li3295163691419">Added constraints and limitations on tag keys and tag values to <a href="dli_01_0455.html">Creating a Flink SQL Job</a> and <a href="dli_01_0457.html">Creating a Flink Jar Job</a>.</li></ul>

7
docs/dli/umn/dli_01_0003.html
View File

@ -1,9 +1,12 @@
<a name="dli_01_0003"></a><a name="dli_01_0003"></a>
<h1 class="topictitle1">Overview</h1>
<div id="body0000001105195630"><div class="section" id="dli_01_0003__section96073483278"><h4 class="sectiontitle">What Is Enhanced Datasource Connection?</h4><p id="dli_01_0003__p22151355203913">Typically, you cannot use DLI to directly access a data source in a VPC other than the one where DLI is because the network between DLI and the data source is disconnected. For proper access, you need to establish a network connection between them.</p>
<p id="dli_01_0003__p1086631203616">DLI provides enhanced connections. Establishing a VPC peering connection allows DLI to communicate with the VPC of the data source, supporting cross-source data analysis.</p>
<div id="body0000001105195630"><div class="section" id="dli_01_0003__section96073483278"><h4 class="sectiontitle">What Is Enhanced Datasource Connection?</h4><p id="dli_01_0003__p522817397547">In cross-source data analysis scenarios, DLI needs to connect to external data sources. However, due to the different VPCs between the data source and DLI, the network cannot be connected, which results in DLI being unable to read data from the data source. DLI's enhanced datasource connection feature enables network connectivity between DLI and the data source.</p>
<p id="dli_01_0003__p712043744118">This section will introduce a solution for cross-VPC data source network connectivity:</p>
<ul id="dli_01_0003__ul080613266425"><li id="dli_01_0003__li4806192604220">Creating an enhanced datasource connection: Establish a VPC peering connection to connect DLI and the data source's VPC network.</li><li id="dli_01_0003__li8806226134218">Testing network connectivity: Verify the connectivity between the queue and the data source's network.</li></ul>
<p id="dli_01_0003__p5494121814320">For details about the data sources that support cross-source access, see <a href="dli_01_0410.html">Cross-Source Analysis Development Methods</a>.</p>
<div class="caution" id="dli_01_0003__note132261042151315"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><p id="dli_01_0003__p1822614217130">In cross-source development scenarios, there is a risk of password leakage if datasource authentication information is directly configured. You are advised to use the datasource authentication provided by DLI. For details, see <a href="dli_01_0561.html">Overview</a>.</p>
</div></div>
</div>
<div class="section" id="dli_01_0003__section12469945134416"><h4 class="sectiontitle">Constraints</h4><ul id="dli_01_0003__ul920917130139"><li id="dli_01_0003__dli_07_0005_li82094136131">Datasource connections cannot be created for the <strong id="dli_01_0003__dli_07_0005_b275994812408">default</strong> queue.</li><li id="dli_01_0003__dli_07_0005_li2599248131320">Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.</li><li id="dli_01_0003__dli_07_0005_li3320184019107"><strong id="dli_01_0003__dli_07_0005_b15858611104116">VPC Administrator</strong> permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.</li><li id="dli_01_0003__dli_07_0005_li196339467176">If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.</li><li id="dli_01_0003__dli_07_0005_li3209111310134">Only queues bound with datasource connections can access datasource tables.</li><li id="dli_01_0003__dli_07_0005_li1120931310136">Datasource tables do not support the preview function.</li><li id="dli_01_0003__dli_07_0005_li11700174614348">When checking the connectivity of datasource connections, the constraints on IP addresses are as follows:<ul id="dli_01_0003__dli_07_0005_ul1254904218358"><li id="dli_01_0003__dli_07_0005_li15478548367">The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.</li><li id="dli_01_0003__dli_07_0005_li151317527373">During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.<p id="dli_01_0003__dli_07_0005_p26801956133713"><a name="dli_01_0003__dli_07_0005_li151317527373"></a><a name="dli_07_0005_li151317527373"></a>For example, <strong id="dli_01_0003__dli_07_0005_b11869171511417">192.168.</strong><em id="dli_01_0003__dli_07_0005_i1917651964115">xx</em><strong id="dli_01_0003__dli_07_0005_b37501722194120">.</strong><em id="dli_01_0003__dli_07_0005_i151751326134116">xx</em> or <strong id="dli_01_0003__dli_07_0005_b4220103184115">192.168.</strong><em id="dli_01_0003__dli_07_0005_i7940153620413">xx</em><strong id="dli_01_0003__dli_07_0005_b11270124004111">.</strong><em id="dli_01_0003__dli_07_0005_i197261843124114">xx</em><strong id="dli_01_0003__dli_07_0005_b14999194774118">:8181</strong>.</p>
</li></ul>

12
docs/dli/umn/dli_01_0005.html
View File

@ -52,9 +52,9 @@
</p></li></ol>
</div>
<div class="section" id="dli_01_0005__section46221651175510"><h4 class="sectiontitle">Creating a Table</h4><p id="dli_01_0005__p56362480175534">Before creating a table, ensure that a database has been created.</p>
<ol id="dli_01_0005__ol12637600103738"><li id="dli_01_0005__li59972234153426"><span>You can create a table on either the <span class="wintitle" id="dli_01_0005__wintitle1434561990"><b>Databases and Tables</b></span> page or the <span class="wintitle" id="dli_01_0005__wintitle541971626"><b>SQL Editor</b></span> page.</span><p><div class="note" id="dli_01_0005__note159813598457"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0005__p4599175984510">Datasource connection tables, such as View tables, HBase (MRS) tables, OpenTSDB (MRS) tables, DWS tables, RDS tables, and CSS tables, cannot be created. You can use SQL to create views and datasource connection tables. For details, see sections <strong id="dli_01_0005__b168941558103516">Creating a View</strong> and <strong id="dli_01_0005__b1989435873520">Creating a Datasource Connection Table</strong> in the <em id="dli_01_0005__i12894358133518">Data Lake Insight SQL Syntax Reference</em>.</p>
<ol id="dli_01_0005__ol12637600103738"><li id="dli_01_0005__li59972234153426"><span>You can create a table on either the <span class="wintitle" id="dli_01_0005__wintitle280732842"><b>Databases and Tables</b></span> page or the <span class="wintitle" id="dli_01_0005__wintitle1052158451"><b>SQL Editor</b></span> page.</span><p><div class="note" id="dli_01_0005__note159813598457"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0005__p4599175984510">Datasource connection tables, such as View tables, HBase (MRS) tables, OpenTSDB (MRS) tables, DWS tables, RDS tables, and CSS tables, cannot be created. You can use SQL to create views and datasource connection tables. For details, see sections <strong id="dli_01_0005__b168941558103516">Creating a View</strong> and <strong id="dli_01_0005__b1989435873520">Creating a Datasource Connection Table</strong> in the <em id="dli_01_0005__i12894358133518">Data Lake Insight SQL Syntax Reference</em>.</p>
</div></div>
<ul id="dli_01_0005__ul22405571153537"><li id="dli_01_0005__li323555153537">To create a table on the <span class="wintitle" id="dli_01_0005__wintitle640209786"><b>Data Management</b></span> page:<ol type="a" id="dli_01_0005__ol2911995153537"><li id="dli_01_0005__li26207957153537">On the left of the management console, choose <span class="uicontrol" id="dli_01_0005__uicontrol1559686145"><b>Data Management</b></span> &gt; <span class="uicontrol" id="dli_01_0005__uicontrol8597462415"><b>Databases and Tables</b></span>.</li><li id="dli_01_0005__li17500832153537">On the <strong id="dli_01_0005__b1211154675314">Databases and Tables</strong> page, select the database for which you want to create a table. In the <strong id="dli_01_0005__b10345811102717">Operation</strong> column, click <strong id="dli_01_0005__b12918326165019">More</strong> &gt; <strong id="dli_01_0005__b63873822719">Create Table</strong> to create a table in the current database.</li></ol>
<ul id="dli_01_0005__ul22405571153537"><li id="dli_01_0005__li323555153537">To create a table on the <span class="wintitle" id="dli_01_0005__wintitle412318792"><b>Data Management</b></span> page:<ol type="a" id="dli_01_0005__ol2911995153537"><li id="dli_01_0005__li26207957153537">On the left of the management console, choose <span class="uicontrol" id="dli_01_0005__uicontrol1559686145"><b>Data Management</b></span> &gt; <span class="uicontrol" id="dli_01_0005__uicontrol8597462415"><b>Databases and Tables</b></span>.</li><li id="dli_01_0005__li17500832153537">On the <strong id="dli_01_0005__b1211154675314">Databases and Tables</strong> page, select the database for which you want to create a table. In the <strong id="dli_01_0005__b10345811102717">Operation</strong> column, click <strong id="dli_01_0005__b12918326165019">More</strong> &gt; <strong id="dli_01_0005__b63873822719">Create Table</strong> to create a table in the current database.</li></ol>
</li><li id="dli_01_0005__li8281288153537">To create a table on the <span class="wintitle" id="dli_01_0005__wintitle8700317181735"><b>SQL Editor</b></span> page:<ol type="a" id="dli_01_0005__ol7422733153537"><li id="dli_01_0005__li66804605153537">On the left of the management console, click <span class="uicontrol" id="dli_01_0005__uicontrol11208161745"><b>SQL Editor</b></span>.</li><li id="dli_01_0005__li17512154252514">In the navigation pane of the displayed <span class="wintitle" id="dli_01_0005__wintitle13614162313185"><b>SQL Editor</b></span> page, click <strong id="dli_01_0005__b4615162311183">Databases</strong>. You can create a table in either of the following ways:<ul id="dli_01_0005__ul782474512255"><li id="dli_01_0005__li1893119202619">Click a database name. In the <strong id="dli_01_0005__b11582159133716">Tables</strong> area, click <span><img id="dli_01_0005__image15644111617506" src="en-us_image_0237539075.png"></span> on the right to create a table in the current database.</li><li id="dli_01_0005__li16824154518252">Click <span><img id="dli_01_0005__image1322473935113" src="en-us_image_0237532018.png"></span> on the right of the database and choose <strong id="dli_01_0005__b135378142190">Create Table</strong> from the shortcut menu to create a table in the current database.</li></ul>
</li></ol>
</li></ul>
@ -108,7 +108,7 @@
<tr id="dli_01_0005__row16943758105944"><td class="cellrowborder" valign="top" width="14.38%" headers="mcps1.3.4.3.2.2.1.2.5.2.4.1.1 "><p id="dli_01_0005__p30267119105944">Type</p>
</td>
<td class="cellrowborder" valign="top" width="71.45%" headers="mcps1.3.4.3.2.2.1.2.5.2.4.1.2 "><p id="dli_01_0005__p35717569105944">Data type of a column. This parameter corresponds to <span class="parmname" id="dli_01_0005__parmname4155795592938"><b>Column Name</b></span>.</p>
<ul id="dli_01_0005__ul2957726293032"><li id="dli_01_0005__li159154616209"><strong id="dli_01_0005__b12861523131117">string</strong>: The data is of the string type.</li><li id="dli_01_0005__li5035776893042"><strong id="dli_01_0005__b84235270617190">int</strong>: Each integer is stored on four bytes.</li><li id="dli_01_0005__li197172662015"><strong id="dli_01_0005__b71123713114">date</strong>: The value ranges from 0000-01-01 to 9999-12-31.</li><li id="dli_01_0005__li15276123414204"><strong id="dli_01_0005__b2073283811110">double</strong>: Each number is stored on eight bytes.</li><li id="dli_01_0005__li122191918102110"><strong id="dli_01_0005__b87406418113">boolean</strong>: Each value is stored on one byte.</li><li id="dli_01_0005__li616884422117"><strong id="dli_01_0005__b842352706172140">decimal</strong>: The valid bits are positive integers between 1 to 38, including 1 and 38. The decimal digits are integers less than 10.</li><li id="dli_01_0005__li6039945193127"><strong id="dli_01_0005__b948204713">smallint/short</strong>: The number is stored on two bytes.</li><li id="dli_01_0005__li157282298387"><strong id="dli_01_0005__b1165212357">bigint/long</strong>: The number is stored on eight bytes.</li><li id="dli_01_0005__li4930135112217"><strong id="dli_01_0005__b1251694319111">timestamp</strong>: The data indicates a date and time. The value can be accurate to six decimal points.</li><li id="dli_01_0005__li482517952312"><strong id="dli_01_0005__b161442457112">float</strong>: Each number is stored on four bytes.</li><li id="dli_01_0005__li246310417404"><strong id="dli_01_0005__b0368141417399">tinyint</strong>: Each number is stored on one byte. Only OBS tables support this data type.</li></ul>
<ul id="dli_01_0005__ul2957726293032"><li id="dli_01_0005__li159154616209"><strong id="dli_01_0005__b12861523131117">string</strong>: The data is of the string type.</li><li id="dli_01_0005__li5035776893042"><strong id="dli_01_0005__b84235270617190">int</strong>: Each integer is stored on four bytes.</li><li id="dli_01_0005__li197172662015"><strong id="dli_01_0005__b71123713114">date</strong>: The value ranges from 0000-01-01 to 9999-12-31.</li><li id="dli_01_0005__li15276123414204"><strong id="dli_01_0005__b2073283811110">double</strong>: Each number is stored on eight bytes.</li><li id="dli_01_0005__li122191918102110"><strong id="dli_01_0005__b87406418113">boolean</strong>: Each value is stored on one byte.</li><li id="dli_01_0005__li616884422117"><strong id="dli_01_0005__b842352706172140">decimal</strong>: The valid bits are positive integers between 1 to 38, including 1 and 38. The decimal digits are integers less than 10.</li><li id="dli_01_0005__li6039945193127"><strong id="dli_01_0005__b1597562473">smallint/short</strong>: The number is stored on two bytes.</li><li id="dli_01_0005__li157282298387"><strong id="dli_01_0005__b1496238909">bigint/long</strong>: The number is stored on eight bytes.</li><li id="dli_01_0005__li4930135112217"><strong id="dli_01_0005__b1251694319111">timestamp</strong>: The data indicates a date and time. The value can be accurate to six decimal points.</li><li id="dli_01_0005__li482517952312"><strong id="dli_01_0005__b161442457112">float</strong>: Each number is stored on four bytes.</li><li id="dli_01_0005__li246310417404"><strong id="dli_01_0005__b0368141417399">tinyint</strong>: Each number is stored on one byte. Only OBS tables support this data type.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="14.17%" headers="mcps1.3.4.3.2.2.1.2.5.2.4.1.3 "><p id="dli_01_0005__p7441963105944">string</p>
</td>
@ -144,7 +144,7 @@
<tbody><tr id="dli_01_0005__row3015013618314"><td class="cellrowborder" valign="top" width="16.6%" headers="mcps1.3.4.3.2.2.1.2.6.2.4.1.1 "><p id="dli_01_0005__p3709008118455">Data Format</p>
</td>
<td class="cellrowborder" valign="top" width="53.06999999999999%" headers="mcps1.3.4.3.2.2.1.2.6.2.4.1.2 "><p id="dli_01_0005__p5150660918455">DLI supports the following data formats:</p>
<ul id="dli_01_0005__ul4173603518455"><li id="dli_01_0005__li4007999518455"><strong id="dli_01_0005__b84235270614338">Parquet</strong>: DLI can read non-compressed data or data that is compressed using Snappy and gzip.</li><li id="dli_01_0005__li2517563618455"><strong id="dli_01_0005__b608475975">CSV</strong>: DLI can read non-compressed data or data that is compressed using gzip.</li><li id="dli_01_0005__li2525413518455"><strong id="dli_01_0005__b2027485322">ORC</strong>: DLI can read non-compressed data or data that is compressed using Snappy.</li><li id="dli_01_0005__li2596063018455"><strong id="dli_01_0005__b986683361">JSON</strong>: DLI can read non-compressed data or data that is compressed using gzip.</li><li id="dli_01_0005__li20203144172320"><strong id="dli_01_0005__b14575133533814">Avro</strong>: DLI can read uncompressed Avro data.</li></ul>
<ul id="dli_01_0005__ul4173603518455"><li id="dli_01_0005__li4007999518455"><strong id="dli_01_0005__b84235270614338">Parquet</strong>: DLI can read non-compressed data or data that is compressed using Snappy and gzip.</li><li id="dli_01_0005__li2517563618455"><strong id="dli_01_0005__b1449184344">CSV</strong>: DLI can read non-compressed data or data that is compressed using gzip.</li><li id="dli_01_0005__li2525413518455"><strong id="dli_01_0005__b68145053">ORC</strong>: DLI can read non-compressed data or data that is compressed using Snappy.</li><li id="dli_01_0005__li2596063018455"><strong id="dli_01_0005__b1694317934">JSON</strong>: DLI can read non-compressed data or data that is compressed using gzip.</li><li id="dli_01_0005__li20203144172320"><strong id="dli_01_0005__b14575133533814">Avro</strong>: DLI can read uncompressed Avro data.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="30.330000000000002%" headers="mcps1.3.4.3.2.2.1.2.6.2.4.1.3 "><p id="dli_01_0005__p2243627218455">CSV</p>
</td>
@ -177,7 +177,7 @@
</tr>
<tr id="dli_01_0005__row21739275111826"><td class="cellrowborder" valign="top" width="16.6%" headers="mcps1.3.4.3.2.2.1.2.6.2.4.1.1 "><p id="dli_01_0005__p39917803111836">User-defined Quotation Character</p>
</td>
<td class="cellrowborder" valign="top" width="53.06999999999999%" headers="mcps1.3.4.3.2.2.1.2.6.2.4.1.2 "><p id="dli_01_0005__p056181715520">This parameter is valid only when <span class="parmname" id="dli_01_0005__parmname1498421651"><b>Data Format</b></span> is set to <span class="parmvalue" id="dli_01_0005__parmvalue1075709676"><b>CSV</b></span> and you select <strong id="dli_01_0005__b50884634">User-defined Quotation Character</strong>.</p>
<td class="cellrowborder" valign="top" width="53.06999999999999%" headers="mcps1.3.4.3.2.2.1.2.6.2.4.1.2 "><p id="dli_01_0005__p056181715520">This parameter is valid only when <span class="parmname" id="dli_01_0005__parmname502358884"><b>Data Format</b></span> is set to <span class="parmvalue" id="dli_01_0005__parmvalue1064295601"><b>CSV</b></span> and you select <strong id="dli_01_0005__b885854190">User-defined Quotation Character</strong>.</p>
<p id="dli_01_0005__p57404199154711">The following quotation characters are supported:</p>
<ul id="dli_01_0005__ul41920423111836"><li id="dli_01_0005__li41739488111836">Single quotation mark (')</li><li id="dli_01_0005__li40111075111836">Double quotation marks (")</li><li id="dli_01_0005__li25455362111836">Others: Enter a user-defined quotation character.</li></ul>
</td>
@ -213,7 +213,7 @@
</table>
</div>
</li></ul>
</p></li><li id="dli_01_0005__li6436679692218"><span>Click <span class="uicontrol" id="dli_01_0005__uicontrol1346028235151847"><b>OK</b></span>.</span><p><p id="dli_01_0005__p2255887315231">After a table is created, you can view and select the table for use on the <span class="wintitle" id="dli_01_0005__wintitle1490820795"><b>Data Management</b></span> page or <strong id="dli_01_0005__b1705654393">SQL Editor</strong> page.</p>
</p></li><li id="dli_01_0005__li6436679692218"><span>Click <span class="uicontrol" id="dli_01_0005__uicontrol1346028235151847"><b>OK</b></span>.</span><p><p id="dli_01_0005__p2255887315231">After a table is created, you can view and select the table for use on the <span class="wintitle" id="dli_01_0005__wintitle1684585740"><b>Data Management</b></span> page or <strong id="dli_01_0005__b555872049">SQL Editor</strong> page.</p>
</p></li><li id="dli_01_0005__li66082750104919"><span>(Optional) After a DLI table is created, you can decide whether to directly import data to the table.</span></li></ol>
</div>
</div>

2
docs/dli/umn/dli_01_0012.html
View File

@ -20,7 +20,7 @@
</li>
<li class="ulchildlink"><strong><a href="dli_01_0489.html">Testing Address Connectivity</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_01_0421.html">Creating a Message Notification Topic</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_01_0421.html">Creating an SMN Topic</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_01_0022.html">Managing Queue Tags</a></strong><br>
</li>

1
docs/dli/umn/dli_01_0013.html
View File

@ -20,7 +20,6 @@
</li><li id="dli_01_0013__li914799171217"><strong id="dli_01_0013__b1392713419255">Method 2: Obtain MRS host information from the /etc/hosts file on an MRS node.</strong><ol id="dli_01_0013__ol207699227017"><li id="dli_01_0013__li67691922306">Log in to any MRS node as user <strong id="dli_01_0013__b187181123279">root</strong>.</li><li id="dli_01_0013__li17769172215019">Run the following command to obtain MRS hosts information. Copy and save the information.<p id="dli_01_0013__p16131162771410"><a name="dli_01_0013__li17769172215019"></a><a name="li17769172215019"></a><strong id="dli_01_0013__b9843629161411">cat /etc/hosts</strong></p>
<div class="fignone" id="dli_01_0013__fig15752030165413"><span class="figcap"><b>Figure 1 </b>Obtaining hosts information</span><br><span><img id="dli_01_0013__image4830201818144" src="en-us_image_0000001586217017.png"></span></div>
</li><li id="dli_01_0013__li1176911221002">Modify host information by referring to <a href="#dli_01_0013__section6859216954">Modifying Host Information</a>.</li></ol>
</li><li id="dli_01_0013__li014759141210"><strong id="dli_01_0013__b565525112718">Method 3: Log in to FusionInsight Manager to obtain host information.</strong><ol id="dli_01_0013__ol3769417115"><li id="dli_01_0013__li676915171813">Log in to FusionInsight Manager.</li><li id="dli_01_0013__li177698179113">On FusionInsight Manager, click <strong id="dli_01_0013__b126271239102310">Hosts</strong>. On the <strong id="dli_01_0013__b7556143492419">Hosts</strong> page, obtain the host names and service IP addresses of the MRS hosts.</li><li id="dli_01_0013__li146541436234">Modify host information by referring to <a href="#dli_01_0013__section6859216954">Modifying Host Information</a>.</li></ol>
</li></ul>
</div>
</div>

9
docs/dli/umn/dli_01_0017.html
View File

@ -56,11 +56,10 @@
</li><li id="dli_01_0017__li821851916016"><strong id="dli_01_0017__b8637165181412">Re-execute</strong>: Execute the job again.</li><li id="dli_01_0017__li12875164791814"><strong id="dli_01_0017__b10242547150">SparkUI</strong>: Display the Spark job execution page.<div class="note" id="dli_01_0017__note151024818191"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_01_0017__ul55667208198"><li id="dli_01_0017__li956713205198">When you execute a job on a created queue, the cluster is restarted. It takes about 10 minutes. If you click <strong id="dli_01_0017__b1274243225511">SparkUI</strong> before the cluster is created, an empty <strong id="dli_01_0017__b12742203215512">projectID</strong> will be cached. The SparkUI page cannot be displayed. You are advised to use a dedicated queue so that the cluster will not be released. Alternatively, wait for a while after the job is submitted (the cluster is created), and then check <strong id="dli_01_0017__b1381493675517">SparkUI</strong>.</li><li id="dli_01_0017__li1359102371912">Currently, only the latest 100 job information records are displayed on the SparkUI of DLI.</li></ul>
</div></div>
</li></ul>
<ul id="dli_01_0017__ul1875134719182"><li id="dli_01_0017__li3883129255">In addition to the preceding operations, the following operations are available for QUERY jobs and asynchronous DDL jobs.<ul id="dli_01_0017__ul14392109145620"><li id="dli_01_0017__li38468323121"><strong id="dli_01_0017__b53531757153820">View Result</strong>: View the job running result.</li><li id="dli_01_0017__li37817583123"><strong id="dli_01_0017__b55953114394">Export Result</strong>: Export the job running result to the created OBS bucket. For details, see <a href="#dli_01_0017__section1152211221244">Exporting Query Results</a>.</li></ul>
<ul id="dli_01_0017__ul1875134719182"><li id="dli_01_0017__li3883129255">In addition to the preceding operations, the following operations are available for QUERY jobs and asynchronous DDL jobs.<ul id="dli_01_0017__ul14392109145620"><li id="dli_01_0017__li38468323121"><strong id="dli_01_0017__b53531757153820">View Result</strong>: View the job running result.</li><li id="dli_01_0017__li37817583123"><strong id="dli_01_0017__b55953114394">Export Result</strong>: Export the job running result to the created OBS bucket.</li></ul>
</li><li id="dli_01_0017__li98421757121616">In addition to the preceding operations, the EXPORT job also includes the following operations:<ul id="dli_01_0017__ul1384275714169"><li id="dli_01_0017__li1684214571164"><strong id="dli_01_0017__b1972253413916">Download</strong></li></ul>
</li><li id="dli_01_0017__li16809489135"><strong id="dli_01_0017__b674619324407">View Log</strong>: Save job logs to the temporary OBS bucket created by DLI.</li><li id="dli_01_0017__li113222516517"><strong id="dli_01_0017__b5645835124010">Export Log</strong>: Export logs to the created OBS bucket. If the job is in the <strong id="dli_01_0017__b19611619124114">Running</strong> state, logs cannot be exported.</li></ul>
<div class="note" id="dli_01_0017__note9317379552"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0017__p1633103716553">To export the logs, you need to obtain the permission to create an OBS bucket.</p>
<p id="dli_01_0017__p663010482567">Log archiving and export are not available for synchronization jobs and jobs running on the default queue.</p>
</li><li id="dli_01_0017__li16809489135"><strong id="dli_01_0017__b674619324407">View Log</strong>: Save job logs to the temporary OBS bucket created by DLI.</li></ul>
<div class="note" id="dli_01_0017__note9317379552"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0017__p663010482567">The <strong id="dli_01_0017__b329102520461">View Log</strong> button is not available for synchronization jobs and jobs running on the default queue.</p>
</div></div>
</td>
</tr>
@ -81,7 +80,7 @@
</div>
<div class="section" id="dli_01_0017__section1152211221244"><a name="dli_01_0017__section1152211221244"></a><a name="section1152211221244"></a><h4 class="sectiontitle">Exporting Query Results</h4><p id="dli_01_0017__p9458163282918">A maximum of 1000 records can be displayed in the query result on the console. To view more or all data, you can export the data to OBS. The procedure is as follows:</p>
<p id="dli_01_0017__p11388428204">You can export results on the <span class="wintitle" id="dli_01_0017__wintitle2135957101316"><b>SQL Jobs</b></span> page or the <span class="wintitle" id="dli_01_0017__wintitle213685711310"><b>SQL Editor</b></span> page.</p>
<ul id="dli_01_0017__ul8360118142113"><li id="dli_01_0017__li83604822117">On the <strong id="dli_01_0017__b11515441261">Job Management</strong> &gt; <span class="wintitle" id="dli_01_0017__wintitle4552143819619"><b>SQL Jobs</b></span> page, you can click <strong id="dli_01_0017__b7545122131411">More</strong> &gt; <span class="uicontrol" id="dli_01_0017__uicontrol05461025146"><b>Export Result</b></span> in the <strong id="dli_01_0017__b105461216146">Operation</strong> column to export the query result.</li><li id="dli_01_0017__li4267191916211">After the query statements are successfully executed on the <span class="wintitle" id="dli_01_0017__wintitle3789726191611"><b>SQL Editor</b></span> page, click <span><img id="dli_01_0017__image1255155017385" src="en-us_image_0000001209516352.png"></span> on the right of the <span class="wintitle" id="dli_01_0017__wintitle979062681614"><b>View Result</b></span> tab page to export the query result.</li></ul>
<ul id="dli_01_0017__ul8360118142113"><li id="dli_01_0017__li83604822117">On the <strong id="dli_01_0017__b11515441261">Job Management</strong> &gt; <span class="wintitle" id="dli_01_0017__wintitle4552143819619"><b>SQL Jobs</b></span> page, you can click <strong id="dli_01_0017__b7545122131411">More</strong> &gt; <span class="uicontrol" id="dli_01_0017__uicontrol05461025146"><b>Export Result</b></span> in the <strong id="dli_01_0017__b105461216146">Operation</strong> column to export the query result.</li><li id="dli_01_0017__li4267191916211">After the query statements are successfully executed on the <span class="wintitle" id="dli_01_0017__wintitle3789726191611"><b>SQL Editor</b></span> page, click <span><img id="dli_01_0017__image1412107111917" src="en-us_image_0000001835430521.png"></span> next to the <span class="wintitle" id="dli_01_0017__wintitle979062681614"><b>View Result</b></span> tab page to export query results.</li></ul>
<div class="note" id="dli_01_0017__note1284717218412"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0017__p108591921114113">If no column of the numeric type is displayed in the query result, the result cannot be exported.</p>
</div></div>
<p id="dli_01_0017__p811117120448"></p>

4
docs/dli/umn/dli_01_0318.html
View File

@ -109,14 +109,14 @@
<td class="cellrowborder" valign="top" width="32.5%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_01_0318__p172317326537">queueActions</p>
</td>
</tr>
<tr id="dli_01_0318__row4776205111511"><td class="cellrowborder" valign="top" width="35.010000000000005%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_01_0318__p18967174814312">Submitting a job</p>
<tr id="dli_01_0318__row4776205111511"><td class="cellrowborder" valign="top" width="35.010000000000005%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_01_0318__p18967174814312">Submitting a job (SQL)</p>
</td>
<td class="cellrowborder" valign="top" width="32.49%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_01_0318__p1996774853120">queue</p>
</td>
<td class="cellrowborder" valign="top" width="32.5%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_01_0318__p87233326537">submitJob</p>
</td>
</tr>
<tr id="dli_01_0318__row23816498017"><td class="cellrowborder" valign="top" width="35.010000000000005%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_01_0318__p188703111119">Canceling a job</p>
<tr id="dli_01_0318__row23816498017"><td class="cellrowborder" valign="top" width="35.010000000000005%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_01_0318__p188703111119">Canceling a job (SQL)</p>
</td>
<td class="cellrowborder" valign="top" width="32.49%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_01_0318__p28711214117">queue</p>
</td>

55
docs/dli/umn/dli_01_0320.html
View File

File diff suppressed because it is too large Load Diff

34
docs/dli/umn/dli_01_0363.html
View File

@ -2,55 +2,54 @@
<h1 class="topictitle1">Creating a Queue</h1>
<div id="body1501038213757"><p id="dli_01_0363__p1731419482089">Before executing a job, you need to create a queue.</p>
<div class="note" id="dli_01_0363__note1293204515206"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_01_0363__ul17341756521"><li id="dli_01_0363__li1174985419502">If you use a sub-account to create a queue for the first time, log in to the DLI management console using the main account and keep records in the DLI database before creating a queue.</li><li id="dli_01_0363__li1373411525216">It takes 6 to 10 minutes for a job running on a new queue for the first time.</li><li id="dli_01_0363__li11343161015521">After a queue is created, if no job is run within one hour, the system releases the queue.</li></ul>
<div class="note" id="dli_01_0363__note1293204515206"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_01_0363__ul17341756521"><li id="dli_01_0363__li1174985419502">If you use a sub-account to create a queue for the first time, log in to the DLI management console using the main account and keep records in the DLI database before creating a queue.</li><li id="dli_01_0363__li1373411525216">It takes 6 to 10 minutes for a job running on a new queue for the first time.</li><li id="dli_01_0363__li11343161015521">After a queue is created, if no job is run within one hour, the system releases the queue.</li><li id="dli_01_0363__li253742810428">Queues with 16 CUs do not support scale-out or scale-in.</li><li id="dli_01_0363__li6119202221710">Queues with 64 CUs do not support scale-in.</li></ul>
</div></div>
<div class="section" id="dli_01_0363__section14223343145314"><h4 class="sectiontitle">Procedure</h4><ol id="dli_01_0363__ol24716195296"><li id="dli_01_0363__li947220195291">You can create a queue on the <span class="wintitle" id="dli_01_0363__wintitle686912613315"><b>Overview</b></span>, <span class="wintitle" id="dli_01_0363__wintitle20869192623118"><b>SQL Editor</b></span>, or <span class="wintitle" id="dli_01_0363__wintitle18870626173120"><b>Queue Management</b></span> page.<ul id="dli_01_0363__ul435871517496"><li id="dli_01_0363__li1359201518491">In the upper right corner of the <strong id="dli_01_0363__b3686177172119">Overview</strong> page, click Create Queue.</li><li id="dli_01_0363__li8634112214911">To create a queue on the <span class="wintitle" id="dli_01_0363__wintitle1782046172416"><b>Queue Management</b></span> page:<ol type="a" id="dli_01_0363__ol1592205194915"><li id="dli_01_0363__li1059116511492">In the navigation pane of the DLI management console, choose <strong id="dli_01_0363__b15623755102515">Resources</strong> &gt;<span class="uicontrol" id="dli_01_0363__uicontrol1712082421418"><b>Queue Management</b></span>.</li><li id="dli_01_0363__li759275184915">In the upper right corner of the <span class="wintitle" id="dli_01_0363__wintitle1629419173214"><b>Queue Management</b></span> page, click <strong id="dli_01_0363__b152951717182111">Create Queue</strong> to create a queue.</li></ol>
</li><li id="dli_01_0363__li10613182974914">To create a queue on the <span class="wintitle" id="dli_01_0363__wintitle18993133744419"><b>SQL Editor</b></span> page:<ol type="a" id="dli_01_0363__ol864692284913"><li id="dli_01_0363__li13646152224918">In the navigation pane of the DLI management console, click <span class="uicontrol" id="dli_01_0363__uicontrol1996404019456"><b>SQL Editor</b></span>.</li><li id="dli_01_0363__li1864613225498">Click <strong id="dli_01_0363__b39030518577">Queues</strong>. On the tab page displayed, click <span><img id="dli_01_0363__image654613242326" src="en-us_image_0237406526.png"></span> on the right to create a queue.</li></ol>
</li></ul>
</li><li id="dli_01_0363__li1472111916293">In the displayed Create Queue dialog box, set related parameters by referring to <a href="#dli_01_0363__table17301125219910">Table 1</a>.<div class="p" id="dli_01_0363__p206522012141215">
<div class="tablenoborder"><a name="dli_01_0363__table17301125219910"></a><a name="table17301125219910"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0363__table17301125219910" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="dli_01_0363__row112973522913"><th align="left" class="cellrowborder" valign="top" width="14.77%" id="mcps1.3.3.2.2.2.1.2.3.1.1"><p id="dli_01_0363__p17297145215918">Parameter</p>
</li><li id="dli_01_0363__li1472111916293">On the <strong id="dli_01_0363__b1674113163515">Create Queue</strong> page displayed, set the parameters according to <a href="#dli_01_0363__table17301125219910">Table 1</a>.
<div class="tablenoborder"><a name="dli_01_0363__table17301125219910"></a><a name="table17301125219910"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0363__table17301125219910" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="dli_01_0363__row112973522913"><th align="left" class="cellrowborder" valign="top" width="14.77%" id="mcps1.3.3.2.2.3.2.3.1.1"><p id="dli_01_0363__p17297145215918">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="85.22999999999999%" id="mcps1.3.3.2.2.2.1.2.3.1.2"><p id="dli_01_0363__p1229775212915">Description</p>
<th align="left" class="cellrowborder" valign="top" width="85.22999999999999%" id="mcps1.3.3.2.2.3.2.3.1.2"><p id="dli_01_0363__p1229775212915">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_01_0363__row132972521295"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="dli_01_0363__p029718521495">Name</p>
<tbody><tr id="dli_01_0363__row132972521295"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.3.2.3.1.1 "><p id="dli_01_0363__p029718521495">Name</p>
</td>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="dli_01_0363__p192978528910">Name of a queue.</p>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.3.2.3.1.2 "><p id="dli_01_0363__p192978528910">Name of a queue.</p>
<ul id="dli_01_0363__ul429710528910"><li id="dli_01_0363__li162973521797">The queue name can contain only digits, letters, and underscores (_), but cannot contain only digits, start with an underscore (_), or be left unspecified.</li><li id="dli_01_0363__li12978521191">The length of the name cannot exceed 128 characters.</li></ul>
<div class="note" id="dli_01_0363__note13297452697"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0363__p192979529919">The queue name is case-insensitive. Uppercase letters will be automatically converted to lowercase letters.</p>
</div></div>
</td>
</tr>
<tr id="dli_01_0363__row82981752094"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="dli_01_0363__p629715213917">Type</p>
<tr id="dli_01_0363__row82981752094"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.3.2.3.1.1 "><p id="dli_01_0363__p629715213917">Type</p>
</td>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><ul id="dli_01_0363__ul1029875217914"><li id="dli_01_0363__li1229745218914"><strong id="dli_01_0363__b3471630195419">For SQL</strong>: compute resources used for SQL jobs.</li><li id="dli_01_0363__li1829818521296"><strong id="dli_01_0363__b8738194132314">For general purpose</strong>: compute resources used for Spark and Flink jobs.<div class="note" id="dli_01_0363__note17601813184718"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0363__p1440855224617">Selecting <strong id="dli_01_0363__b4152202311213">Dedicated Resource Mode</strong> enables you to create a dedicated queue. Enhanced datasource connections can only be created for dedicated queues. </p>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.3.2.3.1.2 "><ul id="dli_01_0363__ul1029875217914"><li id="dli_01_0363__li1229745218914"><strong id="dli_01_0363__b3471630195419">For SQL</strong>: compute resources used for SQL jobs.</li><li id="dli_01_0363__li1829818521296"><strong id="dli_01_0363__b8738194132314">For general purpose</strong>: compute resources used for Spark and Flink jobs.<div class="note" id="dli_01_0363__note17601813184718"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0363__p1440855224617">Selecting <strong id="dli_01_0363__b4152202311213">Dedicated Resource Mode</strong> enables you to create a dedicated queue. Enhanced datasource connections can only be created for dedicated queues. </p>
</div></div>
</li></ul>
</td>
</tr>
<tr id="dli_01_0363__row202991052199"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="dli_01_0363__p132987521911">Specifications</p>
<tr id="dli_01_0363__row202991052199"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.3.2.3.1.1 "><p id="dli_01_0363__p132987521911">Specifications</p>
</td>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="dli_01_0363__p13298105216917">Select queue specifications as required. A CU includes one core and 4 GB memory. You can set the total number of CUs on all compute nodes of a queue. DLI automatically allocates the memory and vCPUs for each node.</p>
<ul id="dli_01_0363__ul92992521991"><li id="dli_01_0363__li529919522917">Fixed specifications include <span class="parmvalue" id="dli_01_0363__parmvalue12849285516"><b>16 CUs</b></span>, <span class="parmvalue" id="dli_01_0363__parmvalue1859916711558"><b>64 CUs</b></span>, <span class="parmvalue" id="dli_01_0363__parmvalue947752245514"><b>256 CUs</b></span>, and <span class="parmvalue" id="dli_01_0363__parmvalue2109152775515"><b>512 CUs</b></span>.</li><li id="dli_01_0363__li16299852795"><strong id="dli_01_0363__b195120381321">Custom</strong>: Set the specifications as required.</li></ul>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.3.2.3.1.2 "><p id="dli_01_0363__p1992094115406">The compute nodes' total number of CUs. One CU equals one vCPU and 4 GB of memory. DLI automatically assigns CPU and memory resources to each compute node, and the client does not need to know how many compute nodes are being used.</p>
</td>
</tr>
<tr id="dli_01_0363__row62996521099"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="dli_01_0363__p1129916526920">Description</p>
<tr id="dli_01_0363__row62996521099"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.3.2.3.1.1 "><p id="dli_01_0363__p1129916526920">Description</p>
</td>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="dli_01_0363__p192991252199">Description of the queue to be created. The description can contain a maximum of 128 characters.</p>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.3.2.3.1.2 "><p id="dli_01_0363__p192991252199">Description of the queue to be created. The description can contain a maximum of 128 characters.</p>
</td>
</tr>
<tr id="dli_01_0363__row1530012524917"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="dli_01_0363__p3299452791">Advanced Settings</p>
<tr id="dli_01_0363__row1530012524917"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.3.2.3.1.1 "><p id="dli_01_0363__p3299452791">Advanced Settings</p>
</td>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><div class="p" id="dli_01_0363__p0300185215911">In the <span class="parmname" id="dli_01_0363__parmname12909114468"><b>Queue Type</b></span> area, select <span class="parmvalue" id="dli_01_0363__parmvalue639181216240"><b>Dedicated Resource Mode</b></span> and then click <strong id="dli_01_0363__b152775425810">Advanced Settings</strong>.<ul id="dli_01_0363__ul4300452895"><li id="dli_01_0363__li829905210913"><strong id="dli_01_0363__b10386158155714">Default</strong>: The system automatically configures the parameter.</li><li id="dli_01_0363__li14300852692"><strong id="dli_01_0363__b5653124465512">Custom</strong><p id="dli_01_0363__p1730085219912"><span class="parmname" id="dli_01_0363__parmname146519343320"><b>CIDR Block</b></span>: You can specify the CIDR block. For details, see <a href="dli_01_0443.html">Modifying the Queue CIDR Block</a>. If DLI enhanced datasource connection is used, the CIDR block of the DLI queue cannot overlap with that of the data source.</p>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.3.2.3.1.2 "><div class="p" id="dli_01_0363__p0300185215911">In the <span class="parmname" id="dli_01_0363__parmname12909114468"><b>Queue Type</b></span> area, select <span class="parmvalue" id="dli_01_0363__parmvalue639181216240"><b>Dedicated Resource Mode</b></span> and then click <strong id="dli_01_0363__b152775425810">Advanced Settings</strong>.<ul id="dli_01_0363__ul4300452895"><li id="dli_01_0363__li829905210913"><strong id="dli_01_0363__b10386158155714">Default</strong>: The system automatically configures the parameter.</li><li id="dli_01_0363__li14300852692"><strong id="dli_01_0363__b5653124465512">Custom</strong><p id="dli_01_0363__p1730085219912"><span class="parmname" id="dli_01_0363__parmname146519343320"><b>CIDR Block</b></span>: You can specify the CIDR block. For details, see <a href="dli_01_0443.html">Modifying the CIDR Block</a>. If DLI enhanced datasource connection is used, the CIDR block of the DLI queue cannot overlap with that of the data source.</p>
<p id="dli_01_0363__p143001521498"><span class="parmname" id="dli_01_0363__parmname371585753911"><b>Queue Type</b></span>: When running an AI-related SQL job, select <span class="parmvalue" id="dli_01_0363__parmvalue74576129408"><b>AI-enhanced</b></span>. When running other jobs, select <span class="parmvalue" id="dli_01_0363__parmvalue124431427194019"><b>Basic</b></span>.</p>
</li></ul>
</div>
</td>
</tr>
<tr id="dli_01_0363__row113007527911"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.2.1.2.3.1.1 "><p id="dli_01_0363__p13300252599">Tag</p>
<tr id="dli_01_0363__row113007527911"><td class="cellrowborder" valign="top" width="14.77%" headers="mcps1.3.3.2.2.3.2.3.1.1 "><p id="dli_01_0363__p13300252599">Tags</p>
</td>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.2.1.2.3.1.2 "><p id="dli_01_0363__p7662154145213">Tags used to identify cloud resources. A tag includes the tag key and tag value. If you want to use the same tag to identify multiple cloud resources, that is, to select the same tag from the drop-down list box for all services, you are advised to create predefined tags on the Tag Management Service (TMS).</p>
<td class="cellrowborder" valign="top" width="85.22999999999999%" headers="mcps1.3.3.2.2.3.2.3.1.2 "><p id="dli_01_0363__p7662154145213">Tags used to identify cloud resources. A tag includes the tag key and tag value. If you want to use the same tag to identify multiple cloud resources, that is, to select the same tag from the drop-down list box for all services, you are advised to create predefined tags on the Tag Management Service (TMS).</p>
<div class="note" id="dli_01_0363__note15300105218914"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_01_0363__ul1830018521911"><li id="dli_01_0363__li186815313365">A maximum of 20 tags can be added.</li><li id="dli_01_0363__li183008521591">Only one tag value can be added to a tag key.</li><li id="dli_01_0363__li14484164661614">The key name in each resource must be unique.</li></ul>
</div></div>
<ul id="dli_01_0363__ul1300052697"><li id="dli_01_0363__li5300952598">Tag key: Enter a tag key name in the text box.<div class="note" id="dli_01_0363__note206961611105313"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0363__p1448010306508">A tag key can contain a maximum of 128 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with <strong id="dli_01_0363__b544594615499">_sys_</strong>.</p>
@ -63,7 +62,6 @@
</tbody>
</table>
</div>
</div>
</li><li id="dli_01_0363__li647261910292">Click <span class="uicontrol" id="dli_01_0363__uicontrol19947846102516"><b>Create Now</b></span> to create a queue.<p id="dli_01_0363__p1543112413517">After a queue is created, you can view and select the queue for use on the <span class="wintitle" id="dli_01_0363__wintitle64862717152748"><b>Queue Management</b></span> page.</p>
<div class="note" id="dli_01_0363__note731816483410"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0363__p193203483416">It takes 6 to 10 minutes for a job running on a new queue for the first time.</p>
</div></div>

11
docs/dli/umn/dli_01_0378.html
View File

@ -5,9 +5,9 @@
<p id="dli_01_0378__p35691490142221">DLI supports standard SQL and is compatible with Spark SQL and Flink SQL. It also supports multiple access modes, and is compatible with mainstream data formats. DLI supports SQL statements and Spark applications for heterogeneous data sources, including CloudTable, RDS, GaussDB(DWS), CSS, OBS, custom databases on ECSs, and offline databases.</p>
</div>
<div class="section" id="dli_01_0378__section189215415182"><h4 class="sectiontitle">Functions</h4><p id="dli_01_0378__p3872545124817">You can query and analyze heterogeneous data sources such as RDS, and GaussDB(DWS) on the cloud using access methods, such as visualized interface, RESTful API, JDBC, and Beeline. The data format is compatible with five mainstream data formats: CSV, JSON, Parquet, and ORC.</p>
<ul id="dli_01_0378__ul95252494910"><li id="dli_01_0378__li37017520555">Basic functions<ul id="dli_01_0378__ul33508814556"><li id="dli_01_0378__li4884112014444">You can use standard SQL statements to query in SQL jobs. For details, see .</li><li id="dli_01_0378__li583064563216">Flink jobs support Flink SQL online analysis. Aggregation functions such as Window and Join, geographic functions, and CEP functions are supported. SQL is used to express service logic, facilitating service implementation. For details, see .</li><li id="dli_01_0378__li652124194912">For spark jobs, fully-managed Spark computing can be performed. You can submit computing tasks through interactive sessions or in batch to analyze data in the fully managed Spark queues. For details, see .</li></ul>
</li><li id="dli_01_0378__li13705141165713">Federated analysis of heterogeneous data sources<ul id="dli_01_0378__ul14754404579"><li id="dli_01_0378__li68911658193117">Spark datasource connection: Data sources such as DWS, RDS, and CSS can be accessed through DLI. For details, see .</li><li id="dli_01_0378__li9919175613471">Interconnection with multiple cloud services is supported in Flink jobs to form a rich stream ecosystem. The DLI stream ecosystem consists of cloud service ecosystems and open source ecosystems.<ul id="dli_01_0378__ul4978181016481"><li id="dli_01_0378__li10978151011487">Cloud service ecosystem: DLI can interconnect with other services in Flink SQL. You can directly use SQL to read and write data from cloud services.</li><li id="dli_01_0378__li4978610154816">Open-source ecosystems: After connections to other VPCs are established through datasource connections, you can access all data sources and output targets (such as Kafka, HBase, and Elasticsearch) supported by Flink and Spark in your dedicated DLI queue.</li></ul>
<p id="dli_01_0378__p11978131024820">For details, see .</p>
<ul id="dli_01_0378__ul95252494910"><li id="dli_01_0378__li37017520555">Basic functions<ul id="dli_01_0378__ul33508814556"><li id="dli_01_0378__li4884112014444">You can use standard SQL statements to query in SQL jobs. </li><li id="dli_01_0378__li583064563216">Flink jobs support Flink SQL online analysis. Aggregation functions such as Window and Join, geographic functions, and CEP functions are supported. SQL is used to express service logic, facilitating service implementation. </li><li id="dli_01_0378__li652124194912">For spark jobs, fully-managed Spark computing can be performed. You can submit computing tasks through interactive sessions or in batch to analyze data in the fully managed Spark queues. </li></ul>
</li><li id="dli_01_0378__li13705141165713">Federated analysis of heterogeneous data sources<ul id="dli_01_0378__ul14754404579"><li id="dli_01_0378__li68911658193117">Spark datasource connection: Data sources such as DWS, RDS, and CSS can be accessed through DLI. </li><li id="dli_01_0378__li9919175613471">Interconnection with multiple cloud services is supported in Flink jobs to form a rich stream ecosystem. The DLI stream ecosystem consists of cloud service ecosystems and open source ecosystems.<ul id="dli_01_0378__ul4978181016481"><li id="dli_01_0378__li10978151011487">Cloud service ecosystem: DLI can interconnect with other services in Flink SQL. You can directly use SQL to read and write data from cloud services.</li><li id="dli_01_0378__li4978610154816">Open-source ecosystems: After connections to other VPCs are established through datasource connections, you can access all data sources and output targets (such as Kafka, HBase, and Elasticsearch) supported by Flink and Spark in your dedicated DLI queue.</li></ul>
<p id="dli_01_0378__p11978131024820"></p>
</li></ul>
</li><li id="dli_01_0378__li620462561618">Storage-compute decoupling<p id="dli_01_0378__p12236173712168"><a name="dli_01_0378__li620462561618"></a><a name="li620462561618"></a>DLI is interconnected with OBS for data analysis. In this architecture where storage and compute are decoupled, resources of these two types are charged separately, helping you reduce costs and improving resource utilization.</p>
<p id="dli_01_0378__p656615493515">You can choose single-AZ or multi-AZ storage when you create an OBS bucket for storing redundant data on the DLI console. The differences between the two storage policies are as follows:</p>
@ -20,10 +20,11 @@
<ul id="dli_01_0378__ul156203189145"><li id="dli_01_0378__li166201818121418">Auto scaling: DLI ensures you always have enough capacity on hand to deal with any traffic spikes.</li></ul>
</div>
<div class="section" id="dli_01_0378__section173909441109"><h4 class="sectiontitle">Accessing DLI</h4><p id="dli_01_0378__p8366328105017">A web-based service management platform is provided. You can access DLI using the management console or HTTPS-based APIs, or connect to the DLI server through the JDBC client.</p>
<ul id="dli_01_0378__ul0613194011284"><li id="dli_01_0378__li172844319285">Using the management console<p id="dli_01_0378__p928643102819"><a name="dli_01_0378__li172844319285"></a><a name="li172844319285"></a>You can submit SQL, Spark, or Flink jobs on the DLI management console. Log in to the management console. Choose <strong id="dli_01_0378__b459287151313">EI Enterprise Intelligence</strong> &gt; <strong id="dli_01_0378__b1317953119138">Data Lake Insight</strong>.</p>
<ul id="dli_01_0378__ul0613194011284"><li id="dli_01_0378__li172844319285">Using the management console<p id="dli_01_0378__p6440171193615"><a name="dli_01_0378__li172844319285"></a><a name="li172844319285"></a>You can submit SQL, Spark, or Flink jobs on the DLI management console.</p>
<p id="dli_01_0378__p165819306367">Log in to the management console and choose <strong id="dli_01_0378__b164061349397">Data Analysis</strong> &gt; <strong id="dli_01_0378__b104713541898">Data Lake Insight</strong>.</p>
</li></ul>
<ul id="dli_01_0378__ul6602184155118"><li id="dli_01_0378__li17602645513">Using APIs<p id="dli_01_0378__p886223541010"><a name="dli_01_0378__li17602645513"></a><a name="li17602645513"></a>If you need to integrate DLI into a third-party system for secondary development, you can call DLI APIs to use the service.</p>
<p id="dli_01_0378__p1932395211161">For details, see <em id="dli_01_0378__i641914274913">Data Lake Insight API Reference</em>.</p>
<p id="dli_01_0378__p129419363381">For details, see <a href="https://docs.otc.t-systems.com/data-lake-insight/api-ref/" target="_blank" rel="noopener noreferrer">Data Lake Insight API Reference</a>.</p>
</li></ul>
</div>
</div>

32
docs/dli/umn/dli_01_0384.html
View File

@ -5,7 +5,7 @@
<p id="dli_01_0384__p55886519497">On the <strong id="dli_01_0384__b943884293511">Overview</strong> page, click <strong id="dli_01_0384__b202117331159">Create Job</strong> in the upper right corner of the <strong id="dli_01_0384__b1696101133814">Spark Jobs</strong> tab or click <strong id="dli_01_0384__b56361152153720">Create Job</strong> in the upper right corner of the <strong id="dli_01_0384__b993313572052">Spark Jobs</strong> page. The Spark job editing page is displayed.</p>
<p id="dli_01_0384__p9427817122315">On the Spark job editing page, a message is displayed, indicating that a temporary DLI data bucket will be created. The created bucket is used to store temporary data generated by DLI, such as job logs and job results. You cannot view job logs if you choose not to create it. The bucket will be created and the default bucket name is used.</p>
<p id="dli_01_0384__p133447114312">If you do not need to create a DLI temporary data bucket and do not want to receive this message, select <strong id="dli_01_0384__b142701146121612">Do not show again</strong> and click <span class="uicontrol" id="dli_01_0384__uicontrol227014464167"><b>Cancel</b></span>.</p>
<div class="section" id="dli_01_0384__section1661873674317"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_01_0384__ul9494144631313"><li id="dli_01_0384__li749524681310">You have uploaded the dependencies to the corresponding OBS bucket on the <span class="wintitle" id="dli_01_0384__wintitle1216872122520"><b>Data Management &gt; Package Management</b></span> page. For details, see <a href="dli_01_0367.html">Creating a Package</a>.</li><li id="dli_01_0384__li7520181309">Before creating a Spark job to access other external data sources, such as OpenTSDB, HBase, Kafka, GaussDB(DWS), RDS, CSS, CloudTable, DCS Redis, and DDS MongoDB, you need to create a cross-source connection to enable the network between the job running queue and external data sources.<ul id="dli_01_0384__ul850415199816"><li id="dli_01_0384__li18451528175">For details about the external data sources that can be accessed by Spark jobs, see <a href="dli_01_0410.html">Cross-Source Analysis Development Methods</a>.</li><li id="dli_01_0384__li28469131945">For details about how to create a datasource connection, see <a href="dli_01_0426.html">Enhanced Datasource Connections</a>.<p id="dli_01_0384__p881561411413">On the <strong id="dli_01_0384__b122201520155014">Resources</strong> &gt; <strong id="dli_01_0384__b1022002035016">Queue Management</strong> page, locate the queue you have created, and choose <strong id="dli_01_0384__b112219202504">More</strong> &gt; <strong id="dli_01_0384__b0221182013507">Test Address Connectivity</strong> in the <strong id="dli_01_0384__b422192015507">Operation</strong> column to check whether the network connection between the queue and the data source is normal. For details, see <a href="dli_01_0489.html">Testing Address Connectivity</a>.</p>
<div class="section" id="dli_01_0384__section1661873674317"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_01_0384__ul9494144631313"><li id="dli_01_0384__li749524681310">You have uploaded the dependencies to the corresponding OBS bucket on the <span class="wintitle" id="dli_01_0384__wintitle1216872122520"><b>Data Management &gt; Package Management</b></span> page. For details, see <a href="dli_01_0367.html">Creating a Package</a>.</li><li id="dli_01_0384__li7520181309">Before creating a Spark job to access other external data sources, such as OpenTSDB, HBase, Kafka, GaussDB(DWS), RDS, CSS, CloudTable, DCS Redis, and DDS, you need to create a cross-source connection to enable the network between the job running queue and external data sources.<ul id="dli_01_0384__ul850415199816"><li id="dli_01_0384__li18451528175">For details about the external data sources that can be accessed by Spark jobs, see <a href="dli_01_0410.html">Cross-Source Analysis Development Methods</a>.</li><li id="dli_01_0384__li28469131945">For details about how to create a datasource connection, see <a href="dli_01_0426.html">Enhanced Datasource Connections</a>.<p id="dli_01_0384__p881561411413">On the <strong id="dli_01_0384__b122201520155014">Resources</strong> &gt; <strong id="dli_01_0384__b1022002035016">Queue Management</strong> page, locate the queue you have created, and choose <strong id="dli_01_0384__b112219202504">More</strong> &gt; <strong id="dli_01_0384__b0221182013507">Test Address Connectivity</strong> in the <strong id="dli_01_0384__b422192015507">Operation</strong> column to check whether the network connection between the queue and the data source is normal. For details, see <a href="dli_01_0489.html">Testing Address Connectivity</a>.</p>
</li></ul>
</li></ul>
</div>
@ -28,6 +28,7 @@
<tr id="dli_01_0384__row12481951165010"><td class="cellrowborder" valign="top" width="17.34%" headers="mcps1.3.6.3.2.2.1.2.3.1.1 "><p id="dli_01_0384__p52486516509">Application</p>
</td>
<td class="cellrowborder" valign="top" width="82.66%" headers="mcps1.3.6.3.2.2.1.2.3.1.2 "><p id="dli_01_0384__p14248351155014">Select the package to be executed. The value can be <strong id="dli_01_0384__b151323445615">.jar</strong> or <strong id="dli_01_0384__b1275553713561">.py</strong>.</p>
<p id="dli_01_0384__p14448114151112">You can select the name of a JAR or pyFile package that has been uploaded to the DLI resource management system. You can also specify an OBS path, for example, <strong id="dli_01_0384__b878815345588">obs://</strong><em id="dli_01_0384__i1789534205815">Bucket name</em><strong id="dli_01_0384__b3789534125813">/</strong><em id="dli_01_0384__i479014347589">Package name</em>.</p>
</td>
</tr>
<tr id="dli_01_0384__row82481351105015"><td class="cellrowborder" valign="top" width="17.34%" headers="mcps1.3.6.3.2.2.1.2.3.1.1 "><p id="dli_01_0384__p324815185016">Main Class (--class)</p>
@ -38,8 +39,8 @@
<tr id="dli_01_0384__row824935175016"><td class="cellrowborder" valign="top" width="17.34%" headers="mcps1.3.6.3.2.2.1.2.3.1.1 "><p id="dli_01_0384__p142481951125016">Spark Arguments (--conf)</p>
</td>
<td class="cellrowborder" valign="top" width="82.66%" headers="mcps1.3.6.3.2.2.1.2.3.1.2 "><p id="dli_01_0384__p1024855155014">Enter a parameter in the format of <strong id="dli_01_0384__b525425345812">key=value</strong>. Press Enter to separate multiple key-value pairs. </p>
<p id="dli_01_0384__p182493511506">These parameters can be replaced using global variables. For example, if you create a global variable <strong id="dli_01_0384__b1464915212517">custom_class</strong> on the <strong id="dli_01_0384__b136551024510">Global Configuration</strong> &gt;<strong id="dli_01_0384__b116551626511"> Global Variables</strong> page, you can use <strong id="dli_01_0384__b0655121656">"spark.sql.catalog"={{custom_class}}</strong> to replace a parameter with this variable after the job is submitted.</p>
<div class="note" id="dli_01_0384__note82491051205017"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_01_0384__ul1417784810398"><li id="dli_01_0384__li12177248163917">The JVM garbage collection algorithm cannot be customized for Spark jobs.</li><li id="dli_01_0384__li11177154815393">If the Spark version is <strong id="dli_01_0384__b194034425294">3.1.1</strong>, configure <strong id="dli_01_0384__b642172473011">Spark parameters (--conf)</strong> to select a dependent module. For details about the configuration, see <a href="dli_01_0384.html">Creating a Spark Job</a>.</li></ul>
<p id="dli_01_0384__p182493511506">These parameters can be replaced with global variables. For example, if you create a global variable <strong id="dli_01_0384__b1464915212517">custom_class</strong> on the <strong id="dli_01_0384__b136551024510">Global Configuration</strong> &gt;<strong id="dli_01_0384__b116551626511"> Global Variables</strong> page, you can use <strong id="dli_01_0384__b0655121656">"spark.sql.catalog"={{custom_class}}</strong> to replace a parameter with this variable after the job is submitted.</p>
<div class="note" id="dli_01_0384__note82491051205017"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_01_0384__ul1417784810398"><li id="dli_01_0384__li12177248163917">The JVM garbage collection algorithm cannot be customized for Spark jobs.</li><li id="dli_01_0384__li11177154815393">If the Spark version is <strong id="dli_01_0384__b194034425294">3.1.1</strong>, configure <strong id="dli_01_0384__b642172473011">Spark parameters (--conf)</strong> to select a dependent module. For details about the example configuration, see <a href="#dli_01_0384__table17326112613420">Table 2</a>.</li></ul>
</div></div>
</td>
</tr>
@ -137,7 +138,14 @@
</tbody>
</table>
</div>
</li><li id="dli_01_0384__li15887141135316">Set the following parameters in advanced settings:<ul id="dli_01_0384__ul794310285537"><li id="dli_01_0384__li14943102814534"><strong id="dli_01_0384__b3256165333218">Select Dependency Resources</strong>: For details about the parameters, see <a href="#dli_01_0384__table6886411195318">Table 3</a>.</li><li id="dli_01_0384__li394332811535"><strong id="dli_01_0384__b58225553324">Configure Resources</strong>: For details about the parameters, see <a href="#dli_01_0384__table388781175311">Table 4</a>.</li></ul>
</li><li id="dli_01_0384__li15887141135316">Set the following parameters in advanced settings:<ul id="dli_01_0384__ul794310285537"><li id="dli_01_0384__li14943102814534"><strong id="dli_01_0384__b3256165333218">Select Dependency Resources</strong>: For details about the parameters, see <a href="#dli_01_0384__table6886411195318">Table 3</a>.</li><li id="dli_01_0384__li394332811535"><strong id="dli_01_0384__b58225553324">Configure Resources</strong>: For details about the parameters, see <a href="#dli_01_0384__table388781175311">Table 4</a>.<div class="note" id="dli_01_0384__note11887101110530"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0384__p51163218518">The parallelism degree of Spark resources is jointly determined by the number of Executors and the number of Executor CPU cores.</p>
<p id="dli_01_0384__p19642104334"><strong id="dli_01_0384__b137602175517">Maximum number of tasks that can be concurrently executed = Number of Executors x Number of Executor CPU cores</strong></p>
<p id="dli_01_0384__p250541481918">You can properly plan compute resource specifications based on the compute CUs of the queue you have purchased.</p>
<p id="dli_01_0384__p12159122010332">Note that Spark tasks need to be jointly executed by multiple roles, such as <strong id="dli_01_0384__b32881436782">driver</strong> and <strong id="dli_01_0384__b0437738984">executor</strong>. So, the number of executors multiplied by the number of executor CPU cores must be less than the number of compute CUs of the queue to prevent other roles from failing to start Spark tasks. For more information about roles for Spark tasks, see <a href="https://spark.apache.org/" target="_blank" rel="noopener noreferrer">Apache Spark</a>.</p>
<p id="dli_01_0384__p1264244293516"><strong id="dli_01_0384__b1617711315362">Calculation formula for Spark job parameters:</strong></p>
<ul id="dli_01_0384__ul14642144253520"><li id="dli_01_0384__li18642442123512">CUs = Driver Cores + Executors x Executor Cores</li><li id="dli_01_0384__li16423421351">Memory = Driver Memory + (Executors x Executor Memory)</li></ul>
</div></div>
</li></ul>
<div class="tablenoborder"><a name="dli_01_0384__table6886411195318"></a><a name="table6886411195318"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0384__table6886411195318" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Parameters for selecting dependency resources</caption><thead align="left"><tr id="dli_01_0384__row148851111165315"><th align="left" class="cellrowborder" valign="top" width="17.34%" id="mcps1.3.6.3.3.2.2.3.1.1"><p id="dli_01_0384__p9885111165318">Parameter</p>
</th>
@ -169,7 +177,8 @@
</thead>
<tbody><tr id="dli_01_0384__row58868119533"><td class="cellrowborder" valign="top" width="24.779999999999998%" headers="mcps1.3.6.3.3.3.2.3.1.1 "><p id="dli_01_0384__p28861911175315">Resource Specifications</p>
</td>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p20886101112539">Select a resource specification from the drop-down list box. The system provides three resource specifications for you to select. The following configuration items in the resource specifications can be modified:</p>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p899133751212">Select a resource specification from the drop-down list box. The system provides three resource specification options for you to choose from.</p>
<p id="dli_01_0384__p20886101112539">Resource specifications involve the following parameters:</p>
<ul id="dli_01_0384__ul9886181185310"><li id="dli_01_0384__li16886161145316">Executor Memory</li><li id="dli_01_0384__li588641115311">Executor Cores</li><li id="dli_01_0384__li6886191145311">Executors</li><li id="dli_01_0384__li48861911155312">Driver Cores</li><li id="dli_01_0384__li128861911105312">Driver Memory</li></ul>
<p id="dli_01_0384__p178861117533">If modified, your modified settings of the items are used.</p>
</td>
@ -177,35 +186,32 @@
<tr id="dli_01_0384__row38865112534"><td class="cellrowborder" valign="top" width="24.779999999999998%" headers="mcps1.3.6.3.3.3.2.3.1.1 "><p id="dli_01_0384__p1688613119536">Executor Memory</p>
</td>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p1688661155311">Customize the configuration item based on the selected resource specifications.</p>
<p id="dli_01_0384__p8357113814199">Memory of each Executor. It is recommended that the ratio of Executor CPU cores to Executor memory be 1:4.</p>
</td>
</tr>
<tr id="dli_01_0384__row98861211155312"><td class="cellrowborder" valign="top" width="24.779999999999998%" headers="mcps1.3.6.3.3.3.2.3.1.1 "><p id="dli_01_0384__p1988611111537">Executor Cores</p>
</td>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p688691175314">Customize the configuration item based on the selected resource specifications.</p>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p12200103320174">Number of CPU cores of each Executor applied for by Spark jobs, which determines the capability of each Executor to execute tasks concurrently.</p>
</td>
</tr>
<tr id="dli_01_0384__row10887111117538"><td class="cellrowborder" valign="top" width="24.779999999999998%" headers="mcps1.3.6.3.3.3.2.3.1.1 "><p id="dli_01_0384__p208861111125312">Executors</p>
</td>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p1588711112534">Customize the configuration item based on the selected resource specifications.</p>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p1588711112534">Number of Executors applied for by a Spark job</p>
</td>
</tr>
<tr id="dli_01_0384__row1488719112533"><td class="cellrowborder" valign="top" width="24.779999999999998%" headers="mcps1.3.6.3.3.3.2.3.1.1 "><p id="dli_01_0384__p588718119539">Driver Cores</p>
</td>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p1388771165313">Customize the configuration item based on the selected resource specifications.</p>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p1388771165313">Number of CPU cores of the driver</p>
</td>
</tr>
<tr id="dli_01_0384__row8887011145316"><td class="cellrowborder" valign="top" width="24.779999999999998%" headers="mcps1.3.6.3.3.3.2.3.1.1 "><p id="dli_01_0384__p48876114538">Driver Memory</p>
</td>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p48871811205310">Customize the configuration item based on the selected resource specifications.</p>
<td class="cellrowborder" valign="top" width="75.22%" headers="mcps1.3.6.3.3.3.2.3.1.2 "><p id="dli_01_0384__p48871811205310">Driver memory size. It is recommended that the ratio of the number of driver CPU cores to the driver memory be 1:4.</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="dli_01_0384__note11887101110530"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0384__p148877114534">Spark job parameter calculation:</p>
<ul id="dli_01_0384__ul388731113532"><li id="dli_01_0384__li1388711105319">Number of CUs = Number of driver CPU cores + Number of executors x Number of executor CPU cores<p id="dli_01_0384__p16887011105316"><a name="dli_01_0384__li1388711105319"></a><a name="li1388711105319"></a>The cluster management plane and driver use some CU resources. Number of Executors * Number of Executor Cores must be smaller than the number of computing CUs of the queue.</p>
</li><li id="dli_01_0384__li18871411135315">Memory = Driver memory + (Number of Executors x Executor memory)</li></ul>
</div></div>
</li><li id="dli_01_0384__li14938925195410">Click <strong id="dli_01_0384__b116961796373">Execute</strong> in the upper right corner of the Spark job editing page.<p id="dli_01_0384__p0554194120548">After the message "Batch processing job submitted successfully" is displayed, you can view the status and logs of the submitted job on the <strong id="dli_01_0384__b02351744153719">Spark Jobs</strong> page.</p>
</li></ol>
</div>

6
docs/dli/umn/dli_01_0385.html
View File

@ -54,9 +54,7 @@
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><ul id="dli_01_0385__en-us_topic_0122090417_ul181927155164"><li id="dli_01_0385__li31578516594"><strong id="dli_01_0385__b105273101889">Edit</strong>: You can modify the current job configuration and re-execute the job.</li><li id="dli_01_0385__li156921658183219"><strong id="dli_01_0385__b1971814139819">SparkUI</strong>: After you click this button, the Spark job execution page is displayed.<div class="note" id="dli_01_0385__note1149352315379"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_01_0385__ul17774418206"><li id="dli_01_0385__li47774118203">The SparkUI page cannot be viewed for jobs in the <strong id="dli_01_0385__b10335047121315">Starting</strong> state.</li><li id="dli_01_0385__li1375744472013">Currently, only the latest 100 job information records are displayed on the SparkUI of DLI.</li></ul>
</div></div>
</li><li id="dli_01_0385__li1370757905"><strong id="dli_01_0385__b6248346869">Terminate Job</strong>: Cancel a job that is being started or running.</li><li id="dli_01_0385__li99661755165919"><strong id="dli_01_0385__b109467691584043">Re-execute</strong>: Run the job again.</li><li id="dli_01_0385__li10304115213253"><strong id="dli_01_0385__b10525747115219">Archive Log</strong>: Save job logs to the temporary bucket created by DLI.</li><li id="dli_01_0385__li113222516517"><strong id="dli_01_0385__b10216205015215">Export Log</strong>: Export logs to the created OBS bucket.<div class="note" id="dli_01_0385__note9317379552"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_01_0385__ul1447511257345"><li id="dli_01_0385__li104752259341">You have the permission to create OBS buckets.</li><li id="dli_01_0385__li10475325103415">If the job is in the <strong id="dli_01_0385__b2161614101418">Running</strong> state, logs cannot be exported.</li></ul>
</div></div>
</li><li id="dli_01_0385__en-us_topic_0122090417_li11155202552811"><strong id="dli_01_0385__b25716541817">Commit Log</strong>: View the logs of submitted jobs.</li><li id="dli_01_0385__li245513563418"><strong id="dli_01_0385__b188631512145">Driver Log</strong>: View the logs of running jobs.</li></ul>
</li><li id="dli_01_0385__li1370757905"><strong id="dli_01_0385__b6248346869">Terminate Job</strong>: Cancel a job that is being started or running.</li><li id="dli_01_0385__li99661755165919"><strong id="dli_01_0385__b109467691584043">Re-execute</strong>: Run the job again.</li><li id="dli_01_0385__li10304115213253"><strong id="dli_01_0385__b10525747115219">Archive Log</strong>: Save job logs to the temporary bucket created by DLI.</li><li id="dli_01_0385__en-us_topic_0122090417_li11155202552811"><strong id="dli_01_0385__b25716541817">Commit Log</strong>: View the logs of submitted jobs.</li><li id="dli_01_0385__li245513563418"><strong id="dli_01_0385__b188631512145">Driver Log</strong>: View the logs of running jobs.</li></ul>
</td>
</tr>
</tbody>
@ -69,8 +67,6 @@
</div>
<div class="section" id="dli_01_0385__en-us_topic_0122090417_section3753111385816"><a name="dli_01_0385__en-us_topic_0122090417_section3753111385816"></a><a name="en-us_topic_0122090417_section3753111385816"></a><h4 class="sectiontitle">Terminating a Job</h4><p id="dli_01_0385__en-us_topic_0122090417_p209202331547">On the <span class="wintitle" id="dli_01_0385__wintitle18382471933"><b>Spark Jobs</b></span> page, choose <strong id="dli_01_0385__b143351153134">More</strong> &gt; <span class="uicontrol" id="dli_01_0385__uicontrol688113571241"><b>Terminate Job</b></span> in the <strong id="dli_01_0385__b7732018513">Operation</strong> column of the job that you want to stop.</p>
</div>
<div class="section" id="dli_01_0385__section64241412354"><h4 class="sectiontitle">Exporting Logs</h4><p id="dli_01_0385__p04251741153511">On the <span class="wintitle" id="dli_01_0385__wintitle1247511611510"><b>Spark Jobs</b></span> page, choose <span class="uicontrol" id="dli_01_0385__uicontrol155241812141510"><b>More </b></span>&gt; <strong id="dli_01_0385__b442820251290">Export Log</strong> in the Operation column of the corresponding job. In the dialog box that is displayed, enter the path of the created OBS bucket and click <strong id="dli_01_0385__b13687165841416">OK</strong>.</p>
</div>
</div>
<div>
<div class="familylinks">

2952
docs/dli/umn/dli_01_0397.html
View File

File diff suppressed because one or more lines are too long

2
docs/dli/umn/dli_01_0402.html
View File

@ -7,7 +7,7 @@
</div>
<div class="section" id="dli_01_0402__section127653118450"><h4 class="sectiontitle">Constraints</h4><ul id="dli_01_0402__ul176557436405"><li id="dli_01_0402__dli_07_0005_li18322179488">A queue named <strong id="dli_01_0402__dli_07_0005_b3301714174419">default</strong> is preset in DLI for you to experience. Resources are allocated on demand. </li><li id="dli_01_0402__dli_07_0005_li114381095272">Queue types:<ul id="dli_01_0402__dli_07_0005_ul16776114122718"><li id="dli_01_0402__dli_07_0005_li58411408270">For SQL: Spark SQL jobs can be submitted to SQL queues.</li><li id="dli_01_0402__dli_07_0005_li57631311152718">For general purpose: The queue is used to run Spark programs, Flink SQL jobs, and Flink Jar jobs.</li></ul>
<p id="dli_01_0402__dli_07_0005_p79475442714">The queue type cannot be changed. If you want to use another queue type, purchase a new queue.</p>
</li><li id="dli_01_0402__dli_07_0005_li88164952717">The region of a queue cannot be changed.</li><li id="dli_01_0402__dli_07_0005_li13490733171810">A newly created queue can be scaled in or out only after a job is executed on the queue.</li><li id="dli_01_0402__dli_07_0005_li9134133792411">DLI queues cannot access the Internet.<p id="dli_01_0402__dli_07_0005_p1748116407241"><a name="dli_01_0402__dli_07_0005_li9134133792411"></a><a name="dli_07_0005_li9134133792411"></a></p>
</li><li id="dli_01_0402__dli_07_0005_li88164952717">The region of a queue cannot be changed.</li><li id="dli_01_0402__dli_07_0005_li253742810428">Queues with 16 CUs do not support scale-out or scale-in.</li><li id="dli_01_0402__dli_07_0005_li6119202221710">Queues with 64 CUs do not support scale-in.</li><li id="dli_01_0402__dli_07_0005_li13490733171810">A newly created queue can be scaled in or out only after a job is executed on the queue.</li><li id="dli_01_0402__dli_07_0005_li9134133792411">DLI queues cannot access the Internet.<p id="dli_01_0402__dli_07_0005_p1748116407241"><a name="dli_01_0402__dli_07_0005_li9134133792411"></a><a name="dli_07_0005_li9134133792411"></a></p>
<p id="dli_01_0402__dli_07_0005_p629764172414"></p>
</li></ul>
</div>

45
docs/dli/umn/dli_01_0403.html
View File

@ -2,6 +2,7 @@
<h1 class="topictitle1">Overview</h1>
<div id="body1543376454662"><p id="dli_01_0403__p1866444210407">On the Job Management page of Flink jobs, you can submit a Flink job. Currently, the following job types are supported:</p>
<ul id="dli_01_0403__ul3319024204014"><li id="dli_01_0403__li153191024124015"><strong id="dli_01_0403__b582202818343">Flink SQL</strong> uses SQL statements to define jobs and can be submitted to any general purpose queue.</li><li id="dli_01_0403__li069317403210"><strong id="dli_01_0403__b208655222421">Flink Jar</strong> customizes a JAR package job based on Flink APIs. It runs on dedicated queues.</li></ul>
<p id="dli_01_0403__p66161828392">Flink job management provides the following functions:</p>
<ul id="dli_01_0403__ul8657743184112"><li id="dli_01_0403__li81824019583"><a href="dli_01_0479.html">Managing Flink Job Permissions</a></li><li id="dli_01_0403__li19657124354118"><a href="dli_01_0455.html">Creating a Flink SQL Job</a></li><li id="dli_01_0403__li49002025195814"><a href="dli_01_0457.html">Creating a Flink Jar Job</a></li><li id="dli_01_0403__li321633381117"><a href="dli_01_0461.html#dli_01_0461__section1950210297542">Editing a job</a></li><li id="dli_01_0403__li149164314117"><a href="dli_01_0461.html#dli_01_0461__section20957159163012">Starting a Job</a></li><li id="dli_01_0403__li1918757145613"><a href="dli_01_0461.html#dli_01_0461__section8678193324114">Stopping a Job</a></li><li id="dli_01_0403__li14179161612574"><a href="dli_01_0461.html#dli_01_0461__section1691624195713">Deleting a Job</a></li><li id="dli_01_0403__li529715414492"><a href="dli_01_0461.html#dli_01_0461__section135831511323">Exporting a Job</a></li><li id="dli_01_0403__li12456172025013"><a href="dli_01_0461.html#dli_01_0461__section75781665389">Importing a Job</a></li><li id="dli_01_0403__li7887305574"><a href="dli_01_0461.html#dli_01_0461__section15861321183619">Modifying Name and Description</a></li><li id="dli_01_0403__li131144243198"><a href="dli_01_0461.html#dli_01_0461__section83412445175">Importing to a Savepoint</a></li><li id="dli_01_0403__li5693174816504"><a href="dli_01_0461.html#dli_01_0461__section11401152191015">Triggering a Savepoint</a></li><li id="dli_01_0403__li255018247317"><a href="dli_01_0461.html#dli_01_0461__section48961252113110">Runtime Configuration</a></li><li id="dli_01_0403__li153181731145913"><a href="dli_01_0462.html">Job Details</a></li></ul>
@ -17,62 +18,62 @@
</div>
<div class="section" id="dli_01_0403__section12526165519235"><h4 class="sectiontitle">Flink Jobs Page</h4><p id="dli_01_0403__en-us_topic_0122090417_p66482014175614">On the <strong id="dli_01_0403__b5725174712329">Overview</strong> page, click <strong id="dli_01_0403__b16730204714323">Flink Jobs</strong> to go to the Flink job management page. Alternatively, you can choose <strong id="dli_01_0403__b177319476328">Job Management</strong> &gt; <strong id="dli_01_0403__b173184718320">Flink Jobs</strong> from the navigation pane on the left. The page displays all Flink jobs. If there are a large number of jobs, they will be displayed on multiple pages. DLI allows you to view jobs in all statuses.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0403__en-us_topic_0122090417_table3950169215120" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Job management parameters</caption><thead align="left"><tr id="dli_01_0403__en-us_topic_0122090417_row2555468715120"><th align="left" class="cellrowborder" valign="top" width="16.07%" id="mcps1.3.6.3.2.3.1.1"><p id="dli_01_0403__en-us_topic_0122090417_p4021197415120">Parameter</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0403__en-us_topic_0122090417_table3950169215120" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Job management parameters</caption><thead align="left"><tr id="dli_01_0403__en-us_topic_0122090417_row2555468715120"><th align="left" class="cellrowborder" valign="top" width="16.07%" id="mcps1.3.7.3.2.3.1.1"><p id="dli_01_0403__en-us_topic_0122090417_p4021197415120">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="83.93%" id="mcps1.3.6.3.2.3.1.2"><p id="dli_01_0403__en-us_topic_0122090417_p3594448915120">Description</p>
<th align="left" class="cellrowborder" valign="top" width="83.93%" id="mcps1.3.7.3.2.3.1.2"><p id="dli_01_0403__en-us_topic_0122090417_p3594448915120">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_01_0403__en-us_topic_0122090417_row46758327132"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p16413434141957">ID</p>
<tbody><tr id="dli_01_0403__en-us_topic_0122090417_row46758327132"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p16413434141957">ID</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p54419740141957">ID of a submitted Flink job, which is generated by the system by default.</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p54419740141957">ID of a submitted Flink job, which is generated by the system by default.</p>
</td>
</tr>
<tr id="dli_01_0403__en-us_topic_0122090417_row32873162171713"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p45480448171713">Name</p>
<tr id="dli_01_0403__en-us_topic_0122090417_row32873162171713"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p45480448171713">Name</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p18579134217227">Name of the submitted Flink job.</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p18579134217227">Name of the submitted Flink job.</p>
</td>
</tr>
<tr id="dli_01_0403__row3289513151315"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__p102901113161318">Type</p>
<tr id="dli_01_0403__row3289513151315"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__p102901113161318">Type</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><p id="dli_01_0403__p1029001315135">Type of the submitted Flink job. Including:</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><p id="dli_01_0403__p1029001315135">Type of the submitted Flink job. Including:</p>
<ul id="dli_01_0403__ul864114454138"><li id="dli_01_0403__li464144518136">Flink SQL: Flink SQL jobs</li><li id="dli_01_0403__li1896411561765">Flink Jar: Flink Jar jobs</li></ul>
</td>
</tr>
<tr id="dli_01_0403__en-us_topic_0122090417_row31011923151038"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p10671857151038">Status</p>
<tr id="dli_01_0403__en-us_topic_0122090417_row31011923151038"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p10671857151038">Status</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p59114099151038">Job statuses, including:</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p59114099151038">Job statuses, including:</p>
<ul id="dli_01_0403__en-us_topic_0122090417_ul32930526154023"><li id="dli_01_0403__li5304438104719"><strong id="dli_01_0403__b1218612714459">Draft</strong></li><li id="dli_01_0403__li230510384473"><strong id="dli_01_0403__b312364144518">Submitting</strong></li><li id="dli_01_0403__li1530543815477"><strong id="dli_01_0403__b10386111119456">Submission failed</strong></li><li id="dli_01_0403__li530483810474"><strong id="dli_01_0403__b54881649194919">Running</strong>: After the job is submitted, a normal result is returned.</li><li id="dli_01_0403__li27807399719"><strong id="dli_01_0403__b343275416497">Running exception</strong>: The job stops running due to an exception.</li><li id="dli_01_0403__li1196810358454"><strong id="dli_01_0403__b194672233472">Downloading</strong></li><li id="dli_01_0403__li194778183469"><strong id="dli_01_0403__b3168162564710">Idle</strong></li><li id="dli_01_0403__li19305838104720"><strong id="dli_01_0403__b5819182654717">Stopping</strong></li><li id="dli_01_0403__li130510384477"><strong id="dli_01_0403__b10747633104717">Stopped</strong></li><li id="dli_01_0403__li1430520388475"><strong id="dli_01_0403__b202671406472">Stopping failed</strong></li><li id="dli_01_0403__li73970453461"><strong id="dli_01_0403__b12394545124713">Creating the savepoint</strong></li><li id="dli_01_0403__li1330553812474"><strong id="dli_01_0403__b158604563497">Completed</strong></li></ul>
</td>
</tr>
<tr id="dli_01_0403__en-us_topic_0122090417_row36301606171658"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p14394959151048">Description</p>
<tr id="dli_01_0403__en-us_topic_0122090417_row36301606171658"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p14394959151048">Description</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p51238775151048">Description of the submitted Flink job.</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p51238775151048">Description of the submitted Flink job.</p>
</td>
</tr>
<tr id="dli_01_0403__row4736911141810"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__p1973721141811">Username</p>
<tr id="dli_01_0403__row4736911141810"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__p1973721141811">Username</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><p id="dli_01_0403__p373781161815">Name of the user who submits a job.</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><p id="dli_01_0403__p373781161815">Name of the user who submits a job.</p>
</td>
</tr>
<tr id="dli_01_0403__en-us_topic_0122090417_row6424839516213"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p50569641162134">Created</p>
<tr id="dli_01_0403__en-us_topic_0122090417_row6424839516213"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p50569641162134">Created</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p18910361162145">Time when a job is created.</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><p id="dli_01_0403__en-us_topic_0122090417_p18910361162145">Time when a job is created.</p>
</td>
</tr>
<tr id="dli_01_0403__row15840729143612"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__p13841152911367">Started</p>
<tr id="dli_01_0403__row15840729143612"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__p13841152911367">Started</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><p id="dli_01_0403__p20841122983612">Time when a Flink job starts to run.</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><p id="dli_01_0403__p20841122983612">Time when a Flink job starts to run.</p>
</td>
</tr>
<tr id="dli_01_0403__row1536633125019"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__p145363334505">Duration</p>
<tr id="dli_01_0403__row1536633125019"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__p145363334505">Duration</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><p id="dli_01_0403__p153603315013">Time consumed by job running.</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><p id="dli_01_0403__p153603315013">Time consumed by job running.</p>
</td>
</tr>
<tr id="dli_01_0403__en-us_topic_0122090417_row1662880815250"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.6.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p475621615250">Operation</p>
<tr id="dli_01_0403__en-us_topic_0122090417_row1662880815250"><td class="cellrowborder" valign="top" width="16.07%" headers="mcps1.3.7.3.2.3.1.1 "><p id="dli_01_0403__en-us_topic_0122090417_p475621615250">Operation</p>
</td>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.6.3.2.3.1.2 "><ul id="dli_01_0403__en-us_topic_0122090417_ul181927155164"><li id="dli_01_0403__en-us_topic_0122090417_li18623798286"><strong id="dli_01_0403__b692115307259">Edit</strong>: Edit a created job. For details, see <a href="dli_01_0461.html#dli_01_0461__section1950210297542">Editing a Job</a>.</li><li id="dli_01_0403__li016442515152"><strong id="dli_01_0403__b4887113372511">Start</strong>: Start and run a job. For details, see <a href="dli_01_0461.html#dli_01_0461__section20957159163012">Starting a Job</a>.</li><li id="dli_01_0403__li1228521944015"><strong id="dli_01_0403__b1893782782517">More</strong><ul id="dli_01_0403__ul2162826144010"><li id="dli_01_0403__en-us_topic_0122090417_li11155202552811"><strong id="dli_01_0403__b857792212255">FlinkUI</strong>: After you click this button, the Flink job execution page is displayed.<div class="note" id="dli_01_0403__note151024818191"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0403__p5501165371013">When you execute a job on a created queue, the cluster is restarted. It takes about 10 minutes. If you click <strong id="dli_01_0403__b25538517556">FlinkUI</strong> before the cluster is created, an empty <strong id="dli_01_0403__b668185785511">projectID</strong> will be cached. The <strong id="dli_01_0403__b541324185618">FlinkUI</strong> page cannot be displayed.</p>
<td class="cellrowborder" valign="top" width="83.93%" headers="mcps1.3.7.3.2.3.1.2 "><ul id="dli_01_0403__en-us_topic_0122090417_ul181927155164"><li id="dli_01_0403__en-us_topic_0122090417_li18623798286"><strong id="dli_01_0403__b692115307259">Edit</strong>: Edit a created job. For details, see <a href="dli_01_0461.html#dli_01_0461__section1950210297542">Editing a Job</a>.</li><li id="dli_01_0403__li016442515152"><strong id="dli_01_0403__b4887113372511">Start</strong>: Start and run a job. For details, see <a href="dli_01_0461.html#dli_01_0461__section20957159163012">Starting a Job</a>.</li><li id="dli_01_0403__li1228521944015"><strong id="dli_01_0403__b1893782782517">More</strong><ul id="dli_01_0403__ul2162826144010"><li id="dli_01_0403__en-us_topic_0122090417_li11155202552811"><strong id="dli_01_0403__b857792212255">FlinkUI</strong>: After you click this button, the Flink job execution page is displayed.<div class="note" id="dli_01_0403__note151024818191"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0403__p5501165371013">When you execute a job on a created queue, the cluster is restarted. It takes about 10 minutes. If you click <strong id="dli_01_0403__b25538517556">FlinkUI</strong> before the cluster is created, an empty <strong id="dli_01_0403__b668185785511">projectID</strong> will be cached. The <strong id="dli_01_0403__b541324185618">FlinkUI</strong> page cannot be displayed.</p>
<p id="dli_01_0403__p181031586197">You are advised to use a dedicated queue so that the cluster will not be released. Alternatively, wait for a while after the job is submitted (the cluster is created), and then check <strong id="dli_01_0403__b890617447568">FlinkUI</strong>.</p>
</div></div>
</li><li id="dli_01_0403__li2385531539"><strong id="dli_01_0403__b1631116258182">Stop</strong>: Stop a Flink job. If this function is unavailable, jobs in the current status cannot be stopped.</li><li id="dli_01_0403__li156921658183219"><strong id="dli_01_0403__b12871121811253">Delete</strong>: Delete a job.<div class="note" id="dli_01_0403__note386711433506"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0403__p16867104316506">A deleted job cannot be restored.</p>

2
docs/dli/umn/dli_01_0410.html
View File

@ -53,7 +53,7 @@
<td class="cellrowborder" valign="top" width="19.23%" headers="mcps1.3.3.3.2.6.1.5 "><p id="dli_01_0410__p1721193775313">√</p>
</td>
</tr>
<tr id="dli_01_0410__row17214376539"><td class="cellrowborder" valign="top" width="22.75%" headers="mcps1.3.3.3.2.6.1.1 "><p id="dli_01_0410__p137211537195311">DDS Mongo</p>
<tr id="dli_01_0410__row17214376539"><td class="cellrowborder" valign="top" width="22.75%" headers="mcps1.3.3.3.2.6.1.1 "><p id="dli_01_0410__p137211537195311">DDS</p>
</td>
<td class="cellrowborder" valign="top" width="19.11%" headers="mcps1.3.3.3.2.6.1.2 "><p id="dli_01_0410__p5721137185312">√</p>
</td>

5
docs/dli/umn/dli_01_0421.html
View File

@ -1,7 +1,8 @@
<a name="dli_01_0421"></a><a name="dli_01_0421"></a>
<h1 class="topictitle1">Creating a Message Notification Topic</h1>
<div id="body1566197386597"><div class="section" id="dli_01_0421__section5242489132"><h4 class="sectiontitle">Scenario</h4><p id="dli_01_0421__p1153362612142">Once you have created a message notification topic, you can <strong id="dli_01_0421__b596222242318">Add subscription</strong> of the topic on the <strong id="dli_01_0421__b179621022142319">Topic Management</strong> page of the Simple Message Notification service. You can select different ways (such as text messages or emails) to subscribe. After the subscription succeeds, any job failure will automatically be sent to your subscription endpoints.</p>
<h1 class="topictitle1">Creating an SMN Topic</h1>
<div id="body1566197386597"><div class="section" id="dli_01_0421__section5242489132"><h4 class="sectiontitle">Scenario</h4><p id="dli_01_0421__p1489611475420">Once you have created an SMN topic, you can easily subscribe to it by going to the <strong id="dli_01_0421__b15416917161219">Topic Management</strong> &gt; <strong id="dli_01_0421__b19866172031211">Topics</strong> page of the SMN console. You can choose to receive notifications via SMS or email. After the subscription is successful, if a job fails, the system automatically sends a message to the subscription endpoint you specified.</p>
<ul id="dli_01_0421__ul1929516112577"><li id="dli_01_0421__li12856309244">If a job fails within 1 minute of submission, a message notification is not triggered.</li><li id="dli_01_0421__li62952119577">If a job fails after 1 minute of submission, the system automatically sends a message to the subscriber terminal you specified.</li></ul>
</div>
<div class="section" id="dli_01_0421__section742629201411"><h4 class="sectiontitle">Procedure</h4><ol id="dli_01_0421__ol25920582507"><li id="dli_01_0421__li9592558175016">On the <strong id="dli_01_0421__b13249815101012">Resources</strong> &gt; <strong id="dli_01_0421__b1990823713261">Queue Management</strong> page, click <strong id="dli_01_0421__b1914153715265">Create SMN Topic</strong> on the upper left side. The <strong id="dli_01_0421__b15914113762619">Create SMN Topic</strong> dialog box is displayed.</li><li id="dli_01_0421__li144161849165115">Select a queue and click <strong id="dli_01_0421__b8335131614183">OK</strong>.<div class="note" id="dli_01_0421__note1848145063719"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_01_0421__ul731513171718"><li id="dli_01_0421__li780591161620">You can select a single queue or all queues.</li><li id="dli_01_0421__li04613151711">If you create a topic for a queue and another topic for all queues, the SMN of all queues does not include the message of the single queue.</li><li id="dli_01_0421__li174141311718">After a message notification topic is created, you will receive a message notification only when a Spark job created on the subscription queue fails.</li></ul>
</div></div>

2
docs/dli/umn/dli_01_0422.html
View File

@ -4,7 +4,7 @@
<div id="body1568103018192"></div>
<div>
<ul class="ullinks">
<li class="ulchildlink"><strong><a href="dli_01_0561.html">Introduction</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_01_0561.html">Overview</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_01_0427.html">Creating a CSS Datasource Authentication</a></strong><br>
</li>

73
docs/dli/umn/dli_01_0440.html
View File

File diff suppressed because it is too large Load Diff

2
docs/dli/umn/dli_01_0441.html
View File

@ -15,7 +15,7 @@
</th>
<th align="left" class="cellrowborder" valign="top" width="14.258574142585742%" id="mcps1.3.2.2.8.1.6"><p id="dli_01_0441__en-us_topic_0206791772_p2783111010508">Tenant Administrator</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="12.75872412758724%" id="mcps1.3.2.2.8.1.7"><p id="dli_01_0441__en-us_topic_0206791772_p177851710155012">DLI Service Admin</p>
<th align="left" class="cellrowborder" valign="top" width="12.75872412758724%" id="mcps1.3.2.2.8.1.7"><p id="dli_01_0441__en-us_topic_0206791772_p177851710155012">DLI Service Administrator</p>
</th>
</tr>
</thead>

292
docs/dli/umn/dli_01_0445.html
View File

File diff suppressed because it is too large Load Diff

2
docs/dli/umn/dli_01_0447.html
View File

@ -1,7 +1,7 @@
<a name="dli_01_0447"></a><a name="dli_01_0447"></a>
<h1 class="topictitle1">Managing Database Permissions</h1>
<div id="body1571736789703"><div class="section" id="dli_01_0447__section2883543204514"><h4 class="sectiontitle">Scenario</h4><ul id="dli_01_0447__ul6751144710454"><li id="dli_01_0447__li18752174718455">You can isolate databases allocated to different users by setting permissions to ensure data query performance.</li><li id="dli_01_0447__li16752047104516">The administrator and database owner have all permissions, which cannot be set or modified by other users.</li></ul>
<div id="body1571736789703"><div class="section" id="dli_01_0447__section2883543204514"><h4 class="sectiontitle">Scenario</h4><ul id="dli_01_0447__ul6751144710454"><li id="dli_01_0447__li18752174718455">By setting permissions, you can assign varying database permissions to different users.</li><li id="dli_01_0447__li16752047104516">The administrator and database owner have all permissions, which cannot be set or modified by other users.</li></ul>
</div>
<div class="section" id="dli_01_0447__section13393554152612"><h4 class="sectiontitle">Precautions</h4><ul id="dli_01_0447__ul16414185422619"><li id="dli_01_0447__li44137546264">Lower-level objects automatically inherit permissions granted to upper-level objects. The hierarchical relationship is database &gt; table &gt; column.</li><li id="dli_01_0447__li971149172410">The database owner, table owner, and <span class="parmname" id="dli_01_0447__parmname13746731125417"><b>authorized</b></span> users can assign permissions on the database and tables.</li><li id="dli_01_0447__li10413145418269">Columns can only inherit the query permission. For details about <span class="parmname" id="dli_01_0447__parmname15888124214540"><b>Inheritable Permissions</b></span>, see <a href="dli_01_0447.html">Managing Database Permissions</a>.</li><li id="dli_01_0447__li16413175413264">The permissions can be revoked only at the initial level to which the permissions are granted. You need to grant and revoke permissions at the same level. You need to grant and revoke permissions at the same level. For example, after you are granted the insertion permission on a database, you can obtain the insertion permission on the tables in the database. Your insertion permission can be revoked only at the database level.</li><li id="dli_01_0447__li29052232217">If you create a database with the same name as a deleted database, the database permissions will not be inherited. In this case, you need to grant the database permissions to users or projects.<p id="dli_01_0447__p08812692719"><a name="dli_01_0447__li29052232217"></a><a name="li29052232217"></a>For example, user A is granted with the permission to delete the <strong id="dli_01_0447__b20228161610456">testdb</strong> database. Delete the database and create another one with the same name. You need to grant user A the deletion permission of the <strong id="dli_01_0447__b76882037184612">testdb</strong> database again.</p>
</li></ul>

12
docs/dli/umn/dli_01_0448.html
View File

@ -1,7 +1,7 @@
<a name="dli_01_0448"></a><a name="dli_01_0448"></a>
<h1 class="topictitle1">Managing Table Permissions</h1>
<div id="body1571736789703"><div class="section" id="dli_01_0448__section2883543204514"><h4 class="sectiontitle">Operation Scenario</h4><ul id="dli_01_0448__ul6751144710454"><li id="dli_01_0448__li18752174718455">You can isolate databases allocated to different users by setting permissions to ensure data query performance.</li><li id="dli_01_0448__li16752047104516">The administrator and database owner have all permissions, which cannot be set or modified by other users.</li><li id="dli_01_0448__li475264711458">When setting database permissions for a new user, ensure that the user group to which the user belongs has the <strong id="dli_01_0448__b17434485377">Tenant Guest</strong> permission. </li></ul>
<div id="body1571736789703"><div class="section" id="dli_01_0448__section2883543204514"><h4 class="sectiontitle">Operation Scenario</h4><ul id="dli_01_0448__ul6751144710454"><li id="dli_01_0448__li18752174718455">By setting permissions, you can assign varying table permissions to different users.</li><li id="dli_01_0448__li16752047104516">The administrator and table owner have all permissions, which cannot be set or modified by other users.</li><li id="dli_01_0448__li475264711458">When setting table permissions for a new user, ensure that the user group the user belongs to has the <strong id="dli_01_0448__b17434485377">Tenant Guest</strong> permission. </li></ul>
</div>
<div class="section" id="dli_01_0448__section9719284599"><h4 class="sectiontitle">Precautions</h4><ul id="dli_01_0448__ul849031211304"><li id="dli_01_0448__li29052232217">If you create a table with the same name as a deleted table, the table permissions will not be inherited. In this case, you need to grant the table permissions to users or projects.<p id="dli_01_0448__p08812692719"><a name="dli_01_0448__li29052232217"></a><a name="li29052232217"></a>For example, user A is granted with the permission to delete the <strong id="dli_01_0448__b1382451218915">testTable</strong> table. Delete the table and create another one with the same name. You need to grant user A the deletion permission of the <strong id="dli_01_0448__b165131017912">testTable</strong> table again.</p>
</li></ul>
@ -55,21 +55,21 @@
</thead>
<tbody><tr id="dli_01_0448__row565224044819"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.1.2.2.2.3.2.3.1.1 "><p id="dli_01_0448__p9652144074816">Authorization Object</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.1.2.2.2.3.2.3.1.2 "><p id="dli_01_0448__p14652174017486">Select <strong id="dli_01_0448__b1908672938">User</strong> or <strong id="dli_01_0448__b746750941">Project</strong>.</p>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.1.2.2.2.3.2.3.1.2 "><p id="dli_01_0448__p14652174017486">Select <strong id="dli_01_0448__b1836684765">User</strong> or <strong id="dli_01_0448__b1550655958">Project</strong>.</p>
</td>
</tr>
<tr id="dli_01_0448__row166532040114818"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.1.2.2.2.3.2.3.1.1 "><p id="dli_01_0448__p565219406486">Username/Project</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.1.2.2.2.3.2.3.1.2 "><ul id="dli_01_0448__ul126532403482"><li id="dli_01_0448__li16652440194810">If you select <strong id="dli_01_0448__b1182990332">User</strong>, enter the IAM username when granting table permissions to the user.<div class="note" id="dli_01_0448__note42264017164"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0448__p2232130161618">The username is an existing IAM user name and has logged in to the DLI management console.</p>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.1.2.2.2.3.2.3.1.2 "><ul id="dli_01_0448__ul126532403482"><li id="dli_01_0448__li16652440194810">If you select <strong id="dli_01_0448__b522860825">User</strong>, enter the IAM username when granting table permissions to the user.<div class="note" id="dli_01_0448__note42264017164"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0448__p2232130161618">The username is an existing IAM user name and has logged in to the DLI management console.</p>
</div></div>
</li><li id="dli_01_0448__li765312406486">If you select <strong id="dli_01_0448__b1209073757">Project</strong>, select the project to be authorized in the current region.<div class="note" id="dli_01_0448__note5437185910377"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0448__p3437159143716">If you select <strong id="dli_01_0448__b222310053715">Project</strong>, you can only view information about the authorized tables and their databases.</p>
</li><li id="dli_01_0448__li765312406486">If you select <strong id="dli_01_0448__b1394871595">Project</strong>, select the project to be authorized in the current region.<div class="note" id="dli_01_0448__note5437185910377"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0448__p3437159143716">If you select <strong id="dli_01_0448__b222310053715">Project</strong>, you can only view information about the authorized tables and their databases.</p>
</div></div>
</li></ul>
</td>
</tr>
<tr id="dli_01_0448__row116546401484"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.1.2.2.2.3.2.3.1.1 "><p id="dli_01_0448__p0653184018489">Non-inheritable Permissions</p>
</td>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.1.2.2.2.3.2.3.1.2 "><div class="p" id="dli_01_0448__p26531940154817">Select a permission to grant it to the user, or deselect a permission to revoke it.<ul id="dli_01_0448__ul186541940194819"><li id="dli_01_0448__li1037614422018">The following permissions are applicable to both user and project authorization:<ul id="dli_01_0448__ul11414111319209"><li id="dli_01_0448__li17280185220227"><strong id="dli_01_0448__b1475611842911">View Table Creation Statement</strong>: This permission allows you to view the statement for creating the current table.</li><li id="dli_01_0448__li6385363379"><strong id="dli_01_0448__b88661213112914">View Table Information</strong>: This permission allows you to view information about the current table.</li><li id="dli_01_0448__li1365364034816"><strong id="dli_01_0448__b2011496059">Select Table</strong>: This permission allows you to query data of the current table.</li><li id="dli_01_0448__li9653240154816"><strong id="dli_01_0448__b614804474">Drop Table</strong>: This permission allows you to delete the current table.</li><li id="dli_01_0448__li56531540144810"><strong id="dli_01_0448__b1808897767">Rename Table</strong>: Rename the current table.</li><li id="dli_01_0448__li1465374011487"><strong id="dli_01_0448__b340672502">Insert</strong>: This permission allows you to insert data into the current table.</li><li id="dli_01_0448__li1965364014817"><strong id="dli_01_0448__b254284073">Overwrite</strong>: This permission allows you to insert data to overwrite the data in the current table.</li><li id="dli_01_0448__li6653194011481"><strong id="dli_01_0448__b1868414236293">Add Column</strong>: This permission allows you to add columns to the current table.</li><li id="dli_01_0448__li186531240124810"><strong id="dli_01_0448__b11229173814372">Grant Permission</strong>: This permission allows you to grant table permissions to other users or projects.</li><li id="dli_01_0448__li66531440174811"><strong id="dli_01_0448__b1752995116373">Revoke Permission</strong>: This permission allows you to revoke the table's permissions that other users or projects have but cannot revoke the table owner's permissions.</li><li id="dli_01_0448__li8654194024817"><strong id="dli_01_0448__b814732953819">View Other Users' Permissions</strong>: This permission allows you to query other users' permission on the current table.</li></ul>
<td class="cellrowborder" valign="top" width="81.25%" headers="mcps1.3.4.2.1.2.2.2.3.2.3.1.2 "><div class="p" id="dli_01_0448__p26531940154817">Select a permission to grant it to the user, or deselect a permission to revoke it.<ul id="dli_01_0448__ul186541940194819"><li id="dli_01_0448__li1037614422018">The following permissions are applicable to both user and project authorization:<ul id="dli_01_0448__ul11414111319209"><li id="dli_01_0448__li17280185220227"><strong id="dli_01_0448__b1475611842911">View Table Creation Statement</strong>: This permission allows you to view the statement for creating the current table.</li><li id="dli_01_0448__li6385363379"><strong id="dli_01_0448__b88661213112914">View Table Information</strong>: This permission allows you to view information about the current table.</li><li id="dli_01_0448__li1365364034816"><strong id="dli_01_0448__b313497768">Select Table</strong>: This permission allows you to query data of the current table.</li><li id="dli_01_0448__li9653240154816"><strong id="dli_01_0448__b291334707">Drop Table</strong>: This permission allows you to delete the current table.</li><li id="dli_01_0448__li56531540144810"><strong id="dli_01_0448__b417028568">Rename Table</strong>: Rename the current table.</li><li id="dli_01_0448__li1465374011487"><strong id="dli_01_0448__b1562278110">Insert</strong>: This permission allows you to insert data into the current table.</li><li id="dli_01_0448__li1965364014817"><strong id="dli_01_0448__b2015126778">Overwrite</strong>: This permission allows you to insert data to overwrite the data in the current table.</li><li id="dli_01_0448__li6653194011481"><strong id="dli_01_0448__b1868414236293">Add Column</strong>: This permission allows you to add columns to the current table.</li><li id="dli_01_0448__li186531240124810"><strong id="dli_01_0448__b11229173814372">Grant Permission</strong>: This permission allows you to grant table permissions to other users or projects.</li><li id="dli_01_0448__li66531440174811"><strong id="dli_01_0448__b1752995116373">Revoke Permission</strong>: This permission allows you to revoke the table's permissions that other users or projects have but cannot revoke the table owner's permissions.</li><li id="dli_01_0448__li8654194024817"><strong id="dli_01_0448__b814732953819">View Other Users' Permissions</strong>: This permission allows you to query other users' permission on the current table.</li></ul>
<div class="p" id="dli_01_0448__p18654640154814">The partition table also has the following permissions:<ul id="dli_01_0448__ul76544409484"><li id="dli_01_0448__li1865415406488"><strong id="dli_01_0448__b46593558555">Add Partition</strong>: This permission allows you to add a partition to a partition table.</li><li id="dli_01_0448__li126548407483"><strong id="dli_01_0448__b126796584556">Delete Partition</strong>: This permission allows you to delete existing partitions from a partition table.</li><li id="dli_01_0448__li965474084819"><strong id="dli_01_0448__b523170183913">Configure Path for Partition</strong>: This permission allows you to set the path of a partition in a partition table to a specified OBS path.</li><li id="dli_01_0448__li1065424019489"><strong id="dli_01_0448__b69983116565">Rename Table Partition</strong>: This permission allows you to rename partitions in a partition table.</li><li id="dli_01_0448__li126547402487"><strong id="dli_01_0448__b088993155619">Restore Table Partition</strong>: This permission allows you to export partition information from the file system and save the information to metadata.</li><li id="dli_01_0448__li86541140134815"><strong id="dli_01_0448__b323918620561">View All Partitions</strong>: This permission allows you to view all partitions in a partition table.</li></ul>
</div>
</li></ul>
@ -92,7 +92,7 @@
</thead>
<tbody><tr id="dli_01_0448__row20659134011489"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.2.2.3.3.2.3.1.1 "><p id="dli_01_0448__p1265724017483">Authorization Object</p>
</td>
<td class="cellrowborder" valign="top" width="80%" headers="mcps1.3.4.2.1.2.2.3.3.2.3.1.2 "><p id="dli_01_0448__p565864034813">Select <strong id="dli_01_0448__b795663969">User</strong> or <strong id="dli_01_0448__b576928764">Project</strong>.</p>
<td class="cellrowborder" valign="top" width="80%" headers="mcps1.3.4.2.1.2.2.3.3.2.3.1.2 "><p id="dli_01_0448__p565864034813">Select <strong id="dli_01_0448__b883842996">User</strong> or <strong id="dli_01_0448__b1515428361">Project</strong>.</p>
</td>
</tr>
<tr id="dli_01_0448__row19659140144815"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.4.2.1.2.2.3.3.2.3.1.1 "><p id="dli_01_0448__p1865915409485">Username/Project</p>

2
docs/dli/umn/dli_01_0451.html
View File

@ -262,7 +262,7 @@
}</pre>
</li></ol>
</div>
<div class="section" id="dli_01_0451__section1493518251395"><h4 class="sectiontitle">Example Custom Policies</h4><ul id="dli_01_0451__ul6232826394"><li id="dli_01_0451__li142171951131513">Example 1: Allow policies<ul id="dli_01_0451__ul4699115781519"><li id="dli_01_0451__li323282619918">Allow users to create tables in all databases of all regions:<pre class="screen" id="dli_01_0451__screen151136133513">{
<div class="section" id="dli_01_0451__section1493518251395"><h4 class="sectiontitle">Example Custom Policies</h4><ul id="dli_01_0451__ul6232826394"><li id="dli_01_0451__li142171951131513">Example 1: Allow policies<ul id="dli_01_0451__ul4699115781519"><li id="dli_01_0451__li323282619918">Allow users to create tables across all databases in all regions:<pre class="screen" id="dli_01_0451__screen151136133513">{
"Version": "1.1",
"Statement": [
{

2
docs/dli/umn/dli_01_0455.html
View File

@ -3,7 +3,7 @@
<h1 class="topictitle1">Creating a Flink SQL Job</h1>
<div id="body1574761276020"><p id="dli_01_0455__p1426222391410">This section describes how to create a Flink SQL job. You can use Flink SQLs to develop jobs to meet your service requirements. Using SQL statements simplifies logic implementation. You can edit Flink SQL statements for your job in the DLI SQL editor. This section describes how to use the SQL editor to write Flink SQL statements.</p>
<p id="dli_01_0455__p0792237205215"><a href="dli_01_0498.html">DLI Flink OpenSource SQL jobs</a> are fully compatible with the syntax of Flink 1.10 and 1.12 provided by the community. In addition, Redis, GaussDB(DWS), and DIS data source types are added based on the community connector.</p>
<div class="section" id="dli_01_0455__section9243114805018"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_01_0455__ul32010161007"><li id="dli_01_0455__li142013162002">You have prepared the data input and data output channels. For details, see <a href="dli_01_0454.html">Preparing Flink Job Data</a>.</li><li id="dli_01_0455__li7520181309">When you use a Flink SQL job to access other external data sources, such as OpenTSDB, HBase, Kafka, DWS, RDS, CSS, CloudTable, DCS Redis, and DDS MongoDB, you need to create a cross-source connection to connect the job running queue to the external data source.<ul id="dli_01_0455__ul3477133419715"><li id="dli_01_0455__li18451528175">For details about the external data sources that can be accessed by Flink jobs, see <a href="dli_01_0410.html">Cross-Source Analysis Development Methods</a>.</li><li id="dli_01_0455__li28469131945">For details about how to create a datasource connection, see <a href="dli_01_0426.html">Enhanced Datasource Connections</a>. After a datasource connection is created, you can choose <strong id="dli_01_0455__b136411947171620">More</strong> &gt; <strong id="dli_01_0455__b8991851171618">Test Address Connectivity</strong> in the <strong id="dli_01_0455__b2161654131616">Operation</strong> column on the <strong id="dli_01_0455__b65150213175">Queue Management</strong> page to check whether the network connection between the queue and the external data source is normal. For details, see <a href="dli_01_0489.html">Testing Address Connectivity</a>.</li></ul>
<div class="section" id="dli_01_0455__section9243114805018"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_01_0455__ul32010161007"><li id="dli_01_0455__li142013162002">You have prepared the data input and data output channels. For details, see <a href="dli_01_0454.html">Preparing Flink Job Data</a>.</li><li id="dli_01_0455__li7520181309">When you use a Flink SQL job to access other external data sources, such as OpenTSDB, HBase, Kafka, DWS, RDS, CSS, CloudTable, DCS Redis, and DDS, you need to create a cross-source connection to connect the job running queue to the external data source.<ul id="dli_01_0455__ul3477133419715"><li id="dli_01_0455__li18451528175">For details about the external data sources that can be accessed by Flink jobs, see <a href="dli_01_0410.html">Cross-Source Analysis Development Methods</a>.</li><li id="dli_01_0455__li28469131945">For details about how to create a datasource connection, see <a href="dli_01_0426.html">Enhanced Datasource Connections</a>. After a datasource connection is created, you can choose <strong id="dli_01_0455__b136411947171620">More</strong> &gt; <strong id="dli_01_0455__b8991851171618">Test Address Connectivity</strong> in the <strong id="dli_01_0455__b2161654131616">Operation</strong> column on the <strong id="dli_01_0455__b65150213175">Queue Management</strong> page to check whether the network connection between the queue and the external data source is normal. For details, see <a href="dli_01_0489.html">Testing Address Connectivity</a>.</li></ul>
</li></ul>
</div>
<div class="section" id="dli_01_0455__section28145411519"><h4 class="sectiontitle">Creating a Flink SQL Job</h4><ol id="dli_01_0455__ol15995121431920"><li id="dli_01_0455__li899510144197"><span>In the left navigation pane of the DLI management console, choose <span class="menucascade" id="dli_01_0455__menucascade10969155116523"><b><span class="uicontrol" id="dli_01_0455__uicontrol3969145120529">Job Management</span></b></span> &gt; <span class="menucascade" id="dli_01_0455__menucascade114421945855"><b><span class="uicontrol" id="dli_01_0455__uicontrol044254519511">Flink Jobs</span></b></span>. The <span class="wintitle" id="dli_01_0455__wintitle6857152523411"><b>Flink Jobs</b></span> page is displayed.</span></li><li id="dli_01_0455__li1832715401765"><span>In the upper right corner of the <span class="wintitle" id="dli_01_0455__wintitle149654113418"><b>Flink Jobs</b></span> page, click <span class="uicontrol" id="dli_01_0455__uicontrol1968964993415"><b>Create Job</b></span>.</span></li><li id="dli_01_0455__li1414012319596"><span>Specify job parameters.</span><p>

2
docs/dli/umn/dli_01_0457.html
View File

@ -2,7 +2,7 @@
<h1 class="topictitle1">Creating a Flink Jar Job</h1>
<div id="body1574761279830"><p id="dli_01_0457__p76741043151420">This section describes how to create a Flink Jar job. You can perform secondary development based on Flink APIs, build your own JAR file, and submit the JAR file to DLI queues. DLI is fully compatible with open-source community APIs. To create a custom Flink job, you need to compile and build application JAR files. You must have a certain understanding of Flink secondary development and have high requirements related to stream computing complexity.</p>
<div class="section" id="dli_01_0457__section13123183424115"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_01_0457__ul198451926172619"><li id="dli_01_0457__li51821947175614">Ensure that a dedicated queue has been created. To create a dedicated queue, select <strong id="dli_01_0457__b46637459575">Dedicated Resource Mode</strong> when you choose the type of a queue during purchase.</li><li id="dli_01_0457__li7520181309">When creating a Flink Jar job to access other external data sources, such as OpenTSDB, HBase, Kafka, GaussDB(DWS), RDS, CSS, CloudTable, DCS Redis, and DDS MongoDB, you need to create a cross-source connection to connect the job running queue to the external data source.<ul id="dli_01_0457__ul850415199816"><li id="dli_01_0457__li18451528175">For details about the external data sources that can be accessed by Flink jobs, see <a href="dli_01_0410.html">Cross-Source Analysis Development Methods</a>.</li><li id="dli_01_0457__li28469131945">For details about how to create a datasource connection, see <a href="dli_01_0426.html">Enhanced Datasource Connections</a>.<p id="dli_01_0457__p881561411413">On the <strong id="dli_01_0457__b232324774013">Resources</strong> &gt; <strong id="dli_01_0457__b132414714406">Queue Management</strong> page, locate the queue you have created, and choose <strong id="dli_01_0457__b12324154712404">More</strong> &gt; <strong id="dli_01_0457__b432415477400">Test Address Connectivity</strong> in the <strong id="dli_01_0457__b4324847174012">Operation</strong> column to check whether the network connection between the queue and the data source is normal. For details, see <a href="dli_01_0489.html">Testing Address Connectivity</a>.</p>
<div class="section" id="dli_01_0457__section13123183424115"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_01_0457__ul198451926172619"><li id="dli_01_0457__li51821947175614">Ensure that a dedicated queue has been created. To create a dedicated queue, select <strong id="dli_01_0457__b46637459575">Dedicated Resource Mode</strong> when you choose the type of a queue during purchase.</li><li id="dli_01_0457__li7520181309">When creating a Flink Jar job to access other external data sources, such as OpenTSDB, HBase, Kafka, GaussDB(DWS), RDS, CSS, CloudTable, DCS Redis, and DDS, you need to create a cross-source connection to connect the job running queue to the external data source.<ul id="dli_01_0457__ul850415199816"><li id="dli_01_0457__li18451528175">For details about the external data sources that can be accessed by Flink jobs, see <a href="dli_01_0410.html">Cross-Source Analysis Development Methods</a>.</li><li id="dli_01_0457__li28469131945">For details about how to create a datasource connection, see <a href="dli_01_0426.html">Enhanced Datasource Connections</a>.<p id="dli_01_0457__p881561411413">On the <strong id="dli_01_0457__b232324774013">Resources</strong> &gt; <strong id="dli_01_0457__b132414714406">Queue Management</strong> page, locate the queue you have created, and choose <strong id="dli_01_0457__b12324154712404">More</strong> &gt; <strong id="dli_01_0457__b432415477400">Test Address Connectivity</strong> in the <strong id="dli_01_0457__b4324847174012">Operation</strong> column to check whether the network connection between the queue and the data source is normal. For details, see <a href="dli_01_0489.html">Testing Address Connectivity</a>.</p>
</li></ul>
</li><li id="dli_01_0457__li17845142616266">When running a Flink Jar job, you need to build the secondary development application code into a Jar package and upload the JAR package to the created OBS bucket. Choose <strong id="dli_01_0457__b14988132715315">Data Management</strong> &gt; <strong id="dli_01_0457__b121417311332">Package Management</strong> to create a package. For details, see <a href="dli_01_0367.html">Creating a Package</a>.<div class="note" id="dli_01_0457__note38679553417"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0457__p108692554412">DLI does not support the download function. If you need to modify the uploaded data file, please edit the local file and upload it again.</p>
</div></div>

12
docs/dli/umn/dli_01_0476.html
View File

@ -4,8 +4,8 @@
<div id="body1585470126556"><div class="section" id="dli_01_0476__section79715592428"><h4 class="sectiontitle">What Is a Global Variable?</h4><p id="dli_01_0476__p1325182820272">DLI allows you to set variables that are frequently used during job development as global variables on the DLI management console. This avoids repeated definitions during job editing and reduces development and maintenance costs. Global variables can be used to replace long and difficult variables, simplifying complex parameters and improving the readability of SQL statements.</p>
<p id="dli_01_0476__p18532104113257">This section describes how to create a global variable.</p>
</div>
<div class="section" id="dli_01_0476__section1910480202411"><h4 class="sectiontitle">Creating Variables</h4><ol id="dli_01_0476__ol101671049115710"><li id="dli_01_0476__li1216717496572">In the navigation pane of the DLI console, choose <strong id="dli_01_0476__b5731101211489">Global Configuration</strong> &gt; <strong id="dli_01_0476__b1799752319522">Global Variables</strong>.</li><li id="dli_01_0476__li11674105415712">On the <strong id="dli_01_0476__b152526335525">Global Variables</strong> page, click <strong id="dli_01_0476__b225718337526">Create</strong> in the upper right corner to create a global variable.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0476__table15710625151416" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters description</caption><thead align="left"><tr id="dli_01_0476__row1771212551410"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.2.2.2.3.2.3.1.1"><p id="dli_01_0476__p4713132520147">Parameter</p>
<div class="section" id="dli_01_0476__section1910480202411"><h4 class="sectiontitle">Creating a Global Variable</h4><ol id="dli_01_0476__ol101671049115710"><li id="dli_01_0476__li1216717496572">In the navigation pane of the DLI console, choose <strong id="dli_01_0476__b5731101211489">Global Configuration</strong> &gt; <strong id="dli_01_0476__b1799752319522">Global Variables</strong>.</li><li id="dli_01_0476__li11674105415712">On the <strong id="dli_01_0476__b152526335525">Global Variables</strong> page, click <strong id="dli_01_0476__b225718337526">Create</strong> in the upper right corner to create a global variable.
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0476__table15710625151416" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Global variable parameters</caption><thead align="left"><tr id="dli_01_0476__row1771212551410"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.2.2.2.3.2.3.1.1"><p id="dli_01_0476__p4713132520147">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="80%" id="mcps1.3.2.2.2.3.2.3.1.2"><p id="dli_01_0476__p1971410256146">Description</p>
</th>
@ -24,19 +24,17 @@
</tbody>
</table>
</div>
<div class="note" id="dli_01_0476__note12830104723217"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_01_0476__ul9760623113011"><li id="dli_01_0476__li1676182313011">Only whitelisted users are allowed to create sensitive variables. To use this function, submit a service ticket to the administrator.</li><li id="dli_01_0476__li18761142314302">If passwords or other sensitive information is involved, you can set variables as sensitive ones.</li></ul>
</div></div>
</li><li id="dli_01_0476__li073251925811">After creating a global variable, use <strong id="dli_01_0476__b83791710155420">{{xxxx}}</strong> in the SQL statement to replace the parameter value set as the global variable. <strong id="dli_01_0476__b1476052145415">xxxx</strong> indicates the variable name. For example, if you set global variable <strong id="dli_01_0476__b12182171914195">abc</strong> to represent the table name, replace the actual table name with <strong id="dli_01_0476__b1827018241196">{{abc}}</strong> in the table creation statement.<pre class="screen" id="dli_01_0476__screen12761219105816">create table {{table_name}} (String1 String, int4 int, varchar1 varchar(10))
partitioned by (int1 int,int2 int,int3 int)</pre>
<div class="note" id="dli_01_0476__note183371023201213"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_01_0476__ul924042213298"><li id="dli_01_0476__li18316224144314">Existing sensitive variables can only be used by their respective creators. Other common global variables are shared by users under the same account and project.</li><li id="dli_01_0476__li990193218295">Do not use global variables in <strong id="dli_01_0476__b587151315372">OPTIONS</strong> of the table creation statements.</li></ul>
<div class="note" id="dli_01_0476__note183371023201213"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0476__p13541133612379">Do not use global variables in <strong id="dli_01_0476__b587151315372">OPTIONS</strong> of the table creation statements.</p>
</div></div>
</li></ol>
</div>
<div class="section" id="dli_01_0476__section17117296169"><h4 class="sectiontitle">Modifying Variables</h4><p id="dli_01_0476__p1052363621613">On the <strong id="dli_01_0476__b1736614509483">Global Variables</strong> page, click <strong id="dli_01_0476__b83679506483">Modify</strong> in the <strong id="dli_01_0476__b8367125011487">Operation</strong> column of a variable to modify the variable value.</p>
<div class="section" id="dli_01_0476__section17117296169"><h4 class="sectiontitle">Modifying a Global Variable</h4><p id="dli_01_0476__p1052363621613">On the <strong id="dli_01_0476__b1736614509483">Global Variables</strong> page, click <strong id="dli_01_0476__b83679506483">Modify</strong> in the <strong id="dli_01_0476__b8367125011487">Operation</strong> column of a variable to modify the variable value.</p>
<div class="note" id="dli_01_0476__note15151759141116"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0476__p10551113410448">If there are multiple global variables with the same name in the same project under an account, delete the redundant global variables to ensure that the global variables are unique in the same project. In this case, all users who have the permission to modify the global variables can change the variable values.</p>
</div></div>
</div>
<div class="section" id="dli_01_0476__section15942174351618"><h4 class="sectiontitle">Deleting Variables</h4><p id="dli_01_0476__p10226649151619">On the <strong id="dli_01_0476__b736112538487">Global Variables</strong> page, click <strong id="dli_01_0476__b13621953104814">Delete</strong> in the <strong id="dli_01_0476__b1036211532486">Operation</strong> column of a variable to delete the variable value.</p>
<div class="section" id="dli_01_0476__section15942174351618"><h4 class="sectiontitle">Deleting a Global Variable</h4><p id="dli_01_0476__p10226649151619">On the <strong id="dli_01_0476__b736112538487">Global Variables</strong> page, click <strong id="dli_01_0476__b13621953104814">Delete</strong> in the <strong id="dli_01_0476__b1036211532486">Operation</strong> column of a variable to delete the variable value.</p>
<div class="note" id="dli_01_0476__note1082305823417"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_01_0476__ul362499184916"><li id="dli_01_0476__li729263804817">If there are multiple global variables with the same name in the same project under an account, delete the global variables created by the user first. If there are only unique global variables, all users who have the delete permission can delete the global variables.</li><li id="dli_01_0476__li26256934919">After a variable is deleted, the variable cannot be used in SQL statements.</li></ul>
</div></div>
</div>

2
docs/dli/umn/dli_01_0487.html
View File

@ -3,7 +3,7 @@
<h1 class="topictitle1">Elastic Queue Scaling</h1>
<div id="body0000001132627173"><div class="section" id="dli_01_0487__section93461821194314"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_01_0487__p685412834314">Elastic scaling can be performed for a newly created queue only when there were jobs running in this queue.</p>
</div>
<div class="section" id="dli_01_0487__section124281747104415"><h4 class="sectiontitle">Precautions</h4><ul id="dli_01_0487__ul799312234711"><li id="dli_01_0487__li1399332124718">If <strong id="dli_01_0487__b13484143125312">Status of queue xxx is assigning, which is not available</strong> is displayed on the <strong id="dli_01_0487__b7484183115319">Elastic Scaling</strong> page, the queue can be scaled only after the queue resources are allocated.</li></ul>
<div class="section" id="dli_01_0487__section124281747104415"><h4 class="sectiontitle">Precautions</h4><ul id="dli_01_0487__ul799312234711"><li id="dli_01_0487__li253742810428">Queues with 16 CUs do not support scale-out or scale-in.</li><li id="dli_01_0487__li6119202221710">Queues with 64 CUs do not support scale-in.</li><li id="dli_01_0487__li1399332124718">If <strong id="dli_01_0487__b13484143125312">Status of queue xxx is assigning, which is not available</strong> is displayed on the <strong id="dli_01_0487__b7484183115319">Elastic Scaling</strong> page, the queue can be scaled only after the queue resources are allocated.</li></ul>
</div>
<div class="section" id="dli_01_0487__section1889631562717"><h4 class="sectiontitle">Scaling Out</h4><p id="dli_01_0487__p1097183618266">If the current queue specifications do not meet service requirements, you can add the number of CUs to scale out the queue.</p>
<div class="note" id="dli_01_0487__note2898825113314"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0487__p7305155319433">Scale-out is time-consuming. After you perform scale-out on the <span class="wintitle" id="dli_01_0487__wintitle1683985816522"><b>Elastic Scaling</b></span> page of DLI, wait for about 10 minutes. The duration is related to the CU amount to add. After a period of time, refresh the <strong id="dli_01_0487__b158461158105216">Queue Management</strong> page and check whether values of <strong id="dli_01_0487__b58478587522">Specifications</strong> and <strong id="dli_01_0487__b2848658185216">Actual CUs</strong> are the same to determine whether the scale-out is successful. Alternatively, on the <strong id="dli_01_0487__b1091191817541">Job Management</strong> page, check the status of the <strong id="dli_01_0487__b179184185545">SCALE_QUEUE</strong> SQL job. If the job status is <strong id="dli_01_0487__b12918161819545">Scaling</strong>, the queue is being scaled out.</p>

6
docs/dli/umn/dli_01_0498.html
View File

@ -3,7 +3,7 @@
<h1 class="topictitle1">(Recommended) Creating a Flink OpenSource SQL Job</h1>
<div id="body0000001078931613"><p id="dli_01_0498__p085764624712">This section describes how to create a Flink OpenSource SQL job.</p>
<p id="dli_01_0498__p11239138144511">DLI Flink OpenSource SQL jobs are fully compatible with the syntax of Flink 1.10 and 1.12 provided by the community. In addition, Redis, GaussDB(DWS), and DIS data source types are added based on the community connector. For details about the syntax and restrictions of Flink SQL DDL, DML, and functions, see <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/" target="_blank" rel="noopener noreferrer">Table API &amp; SQL</a>.</p>
<div class="section" id="dli_01_0498__section9243114805018"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_01_0498__ul23468771316"><li id="dli_01_0498__li434615710136">You have prepared the data input and data output channels. For details, see <a href="dli_01_0454.html">Preparing Flink Job Data</a>.</li><li id="dli_01_0498__li7520181309">Before creating a Flink OpenSource SQL job to access other external data sources, such as OpenTSDB, HBase, Kafka, GaussDB(DWS), RDS, CSS, CloudTable, DCS Redis, and DDS MongoDB, you need to create a cross-source connection to connect the job running queue to the external data source.<ul id="dli_01_0498__ul850415199816"><li id="dli_01_0498__li18451528175">For details about the external data sources that can be accessed by Flink jobs, see <a href="dli_01_0410.html">Cross-Source Analysis Development Methods</a>.</li><li id="dli_01_0498__li28469131945">For details about how to create a datasource connection, see <a href="dli_01_0426.html">Enhanced Datasource Connections</a>.<p id="dli_01_0498__p881561411413">On the <strong id="dli_01_0498__b1715483452718">Resources</strong> &gt; <strong id="dli_01_0498__b15450175204114">Queue Management</strong> page, locate the queue you have created, and choose <strong id="dli_01_0498__b645115144115">More</strong> &gt; <strong id="dli_01_0498__b44511659418">Test Address Connectivity</strong> in the <strong id="dli_01_0498__b13451175144117">Operation</strong> column to check whether the network connection between the queue and the data source is normal. For details, see <a href="dli_01_0489.html">Testing Address Connectivity</a>.</p>
<div class="section" id="dli_01_0498__section9243114805018"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_01_0498__ul23468771316"><li id="dli_01_0498__li434615710136">You have prepared the data input and data output channels. For details, see <a href="dli_01_0454.html">Preparing Flink Job Data</a>.</li><li id="dli_01_0498__li7520181309">Before creating a Flink OpenSource SQL job to access other external data sources, such as OpenTSDB, HBase, Kafka, GaussDB(DWS), RDS, CSS, CloudTable, DCS Redis, and DDS, you need to create a cross-source connection to connect the job running queue to the external data source.<ul id="dli_01_0498__ul850415199816"><li id="dli_01_0498__li18451528175">For details about the external data sources that can be accessed by Flink jobs, see <a href="dli_01_0410.html">Cross-Source Analysis Development Methods</a>.</li><li id="dli_01_0498__li28469131945">For details about how to create a datasource connection, see <a href="dli_01_0426.html">Enhanced Datasource Connections</a>.<p id="dli_01_0498__p881561411413">On the <strong id="dli_01_0498__b1715483452718">Resources</strong> &gt; <strong id="dli_01_0498__b15450175204114">Queue Management</strong> page, locate the queue you have created, and choose <strong id="dli_01_0498__b645115144115">More</strong> &gt; <strong id="dli_01_0498__b44511659418">Test Address Connectivity</strong> in the <strong id="dli_01_0498__b13451175144117">Operation</strong> column to check whether the network connection between the queue and the data source is normal. For details, see <a href="dli_01_0489.html">Testing Address Connectivity</a>.</p>
</li></ul>
</li></ul>
</div>
@ -164,7 +164,9 @@
</div>
<div class="section" id="dli_01_0498__section11791177104913"><h4 class="sectiontitle">Simplified Stream Graph</h4><p id="dli_01_0498__p157037122504">On the OpenSource SQL job editing page, click <strong id="dli_01_0498__b550916143514">Simplified Stream Graph</strong>.</p>
</div>
<div class="section" id="dli_01_0498__section17641917141017"><h4 class="sectiontitle">Static stream graph</h4><p id="dli_01_0498__p11908441737">On the OpenSource SQL job editing page, click <strong id="dli_01_0498__b557133345111">Static Stream Graph</strong>.</p>
<div class="section" id="dli_01_0498__section17641917141017"><h4 class="sectiontitle">Static Stream Graph</h4><p id="dli_01_0498__p11908441737">On the OpenSource SQL job editing page, click <strong id="dli_01_0498__b557133345111">Static Stream Graph</strong>.</p>
<div class="note" id="dli_01_0498__note49401956132817"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0498__p194012568287">If you use a UDF in a Flink OpenSource SQL job, it is not possible to generate a static stream graph.</p>
</div></div>
<p id="dli_01_0498__p79913215586">The <strong id="dli_01_0498__b571116467517">Static Stream Graph</strong> page also allows you to:</p>
<ul id="dli_01_0498__ul7168154916586"><li id="dli_01_0498__li7168184916582">Estimate concurrencies. Click <strong id="dli_01_0498__b620991765916">Estimate Concurrencies</strong> on the <strong id="dli_01_0498__b98210522412">Static Stream Graph</strong> page to estimate concurrencies. Click <strong id="dli_01_0498__b47491917352">Restore Initial Value</strong> to restore the initial value after concurrency estimation.</li><li id="dli_01_0498__li151782516010">Zoom in or out the page.</li><li id="dli_01_0498__li18171025907">Expand or merge operator chains.</li><li id="dli_01_0498__li915967174520">You can edit <strong id="dli_01_0498__b1615017441173">Parallelism</strong>, <strong id="dli_01_0498__b1064313110188">Output rate</strong>, and <strong id="dli_01_0498__b1848142311186">Rate factor</strong>.<ul id="dli_01_0498__ul1748731514"><li id="dli_01_0498__li12484195012504"><strong id="dli_01_0498__b111881141190">Parallelism</strong>: indicates the number of concurrent tasks.</li><li id="dli_01_0498__li79782427517"><strong id="dli_01_0498__b7571132313190">Output rate</strong>: indicates the data traffic of an operator. The unit is piece/s.</li><li id="dli_01_0498__li1240574845118"><strong id="dli_01_0498__b161071637131912">Rate factor</strong>: indicates the retention rate after data is processed by operators. Rate factor = Data output volume of an operator/Data input volume of the operator (Unit: %)</li></ul>
</li></ul>

8
docs/dli/umn/dli_01_0512.html
View File

@ -3,14 +3,8 @@
<h1 class="topictitle1">Developing and Submitting a Spark SQL Job Using the TPC-H Sample Template</h1>
<div id="body0000001220042371"><p id="dli_01_0512__p3438152482217">DLI allows you to customize query templates or save frequently used SQL statements as templates to facilitate SQL operations. After templates are saved, you do not need to write SQL statements. You can directly perform the SQL operations using the templates.</p>
<p id="dli_01_0512__p27712208201">The current system provides various standard TPC-H query statement templates. You can select a template as needed. This example shows how to use a TPC-H template to develop and submit a Spark SQL job.</p>
<p id="dli_01_0512__p292291825719"><a href="#dli_01_0512__section19012773105034">Step 1: Log In to the Management Console</a></p>
<p id="dli_01_0512__p11591724205712"><a href="#dli_01_0512__section21590507141153">Step 2: Execute the TPC-H Sample Template and View the Result</a></p>
<p id="dli_01_0512__p168551319192615">For details about the templates, see SQL Template Management.</p>
<div class="section" id="dli_01_0512__section19012773105034"><a name="dli_01_0512__section19012773105034"></a><a name="section19012773105034"></a><h4 class="sectiontitle">Step 1: Log In to the Management Console</h4><ol id="dli_01_0512__ol33542655111116"><li id="dli_01_0512__li26251138164819">Log in to the management console.</li><li id="dli_01_0512__li2096805415940">Click <strong id="dli_01_0512__b1375114615101">Service List</strong> and choose <strong id="dli_01_0512__b12909851111014">Analytics</strong> &gt; <strong id="dli_01_0512__b6616105581019">Data Lake Insight</strong>.<div class="note" id="dli_01_0512__note231512513316"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_01_0512__p231925193114">You need to perform authorization when accessing the DLI management console for the first time. For details, see "Global Configuration" &gt; "Service Authorization" in <em id="dli_01_0512__i1680013311151">Data Lake Insight User Guide</em>.</p>
</div></div>
</li></ol>
</div>
<div class="section" id="dli_01_0512__section21590507141153"><a name="dli_01_0512__section21590507141153"></a><a name="section21590507141153"></a><h4 class="sectiontitle">Step 2: Execute the TPC-H Sample Template and View the Result</h4><ol id="dli_01_0512__ol39318463153849"><li id="dli_01_0512__li237731311316">On the DLI management console, choose <strong id="dli_01_0512__b122621281817">Job Templates</strong> &gt; <strong id="dli_01_0512__b12449158183">SQL Templates</strong>, and click the <strong id="dli_01_0512__b25811917191816">Sample Templates</strong> tab. Locate the <strong id="dli_01_0512__b11586420111811">Q1_Price_summary_report_query</strong> template under <strong id="dli_01_0512__b1755062791819">tpchQuery</strong>, and click <strong id="dli_01_0512__b3909152961818">Execute</strong> in the <strong id="dli_01_0512__b8156932131816">Operation</strong> column. The <strong id="dli_01_0512__b198031535131810">SQL Editor</strong> page is displayed.<p id="dli_01_0512__p1290842593220"></p>
<div class="section" id="dli_01_0512__section21590507141153"><h4 class="sectiontitle">Procedure</h4><ol id="dli_01_0512__ol39318463153849"><li id="dli_01_0512__li52132251577">Log in to the DLI management console.</li><li id="dli_01_0512__li237731311316">On the DLI management console, choose <strong id="dli_01_0512__b122621281817">Job Templates</strong> &gt; <strong id="dli_01_0512__b12449158183">SQL Templates</strong>, and click the <strong id="dli_01_0512__b25811917191816">Sample Templates</strong> tab. Locate the <strong id="dli_01_0512__b11586420111811">Q1_Price_summary_report_query</strong> template under <strong id="dli_01_0512__b1755062791819">tpchQuery</strong>, and click <strong id="dli_01_0512__b3909152961818">Execute</strong> in the <strong id="dli_01_0512__b8156932131816">Operation</strong> column. The <strong id="dli_01_0512__b198031535131810">SQL Editor</strong> page is displayed.<p id="dli_01_0512__p1290842593220"></p>
</li><li id="dli_01_0512__li23198164153933">In the upper part of the editing window, set <strong id="dli_01_0512__b18464102717327">Engine</strong> to <strong id="dli_01_0512__b922973312320">spark</strong>, <strong id="dli_01_0512__b15155355327">Queues</strong> to <strong id="dli_01_0512__b19307193712320">default</strong>, and <strong id="dli_01_0512__b1593184253213">Databases</strong> to <strong id="dli_01_0512__b1562965219321">default</strong>, and click <strong id="dli_01_0512__b139531113336">Execute</strong>.<p id="dli_01_0512__p016617263419"></p>
</li><li id="dli_01_0512__li1588111441197">View the query result in the <strong id="dli_01_0512__b8520192512337">View Result</strong> tab in the lower part of the SQL Editor page.<p id="dli_01_0512__p185671909495"></p>
</li></ol>

3
docs/dli/umn/dli_01_0531.html
View File

@ -113,8 +113,7 @@ CREATE TABLE jdbcSink (
'connector' = 'jdbc',
'url' = "jdbc:mysql://<em id="dli_01_0531__i74171211428"><strong id="dli_01_0531__b54171011126">172.16.0.116:3306</strong></em>/<em id="dli_01_0531__i1417151128"><strong id="dli_01_0531__b8417217217">rds-dliflink</strong></em>", <strong id="dli_01_0531__b114171311529">// </strong> <strong id="dli_01_0531__b8160134719270">testrdsdb indicates the name of the created RDS database. Replace the IP address and port number with those of the RDS for MySQL instance.</strong>
'table-name' = 'orders',
'username' = "<em id="dli_01_0531__i183882014215"><strong id="dli_01_0531__b103810202217">xxxxx</strong></em>", <strong id="dli_01_0531__b2038162011213">// </strong> <strong id="dli_01_0531__b1493611512284">Username of the RDS for MySQL DB instance</strong>
'password'="<em id="dli_01_0531__i1418730324"><strong id="dli_01_0531__b718163016211">xxxxx</strong></em>", <strong id="dli_01_0531__b15181830820">// </strong> <strong id="dli_01_0531__b128262522917">Password of the RDS for MySQL DB instance</strong>
'pwd_auth_name'="<em id="dli_01_0531__i41515154612"><strong id="dli_01_0531__b81518164610">xxxxx</strong></em>", <strong id="dli_01_0531__b4136196124617">// </strong><strong id="dli_01_0531__b467463010463">Name of the datasource authentication of the password type created on DLI. If datasource authentication is used, you do not need to set the username and password for the job.</strong>
'sink.buffer-flush.max-rows' = '1'
);

33
docs/dli/umn/dli_01_0534.html
View File

@ -9,46 +9,47 @@
<div class="section" id="dli_01_0534__section1469261818254"><h4 class="sectiontitle">Notes</h4><ul id="dli_01_0534__ul932795302517"><li id="dli_01_0534__li43271653182520">During dynamic scaling of a Flink job, if queue resources are occupied and the remaining resources are insufficient for starting the job, the job may fail to be restored.</li><li id="dli_01_0534__li532795352513">When the resources that can be used by a Flink job are dynamically scaled in or out, the background job needs to be stopped and then restored from the savepoint. So, the job cannot process data before the restoration is successful.</li><li id="dli_01_0534__li232719539252">Savepoints need to be triggered during scaling. So, you must configure an OBS bucket, save logs, and enable checkpointing.</li><li id="dli_01_0534__li1832717539257">Do not set the scaling detection period to a small value to avoid frequent job start and stop.</li><li id="dli_01_0534__li4327135382519">The restoration duration of a scaling job is affected by the savepoint size. If the savepoint size is large, the restoration may take a long time.</li><li id="dli_01_0534__li25721244265">To adjust the configuration items of dynamic scaling, you need to stop the job, edit the job, and submit the job for the modification to take effect.</li></ul>
</div>
<div class="section" id="dli_01_0534__section283262514278"><h4 class="sectiontitle">Procedure</h4><p id="dli_01_0534__p3274105817283">Dynamic scaling applies to Flink OpenSource SQL and Flink Jar jobs.</p>
<ol id="dli_01_0534__ol1755414673117"><li id="dli_01_0534__li135541164314">Log in to the DLI management console.</li><li id="dli_01_0534__li11743113163119">In the navigation pane on the left, choose <strong id="dli_01_0534__b1052873511205">Job Management</strong> &gt; <strong id="dli_01_0534__b1940783812014">Flink Jobs</strong>.</li><li id="dli_01_0534__li16001432193110">Select the job for which you want to enable dynamic scaling and click <strong id="dli_01_0534__b176671769221">Edit</strong> in the <strong id="dli_01_0534__b17191902220">Operation</strong> column.</li><li id="dli_01_0534__li0364205773115">On the right of the page displayed, click <strong id="dli_01_0534__b132918122712">Runtime Configuration</strong>.</li><li id="dli_01_0534__li36665633410">Configure the following parameters:
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0534__table118871836133413" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Dynamic scaling parameters</caption><thead align="left"><tr id="dli_01_0534__row15262133763418"><th align="left" class="cellrowborder" valign="top" width="29.38%" id="mcps1.3.3.3.5.1.2.4.1.1"><p id="dli_01_0534__p13262143718349">Parameter</p>
<ol id="dli_01_0534__ol1755414673117"><li id="dli_01_0534__li135541164314">Log in to the DLI management console.</li><li id="dli_01_0534__li11743113163119">In the navigation pane on the left, choose <strong id="dli_01_0534__b1052873511205">Job Management</strong> &gt; <strong id="dli_01_0534__b1940783812014">Flink Jobs</strong>.</li><li id="dli_01_0534__li16001432193110">Select the job for which you want to enable dynamic scaling and click <strong id="dli_01_0534__b176671769221">Edit</strong> in the <strong id="dli_01_0534__b17191902220">Operation</strong> column.<ul id="dli_01_0534__ul361777123111"><li id="dli_01_0534__li561718763111">For a Flink OpenSource SQL job, click <strong id="dli_01_0534__b434410219453">Runtime Configuration</strong> on the right to configure dynamic scaling parameters.</li><li id="dli_01_0534__li2617072314">For a Flink Jar job, click the <strong id="dli_01_0534__b33726174618">Runtime Configuration</strong> box to configure dynamic scaling parameters.</li></ul>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0534__table118871836133413" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Dynamic scaling parameters</caption><thead align="left"><tr id="dli_01_0534__row15262133763418"><th align="left" class="cellrowborder" valign="top" width="29.38%" id="mcps1.3.3.3.3.4.2.4.1.1"><p id="dli_01_0534__p13262143718349">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="14.69%" id="mcps1.3.3.3.5.1.2.4.1.2"><p id="dli_01_0534__p6262153733410">Default Value</p>
<th align="left" class="cellrowborder" valign="top" width="14.69%" id="mcps1.3.3.3.3.4.2.4.1.2"><p id="dli_01_0534__p6262153733410">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="55.93%" id="mcps1.3.3.3.5.1.2.4.1.3"><p id="dli_01_0534__p192621137103419">Description</p>
<th align="left" class="cellrowborder" valign="top" width="55.93%" id="mcps1.3.3.3.3.4.2.4.1.3"><p id="dli_01_0534__p192621137103419">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_01_0534__row1726217372341"><td class="cellrowborder" valign="top" width="29.38%" headers="mcps1.3.3.3.5.1.2.4.1.1 "><p id="dli_01_0534__p1726217371344">flink.dli.job.scale.enable</p>
<tbody><tr id="dli_01_0534__row1726217372341"><td class="cellrowborder" valign="top" width="29.38%" headers="mcps1.3.3.3.3.4.2.4.1.1 "><p id="dli_01_0534__p1726217371344">flink.dli.job.scale.enable</p>
</td>
<td class="cellrowborder" valign="top" width="14.69%" headers="mcps1.3.3.3.5.1.2.4.1.2 "><p id="dli_01_0534__p9262173711340">false</p>
<td class="cellrowborder" valign="top" width="14.69%" headers="mcps1.3.3.3.3.4.2.4.1.2 "><p id="dli_01_0534__p9262173711340">false</p>
</td>
<td class="cellrowborder" valign="top" width="55.93%" headers="mcps1.3.3.3.5.1.2.4.1.3 "><p id="dli_01_0534__p11262637183411">Whether to enable dynamic scaling, that is, whether to allow DLI to adjust the resources used by jobs based on job loads and job priorities.</p>
<td class="cellrowborder" valign="top" width="55.93%" headers="mcps1.3.3.3.3.4.2.4.1.3 "><p id="dli_01_0534__p11262637183411">Whether to enable dynamic scaling, that is, whether to allow DLI to adjust the resources used by jobs based on job loads and job priorities.</p>
<p id="dli_01_0534__p11262143703414">If this parameter is set to <strong id="dli_01_0534__b19863123202918">false</strong>, the function is disabled.</p>
<p id="dli_01_0534__p32628379346">If this parameter is set to <strong id="dli_01_0534__b13668143617297">true</strong>, the function is enabled.</p>
<p id="dli_01_0534__p19262163753419">The default value is <strong id="dli_01_0534__b56509152022759">false</strong>.</p>
</td>
</tr>
<tr id="dli_01_0534__row1726293743419"><td class="cellrowborder" valign="top" width="29.38%" headers="mcps1.3.3.3.5.1.2.4.1.1 "><p id="dli_01_0534__p7262537193413">flink.dli.job.scale.interval</p>
<tr id="dli_01_0534__row1726293743419"><td class="cellrowborder" valign="top" width="29.38%" headers="mcps1.3.3.3.3.4.2.4.1.1 "><p id="dli_01_0534__p7262537193413">flink.dli.job.scale.interval</p>
</td>
<td class="cellrowborder" valign="top" width="14.69%" headers="mcps1.3.3.3.5.1.2.4.1.2 "><p id="dli_01_0534__p22621237173419">30</p>
<td class="cellrowborder" valign="top" width="14.69%" headers="mcps1.3.3.3.3.4.2.4.1.2 "><p id="dli_01_0534__p22621237173419">30</p>
</td>
<td class="cellrowborder" valign="top" width="55.93%" headers="mcps1.3.3.3.5.1.2.4.1.3 "><p id="dli_01_0534__p52621637183412">Interval for checking whether to scale the resources for the current job, in minutes. The default value is <strong id="dli_01_0534__b7162343123317">30</strong>. For example, <strong id="dli_01_0534__b113422221355">30</strong> indicates that the job is checked every 30 minutes to determine whether to scale in or out the resources used by the job.</p>
<td class="cellrowborder" valign="top" width="55.93%" headers="mcps1.3.3.3.3.4.2.4.1.3 "><p id="dli_01_0534__p52621637183412">Interval for checking whether to scale the resources for the current job, in minutes. The default value is <strong id="dli_01_0534__b7162343123317">30</strong>. For example, <strong id="dli_01_0534__b113422221355">30</strong> indicates that the job is checked every 30 minutes to determine whether to scale in or out the resources used by the job.</p>
<p id="dli_01_0534__p13262153793411">Note: This configuration is effective only when dynamic scaling is enabled.</p>
</td>
</tr>
<tr id="dli_01_0534__row4262437143415"><td class="cellrowborder" valign="top" width="29.38%" headers="mcps1.3.3.3.5.1.2.4.1.1 "><p id="dli_01_0534__p122621937163415">flink.dli.job.cu.max</p>
<tr id="dli_01_0534__row4262437143415"><td class="cellrowborder" valign="top" width="29.38%" headers="mcps1.3.3.3.3.4.2.4.1.1 "><p id="dli_01_0534__p122621937163415">flink.dli.job.cu.max</p>
</td>
<td class="cellrowborder" valign="top" width="14.69%" headers="mcps1.3.3.3.5.1.2.4.1.2 "><p id="dli_01_0534__p926233773415">Initial CU value</p>
<td class="cellrowborder" valign="top" width="14.69%" headers="mcps1.3.3.3.3.4.2.4.1.2 "><p id="dli_01_0534__p926233773415">Initial CU value</p>
</td>
<td class="cellrowborder" valign="top" width="55.93%" headers="mcps1.3.3.3.5.1.2.4.1.3 "><p id="dli_01_0534__p2262143753414">Maximum number of CUs that can be used by the current job during dynamic scaling. If this parameter is not set, the default value is the initial total number of CUs of the job.</p>
<td class="cellrowborder" valign="top" width="55.93%" headers="mcps1.3.3.3.3.4.2.4.1.3 "><p id="dli_01_0534__p2262143753414">Maximum number of CUs that can be used by the current job during dynamic scaling. If this parameter is not set, the default value is the initial total number of CUs of the job.</p>
<p id="dli_01_0534__p1926212373341">Note: The value of this parameter cannot be smaller than the total number of CUs configured by the user. In addition, this parameter is effective only when dynamic scaling is enabled.</p>
</td>
</tr>
<tr id="dli_01_0534__row1926273715341"><td class="cellrowborder" valign="top" width="29.38%" headers="mcps1.3.3.3.5.1.2.4.1.1 "><p id="dli_01_0534__p92621637173415">flink.dli.job.cu.min</p>
<tr id="dli_01_0534__row1926273715341"><td class="cellrowborder" valign="top" width="29.38%" headers="mcps1.3.3.3.3.4.2.4.1.1 "><p id="dli_01_0534__p92621637173415">flink.dli.job.cu.min</p>
</td>
<td class="cellrowborder" valign="top" width="14.69%" headers="mcps1.3.3.3.5.1.2.4.1.2 "><p id="dli_01_0534__p1126283717341">2</p>
<td class="cellrowborder" valign="top" width="14.69%" headers="mcps1.3.3.3.3.4.2.4.1.2 "><p id="dli_01_0534__p1126283717341">2</p>
</td>
<td class="cellrowborder" valign="top" width="55.93%" headers="mcps1.3.3.3.5.1.2.4.1.3 "><p id="dli_01_0534__p126214373342">Minimum number of CUs that can be used by the current job during dynamic scaling. The default value is <strong id="dli_01_0534__b1997810397146">2</strong>.</p>
<td class="cellrowborder" valign="top" width="55.93%" headers="mcps1.3.3.3.3.4.2.4.1.3 "><p id="dli_01_0534__p126214373342">Minimum number of CUs that can be used by the current job during dynamic scaling. The default value is <strong id="dli_01_0534__b1997810397146">2</strong>.</p>
<p id="dli_01_0534__p5262437163420">Note: The value of this parameter cannot be greater than the total number of CUs configured by the user. In addition, this parameter is effective only when dynamic scaling is enabled.</p>
</td>
</tr>

2
docs/dli/umn/dli_01_0538.html
View File

@ -14,7 +14,7 @@
</li>
<li class="ulchildlink"><strong><a href="dli_07_0002.html">Application Scenarios</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_07_0005.html">Constraints</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_07_0005.html">Constraints and Limitations</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_07_0006.html">Permissions Management</a></strong><br>
</li>

2
docs/dli/umn/dli_01_0550.html
View File

@ -13,7 +13,7 @@
<ul id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_ul37351029162712"><li id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_li12735132932714">Domain name, project name, and project ID<p id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_p1973542918275"><a name="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_li12735132932714"></a><a name="en-us_topic_0206791776_en-us_topic_0040259342_li12735132932714"></a>Log in to the management console using the cloud account, click the username in the upper right corner, select <strong id="dli_01_0550__en-us_topic_0206791776_b18872194484610">My Credentials</strong> from the drop-down list, and obtain the domain name, project name, and project ID on the <strong id="dli_01_0550__en-us_topic_0206791776_b687374413466">My Credentials</strong> page.</p>
</li><li id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_li13735182917270">Quota information, including:<ul id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_ul6735102912273"><li id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_li1073516296277">Service name</li><li id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_li6735152972712">Quota type</li><li id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_li773592992718">Required quota</li></ul>
</li></ul>
<p id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_p126761744182712"><a href="https://docs.otc.t-systems.com/en-us/public/learnmore.html" target="_blank" rel="noopener noreferrer">Learn how to obtain the service hotline and email address.</a></p>
<p id="dli_01_0550__en-us_topic_0206791776_en-us_topic_0040259342_p126761744182712"><a href="https://open-telekom-cloud.com/en/contact" target="_blank" rel="noopener noreferrer">Learn how to obtain the service hotline and email address.</a></p>
</div>
</div>
<div>

2
docs/dli/umn/dli_01_0552.html
View File

@ -53,7 +53,7 @@
<tbody><tr id="dli_01_0552__row153618208396"><td class="cellrowborder" valign="top" width="16.35%" headers="mcps1.3.3.2.5.2.1.2.2.3.1.1 "><p id="dli_01_0552__p1379843245517">Tag key</p>
</td>
<td class="cellrowborder" valign="top" width="83.65%" headers="mcps1.3.3.2.5.2.1.2.2.3.1.2 "><p id="dli_01_0552__p579853215514">You can specify the tag key in either of the following ways:</p>
<ul id="dli_01_0552__ul167988328555"><li id="dli_01_0552__li177982322556">Click the text box for tag key and select a predefined tag key from the drop-down list.<p id="dli_01_0552__p1079818321559"><a name="dli_01_0552__li177982322556"></a><a name="li177982322556"></a>To add a predefined tag, you need to create one on TMS and then select it from the <span class="parmname" id="dli_01_0552__parmname980347454"><b>Tag key</b></span> drop-down list. You can click <strong id="dli_01_0552__b4491062884179">View predefined tags</strong> to go to the <strong id="dli_01_0552__b20231504734179">Predefined Tags</strong> page of the TMS console. Then, click <strong id="dli_01_0552__b19884236014179">Create Tag</strong> in the upper corner of the page to create a predefined tag.</p>
<ul id="dli_01_0552__ul167988328555"><li id="dli_01_0552__li177982322556">Click the text box for tag key and select a predefined tag key from the drop-down list.<p id="dli_01_0552__p1079818321559"><a name="dli_01_0552__li177982322556"></a><a name="li177982322556"></a>To add a predefined tag, you need to create one on TMS and then select it from the <span class="parmname" id="dli_01_0552__parmname1681792971"><b>Tag key</b></span> drop-down list. You can click <strong id="dli_01_0552__b4491062884179">View predefined tags</strong> to go to the <strong id="dli_01_0552__b20231504734179">Predefined Tags</strong> page of the TMS console. Then, click <strong id="dli_01_0552__b19884236014179">Create Tag</strong> in the upper corner of the page to create a predefined tag.</p>
<p id="dli_01_0552__p17798632185518"></p>
</li></ul>
<ul id="dli_01_0552__ul19798133225511"><li id="dli_01_0552__li979816327556">Enter a tag key in the text box.<div class="note" id="dli_01_0552__note137981432165512"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_01_0552__dli_01_0363_p1448010306508_1">A tag key can contain a maximum of 128 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with <strong id="dli_01_0552__dli_01_0363_b544594615499_1">_sys_</strong>.</p>

25
docs/dli/umn/dli_01_0559.html
View File

@ -3,36 +3,33 @@
<h1 class="topictitle1">Creating a Password Datasource Authentication</h1>
<div id="body0000001620822857"><div class="section" id="dli_01_0559__section1255517314463"><h4 class="sectiontitle">Scenario</h4><p id="dli_01_0559__p18861153614387">Create a password datasource authentication on the DLI console to store passwords of the GaussDB(DWS), RDS, DCS, and DDS data sources to DLI. This will allow you to access to the data sources without having to configure a username and password in SQL jobs.</p>
</div>
<div class="section" id="dli_01_0559__section250912864611"><h4 class="sectiontitle">Procedure</h4><ol id="dli_01_0559__ol43051810573"><li id="dli_01_0559__li130418155714">Create a Kafka instance.<p id="dli_01_0559__p4113142115571"><a name="dli_01_0559__li130418155714"></a><a name="li130418155714"></a>When creating a Kafka instance, enable SASL_SSL for Kafka. Once SASL_SSL is enabled, data can be encrypted for transmission, which improves security.</p>
</li><li id="dli_01_0559__li12307189579">Download the authentication credential.<ol type="a" id="dli_01_0559__ol395732985719"><li id="dli_01_0559__li130161812571">Log in to the Kafka console and click a Kafka instance to access its details page.</li><li id="dli_01_0559__li7305183576">In the connection information, find the SSL certificate and click <strong id="dli_01_0559__b173213132915">Download</strong>.</li></ol>
</li><li id="dli_01_0559__li2302018145713">Upload the authentication credential to the OBS bucket.<p id="dli_01_0559__p8436414145916"><a name="dli_01_0559__li2302018145713"></a><a name="li2302018145713"></a></p>
</li><li id="dli_01_0559__li153031865715">Create a datasource authentication.<ol type="a" id="dli_01_0559__ol93301231175919"><li id="dli_01_0559__li161611037195919">Log in to the DLI management console.</li><li id="dli_01_0559__li7242128104311">Choose <strong id="dli_01_0559__b112142712913">Datasource Connections</strong>. On the page displayed, click <strong id="dli_01_0559__b12122720297">Datasource Authentication</strong>.</li><li id="dli_01_0559__li128216271309">Click <strong id="dli_01_0559__b162298407232545">Create</strong>.<p id="dli_01_0559__p19696271803">Configure authentication parameters according to <a href="#dli_01_0559__table154674113817">Table 1</a>.</p>
<div class="section" id="dli_01_0559__section250912864611"><h4 class="sectiontitle">Procedure</h4><ol id="dli_01_0559__ol43051810573"><li id="dli_01_0559__li153031865715">Create a datasource authentication.<ol type="a" id="dli_01_0559__ol93301231175919"><li id="dli_01_0559__li161611037195919">Log in to the DLI management console.</li><li id="dli_01_0559__li7242128104311">Choose <strong id="dli_01_0559__b112142712913">Datasource Connections</strong>. On the page displayed, click <strong id="dli_01_0559__b12122720297">Datasource Authentication</strong>.</li><li id="dli_01_0559__li128216271309">Click <strong id="dli_01_0559__b162298407232545">Create</strong>.<p id="dli_01_0559__p19696271803">Configure authentication parameters according to <a href="#dli_01_0559__table154674113817">Table 1</a>.</p>
<div class="tablenoborder"><a name="dli_01_0559__table154674113817"></a><a name="table154674113817"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0559__table154674113817" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="dli_01_0559__row144714411388"><th align="left" class="cellrowborder" valign="top" width="16.2%" id="mcps1.3.2.2.4.1.3.3.2.3.1.1"><p id="dli_01_0559__p174784113381">Parameter</p>
<div class="tablenoborder"><a name="dli_01_0559__table154674113817"></a><a name="table154674113817"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_01_0559__table154674113817" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="dli_01_0559__row144714411388"><th align="left" class="cellrowborder" valign="top" width="16.2%" id="mcps1.3.2.2.1.1.3.3.2.3.1.1"><p id="dli_01_0559__p174784113381">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="83.8%" id="mcps1.3.2.2.4.1.3.3.2.3.1.2"><p id="dli_01_0559__p4471841163812">Description</p>
<th align="left" class="cellrowborder" valign="top" width="83.8%" id="mcps1.3.2.2.1.1.3.3.2.3.1.2"><p id="dli_01_0559__p4471841163812">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_01_0559__row1818893010186"><td class="cellrowborder" valign="top" width="16.2%" headers="mcps1.3.2.2.4.1.3.3.2.3.1.1 "><p id="dli_01_0559__p180835121817">Type</p>
<tbody><tr id="dli_01_0559__row1818893010186"><td class="cellrowborder" valign="top" width="16.2%" headers="mcps1.3.2.2.1.1.3.3.2.3.1.1 "><p id="dli_01_0559__p180835121817">Type</p>
</td>
<td class="cellrowborder" valign="top" width="83.8%" headers="mcps1.3.2.2.4.1.3.3.2.3.1.2 "><p id="dli_01_0559__p40193541812">Select <strong id="dli_01_0559__b122858781232545">Password</strong>.</p>
<td class="cellrowborder" valign="top" width="83.8%" headers="mcps1.3.2.2.1.1.3.3.2.3.1.2 "><p id="dli_01_0559__p40193541812">Select <strong id="dli_01_0559__b122858781232545">Password</strong>.</p>
</td>
</tr>
<tr id="dli_01_0559__row3471641183820"><td class="cellrowborder" valign="top" width="16.2%" headers="mcps1.3.2.2.4.1.3.3.2.3.1.1 "><p id="dli_01_0559__p194784133819">Authentication Certificate</p>
<tr id="dli_01_0559__row3471641183820"><td class="cellrowborder" valign="top" width="16.2%" headers="mcps1.3.2.2.1.1.3.3.2.3.1.1 "><p id="dli_01_0559__p194784133819">Authentication Certificate</p>
</td>
<td class="cellrowborder" valign="top" width="83.8%" headers="mcps1.3.2.2.4.1.3.3.2.3.1.2 "><p id="dli_01_0559__p1547341143820">Name of the datasource authentication to be created.</p>
<td class="cellrowborder" valign="top" width="83.8%" headers="mcps1.3.2.2.1.1.3.3.2.3.1.2 "><p id="dli_01_0559__p1547341143820">Name of the datasource authentication to be created.</p>
<ul id="dli_01_0559__ul74754143813"><li id="dli_01_0559__li17471241193816">The name can contain only digits, letters, and underscores (_), but cannot contain only digits or start with an underscore (_).</li><li id="dli_01_0559__li649194153815">The name can contain a maximum of 128 characters.</li></ul>
</td>
</tr>
<tr id="dli_01_0559__row174944193814"><td class="cellrowborder" valign="top" width="16.2%" headers="mcps1.3.2.2.4.1.3.3.2.3.1.1 "><p id="dli_01_0559__p1585551512412">Username</p>
<tr id="dli_01_0559__row174944193814"><td class="cellrowborder" valign="top" width="16.2%" headers="mcps1.3.2.2.1.1.3.3.2.3.1.1 "><p id="dli_01_0559__p1585551512412">Username</p>
</td>
<td class="cellrowborder" valign="top" width="83.8%" headers="mcps1.3.2.2.4.1.3.3.2.3.1.2 "><p id="dli_01_0559__p7848141594116">Username for accessing the data source.</p>
<td class="cellrowborder" valign="top" width="83.8%" headers="mcps1.3.2.2.1.1.3.3.2.3.1.2 "><p id="dli_01_0559__p7848141594116">Username for accessing the data source.</p>
</td>
</tr>
<tr id="dli_01_0559__row1049124111387"><td class="cellrowborder" valign="top" width="16.2%" headers="mcps1.3.2.2.4.1.3.3.2.3.1.1 "><p id="dli_01_0559__p8844171516413">Password</p>
<tr id="dli_01_0559__row1049124111387"><td class="cellrowborder" valign="top" width="16.2%" headers="mcps1.3.2.2.1.1.3.3.2.3.1.1 "><p id="dli_01_0559__p8844171516413">Password</p>
</td>
<td class="cellrowborder" valign="top" width="83.8%" headers="mcps1.3.2.2.4.1.3.3.2.3.1.2 "><p id="dli_01_0559__p11843181519415">Password for accessing the data source.</p>
<td class="cellrowborder" valign="top" width="83.8%" headers="mcps1.3.2.2.1.1.3.3.2.3.1.2 "><p id="dli_01_0559__p11843181519415">Password for accessing the data source.</p>
</td>
</tr>
</tbody>

8
docs/dli/umn/dli_01_0561.html
View File

@ -1,9 +1,11 @@
<a name="dli_01_0561"></a><a name="dli_01_0561"></a>
<h1 class="topictitle1">Introduction</h1>
<div id="body0000001570503174"><div class="section" id="dli_01_0561__section145388371448"><h4 class="sectiontitle">What Is Datasource Authentication?</h4><p id="dli_01_0561__p868961725320">Datasource authentication is used to manage authentication information for accessing specified data sources. After datasource authentication is configured, you do not need to repeatedly configure data source authentication information in jobs, improving data source authentication security while enabling DLI to securely access data sources.</p>
<h1 class="topictitle1">Overview</h1>
<div id="body0000001570503174"><div class="section" id="dli_01_0561__section145388371448"><h4 class="sectiontitle">What Is Datasource Authentication?</h4><p id="dli_01_0561__p1764518593914">When analyzing across multiple sources, it is not recommended to configure authentication information directly in a job as it can lead to password leakage. Instead, you are advised to use either Data Encryption Workshop (DEW) or datasource authentication provided by DLI to securely store data source authentication information.</p>
<ul id="dli_01_0561__ul156239498237"><li id="dli_01_0561__li126055019296">DEW is a comprehensive cloud-based encryption service that addresses data security, key security, and complex key management issues. You are advised to use DEW to store authentication information for data sources. </li><li id="dli_01_0561__li10624114917236">Datasource authentication is used to manage authentication information for accessing specified data sources. After datasource authentication is configured, you do not need to repeatedly configure data source authentication information in jobs, improving data source authentication security while enabling DLI to securely access data sources.</li></ul>
<p id="dli_01_0561__p57041040182319">This section describes how to use datasource authentication provided by DLI.</p>
</div>
<div class="section" id="dli_01_0561__section517414134610"><h4 class="sectiontitle">Constraints</h4><ul id="dli_01_0561__ul5202111174615"><li id="dli_01_0561__dli_07_0005_li4956171012143">Compared with datasource authentication provided by DLI, you are advised to use Data Encryption Worksop (DEW) to store data source authentication information.</li><li id="dli_01_0561__dli_07_0005_li5740129958">Only Spark SQL and Flink OpenSource SQL 1.12 jobs support datasource authentication.</li><li id="dli_01_0561__dli_07_0005_li154891441163">DLI supports four types of datasource authentication. Select an authentication type specific to each data source.<ul id="dli_01_0561__dli_07_0005_ul0871132418539"><li id="dli_01_0561__dli_07_0005_li620019356202">CSS: applies to 6.5.4 or later CSS clusters with the security mode enabled.</li><li id="dli_01_0561__dli_07_0005_li1048719385204">Kerberos: applies to MRS security clusters with Kerberos authentication enabled.</li><li id="dli_01_0561__dli_07_0005_li1738715443207">Kafka_SSL: applies to Kafka with SSL enabled.</li><li id="dli_01_0561__dli_07_0005_li11175781668">Password: applies to GaussDB(DWS), RDS, DDS, and DCS.</li></ul>
<div class="section" id="dli_01_0561__section517414134610"><h4 class="sectiontitle">Constraints</h4><ul id="dli_01_0561__ul5202111174615"><li id="dli_01_0561__dli_07_0005_li5740129958">Only Spark SQL and Flink OpenSource SQL 1.12 jobs support datasource authentication.</li><li id="dli_01_0561__dli_07_0005_li154891441163">DLI supports four types of datasource authentication. Select an authentication type specific to each data source.<ul id="dli_01_0561__dli_07_0005_ul0871132418539"><li id="dli_01_0561__dli_07_0005_li620019356202">CSS: applies to 6.5.4 or later CSS clusters with the security mode enabled.</li><li id="dli_01_0561__dli_07_0005_li1048719385204">Kerberos: applies to MRS security clusters with Kerberos authentication enabled.</li><li id="dli_01_0561__dli_07_0005_li1738715443207">Kafka_SSL: applies to Kafka with SSL enabled.</li><li id="dli_01_0561__dli_07_0005_li11175781668">Password: applies to GaussDB(DWS), RDS, DDS, and DCS.</li></ul>
</li></ul>
</div>
<div class="section" id="dli_01_0561__section654944414569"><h4 class="sectiontitle">Datasource Authentication Types</h4><p id="dli_01_0561__p2244194574619">DLI supports four types of datasource authentication. Select an authentication type specific to each data source.</p>

2
docs/dli/umn/dli_01_0567.html
View File

@ -4,7 +4,7 @@
<div id="body0000001641473501"><div class="section" id="dli_01_0567__section262694414374"><h4 class="sectiontitle">DLI Job Type</h4><p id="dli_01_0567__p125261351203715">DLI provides the following job types:</p>
<ul id="dli_01_0567__ul68991322153814"><li id="dli_01_0567__li5899122103814">SQL job: SQL jobs provide you with standard SQL statements and are compatible with Spark SQL and Presto SQL (based on Presto). You can query and analyze heterogeneous data sources on the cloud through visualized APIs, JDBC, ODBC, or Beeline. SQL jobs are compatible with mainstream data formats such as CSV, JSON, Parquet, Carbon, and ORC.</li><li id="dli_01_0567__li3899102233818">Flink job: Flink jobs are real-time streaming big data analysis service jobs running on the public cloud. In full hosting mode, you only need to focus on Stream SQL services and execute jobs instantly without being aware of compute clusters. Flink jobs are fully compatible with Apache Flink APIs.</li><li id="dli_01_0567__li389915226389">Spark job: Spark jobs provide fully-managed Spark compute services. You can submit jobs through the GUI or RESTful APIs. Full-stack Spark jobs, such as Spark Core, DataSet, Streaming, MLlib, and GraphX jobs, are supported.</li></ul>
</div>
<div class="section" id="dli_01_0567__section189421214144012"><h4 class="sectiontitle">Constraints</h4><ul id="dli_01_0567__ul476744912461"><li id="dli_01_0567__dli_07_0005_li27671949124616">Only the latest 100 jobs are displayed on DLI's SparkUI.</li><li id="dli_01_0567__dli_07_0005_li196912773210">A maximum of 1,000 job results can be displayed on the console. To view more or all jobs, export the job data to OBS.</li><li id="dli_01_0567__dli_07_0005_li86314794718">To export job run logs, you must have the permission to access OBS buckets. You need to configure a DLI job bucket on the <strong id="dli_01_0567__dli_07_0005_b167715312233">Global Configuration</strong> &gt; <strong id="dli_01_0567__dli_07_0005_b094841622417">Project</strong> page in advance.</li><li id="dli_01_0567__dli_07_0005_li1187712984716"><strong id="dli_01_0567__dli_07_0005_b29691936142413">View Log</strong> and <strong id="dli_01_0567__dli_07_0005_b149691236162410">Export Log</strong> buttons are not available for synchronization jobs and jobs running on the default queue.</li><li id="dli_01_0567__dli_07_0005_li196535311455">Only Spark jobs support custom images. </li><li id="dli_01_0567__dli_07_0005_li87418459489">An elastic resource pool supports a maximum of 32,000 CUs.</li></ul>
<div class="section" id="dli_01_0567__section189421214144012"><h4 class="sectiontitle">Constraints</h4><ul id="dli_01_0567__ul476744912461"><li id="dli_01_0567__dli_07_0005_li27671949124616">Only the latest 100 jobs are displayed on DLI's SparkUI.</li><li id="dli_01_0567__dli_07_0005_li196912773210">A maximum of 1,000 job results can be displayed on the console. To view more or all jobs, export the job data to OBS.</li><li id="dli_01_0567__dli_07_0005_li86314794718">To export job run logs, you must have the permission to access OBS buckets. You need to configure a DLI job bucket on the <strong id="dli_01_0567__dli_07_0005_b167715312233">Global Configuration</strong> &gt; <strong id="dli_01_0567__dli_07_0005_b094841622417">Project</strong> page in advance.</li><li id="dli_01_0567__dli_07_0005_li1187712984716">The <strong id="dli_01_0567__dli_07_0005_b29691936142413">View Log</strong> button is not available for synchronization jobs and jobs running on the default queue.</li><li id="dli_01_0567__dli_07_0005_li196535311455">Only Spark jobs support custom images. </li><li id="dli_01_0567__dli_07_0005_li87418459489">An elastic resource pool supports a maximum of 32,000 CUs.</li></ul>
</div>
</div>
<div>

6
docs/dli/umn/dli_03_0017.html
View File

@ -1,7 +1,9 @@
<a name="dli_03_0017"></a><a name="dli_03_0017"></a>
<h1 class="topictitle1">How Do I Set the AK/SK for a Queue to Operate an OBS Table?</h1>
<div id="body8662426"><ul id="dli_03_0017__en-us_topic_0206809306_ul0763234018"><li id="dli_03_0017__en-us_topic_0206809306_li4988101981814">If the AK and SK are obtained, set the parameters as follows:<ul id="dli_03_0017__en-us_topic_0206809306_ul1681421101910"><li id="dli_03_0017__en-us_topic_0206809306_li16773264014">Create SparkContext using code<pre class="screen" id="dli_03_0017__en-us_topic_0206809306_screen445105195">val sc: SparkContext = new SparkContext()
<div id="body8662426"><div class="note" id="dli_03_0017__en-us_topic_0206809306_note84662050195919"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_03_0017__en-us_topic_0206809306_p1140569541">Hard-coded or plaintext AK and SK pose significant security risks. To ensure security, encrypt your AK and SK, store them in configuration files or environment variables, and decrypt them when needed.</p>
</div></div>
<ul id="dli_03_0017__en-us_topic_0206809306_ul0763234018"><li id="dli_03_0017__en-us_topic_0206809306_li4988101981814">If the AK and SK are obtained, set the parameters as follows:<ul id="dli_03_0017__en-us_topic_0206809306_ul1681421101910"><li id="dli_03_0017__en-us_topic_0206809306_li16773264014">Create SparkContext using code<pre class="screen" id="dli_03_0017__en-us_topic_0206809306_screen445105195">val sc: SparkContext = new SparkContext()
sc.hadoopConfiguration.set("fs.obs.access.key", ak)
sc.hadoopConfiguration.set("fs.obs.secret.key", sk)</pre>
</li><li id="dli_03_0017__en-us_topic_0206809306_li15659814124010">Create SparkSession using code<pre class="screen" id="dli_03_0017__en-us_topic_0206809306_screen1327819344407">val sparkSession: SparkSession = SparkSession
@ -24,8 +26,6 @@ sc.hadoopConfiguration.set("fs.obs.session.token", sts)</pre>
.getOrCreate()</pre>
</li></ul>
</li></ul>
<div class="note" id="dli_03_0017__en-us_topic_0206809306_note10661101412119"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_03_0017__en-us_topic_0206809306_p64720816231">For security purposes, you are advised not to include the AK and SK information in the OBS path. In addition, if a table is created in the OBS directory, the OBS path specified by the <strong id="dli_03_0017__en-us_topic_0206809306_b1290074917417">Path</strong> field cannot contain the AK and SK information.</p>
</div></div>
</div>
<div>
<div class="familylinks">

2
docs/dli/umn/dli_03_0032.html
View File

@ -6,7 +6,7 @@
<ul id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_ul37351029162712"><li id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_li12735132932714">Domain name, project name, and project ID<p id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_p1973542918275"><a name="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_li12735132932714"></a><a name="en-us_topic_0215567055_en-us_topic_0040259342_li12735132932714"></a>Log in to the management console, click the username in the upper-right corner, choose <strong id="dli_03_0032__en-us_topic_0215567055_b1830113104919">My Credentials</strong>, and obtain the domain name, project name, and project ID.</p>
</li><li id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_li13735182917270">Quota information, including:<ul id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_ul6735102912273"><li id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_li1073516296277">Service name</li><li id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_li6735152972712">Quota type</li><li id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_li773592992718">Required quota</li></ul>
</li></ul>
<p id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_p126761744182712"><a href="https://docs.otc.t-systems.com/en-us/public/learnmore.html" target="_blank" rel="noopener noreferrer">Learn how to obtain the service hotline and email address.</a></p>
<p id="dli_03_0032__en-us_topic_0215567055_en-us_topic_0040259342_p126761744182712"><a href="https://open-telekom-cloud.com/en/contact" target="_blank" rel="noopener noreferrer">Learn how to obtain the service hotline and email address.</a></p>
</div>
</div>
<div>

2
docs/dli/umn/dli_03_0037.html
View File

@ -66,7 +66,7 @@
</li>
<li class="ulchildlink"><strong><a href="dli_03_0105.html">How Do I Locate a Flink Job Running Error?</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_03_0136.html">How Do I Know Whether a Flink Job Can Be Restored from a Checkpoint After Being Restarted?</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_03_0136.html">How Can I Check if a Flink Job Can Be Restored From a Checkpoint After Restarting It?</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_03_0040.html">Why Does DIS Stream Not Exist During Job Semantic Check?</a></strong><br>
</li>

62
docs/dli/umn/dli_03_0044.html
View File

@ -1,9 +1,69 @@
<a name="dli_03_0044"></a><a name="dli_03_0044"></a>
<h1 class="topictitle1">Does a Flink JAR Job Support Configuration File Upload? How Do I Upload a Configuration File?</h1>
<div id="body8662426"><p id="dli_03_0044__en-us_topic_0261373736_p436703917344">Configuration files can be uploaded for user-defined jobs (JAR).</p>
<div id="body8662426"><div class="section" id="dli_03_0044__en-us_topic_0261373736_section1812017404262"><h4 class="sectiontitle">Uploading a Configuration File for a Flink JAR Job</h4><p id="dli_03_0044__en-us_topic_0261373736_p436703917344">Configuration files can be uploaded for user-defined jobs (JAR).</p>
<ol id="dli_03_0044__en-us_topic_0261373736_ol1626774516343"><li id="dli_03_0044__en-us_topic_0261373736_li19615165223414">Upload the configuration file to DLI through <strong id="dli_03_0044__en-us_topic_0261373736_b1283912540351">Package Management</strong>.</li><li id="dli_03_0044__en-us_topic_0261373736_li951010414351">In the <strong id="dli_03_0044__en-us_topic_0261373736_b36456499366">Other Dependencies</strong> area of the Flink JAR job, select the created DLI package.</li><li id="dli_03_0044__en-us_topic_0261373736_li6268204593415">Load the file through <strong id="dli_03_0044__en-us_topic_0261373736_b3618202417370">ClassName.class.getClassLoader().getResource("userData/fileName")</strong> in the code. In the file name, <strong id="dli_03_0044__en-us_topic_0261373736_b17920430173719">fileName</strong> indicates the name of the file to be accessed, and <strong id="dli_03_0044__en-us_topic_0261373736_b82075350378">ClassName</strong> indicates the name of the class that needs to access the file.</li></ol>
</div>
<div class="section" id="dli_03_0044__en-us_topic_0261373736_section4291385272"><h4 class="sectiontitle">Using a Configuration File</h4><ul id="dli_03_0044__en-us_topic_0261373736_ul11302205172912"><li id="dli_03_0044__en-us_topic_0261373736_li0302125113292">Solution 1: Load the file content to the memory in the <strong id="dli_03_0044__en-us_topic_0261373736_b15747173954310">main</strong> function and broadcast the content to each taskmanager. This method is applicable to the scenario where a small number of variables need to be loaded in advance.</li><li id="dli_03_0044__en-us_topic_0261373736_li63027512297">Solution 2: Load the file when initializing the operator in <strong id="dli_03_0044__en-us_topic_0261373736_b1431862812572">open</strong>. A relative or absolute path can be used.<p id="dli_03_0044__en-us_topic_0261373736_p5620103717292">Take Kafka sink as an example. Two files (<strong id="dli_03_0044__en-us_topic_0261373736_b268353912473">userData/kafka-sink.conf</strong> and <strong id="dli_03_0044__en-us_topic_0261373736_b989964411478">userData/client.truststore.jks</strong>) need to be loaded.</p>
<ul id="dli_03_0044__en-us_topic_0261373736_ul2793324324"><li id="dli_03_0044__en-us_topic_0261373736_li169441528113212"><strong id="dli_03_0044__en-us_topic_0261373736_b963173510326">Example of using a relative path:</strong><pre class="screen" id="dli_03_0044__en-us_topic_0261373736_screen1075255119304"><strong id="dli_03_0044__en-us_topic_0261373736_b1460554803110">Relative path: confPath = userData/kafka-sink.conf</strong>
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
initConf();
producer = new KafkaProducer&lt;&gt;(props);
}
private void initConf() {
try {
URL url = DliFlinkDemoDis2Kafka.class.getClassLoader().getResource(confPath);
if (url != null) {
LOGGER.info("kafka main-url: " + url.getFile());
} else {
LOGGER.info("kafka url error......");
}
InputStream inputStream = new BufferedInputStream(new FileInputStream(new File(url.getFile()).getAbsolutePath()));
props.load(new InputStreamReader(inputStream, "UTF-8"));
topic = props.getProperty("topic");
partition = Integer.parseInt(props.getProperty("partition"));
vaildProps();
} catch (Exception e) {
LOGGER.info("load kafka conf failed");
e.printStackTrace();
}
}</pre>
<div class="fignone" id="dli_03_0044__en-us_topic_0261373736_fig113341226163510"><span class="figcap"><b>Figure 1 </b>Example relative path configuration</span><br><span><img id="dli_03_0044__en-us_topic_0261373736_image43341626133516" src="en-us_image_0000001716012053.png"></span></div>
</li><li id="dli_03_0044__en-us_topic_0261373736_li18170201563315"><strong id="dli_03_0044__en-us_topic_0261373736_b17620488330">Example of using an absolute path:</strong><pre class="screen" id="dli_03_0044__en-us_topic_0261373736_screen18148631133115"><strong id="dli_03_0044__en-us_topic_0261373736_b18665103433118">Absolute path: confPath = userData/kafka-sink.conf / path = </strong><strong id="dli_03_0044__en-us_topic_0261373736_b1015152918334">/opt/data1/hadoop/tmp/usercache/omm/appcache/application_xxx_0015/container_xxx_0015_01_000002/userData/client.truststore.jks</strong>
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
initConf();
String path = DliFlinkDemoDis2Kafka.class.getClassLoader().getResource("userData/client.truststore.jks").getPath();
LOGGER.info("kafka abs path " + path);
props.setProperty("ssl.truststore.location", path);
producer = new KafkaProducer&lt;&gt;(props);
}
private void initConf() {
try {
URL url = DliFlinkDemoDis2Kafka.class.getClassLoader().getResource(confPath);
if (url != null) {
LOGGER.info("kafka main-url: " + url.getFile());
} else {
LOGGER.info("kafka url error......");
}
InputStream inputStream = new BufferedInputStream(new FileInputStream(new File(url.getFile()).getAbsolutePath()));
props.load(new InputStreamReader(inputStream, "UTF-8"));
topic = props.getProperty("topic");
partition = Integer.parseInt(props.getProperty("partition"));
vaildProps();
} catch (Exception e) {
LOGGER.info("load kafka conf failed");
e.printStackTrace();
}
}</pre>
<div class="fignone" id="dli_03_0044__en-us_topic_0261373736_fig10193131773610"><span class="figcap"><b>Figure 2 </b>Example absolute path configuration</span><br><span><img id="dli_03_0044__en-us_topic_0261373736_image1928142163712" src="en-us_image_0000001668054032.png"></span></div>
</li></ul>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_03_0037.html">Flink Jobs</a></div>

2
docs/dli/umn/dli_03_0045.html
View File

@ -2,7 +2,7 @@
<h1 class="topictitle1">Why Is the OBS Bucket Selected for Job Not Authorized?</h1>
<div id="body8662426"><p id="dli_03_0045__en-us_topic_0215567060_p10924162312166">If the OBS bucket selected for a job is not authorized, perform the following steps:</p>
<ol id="dli_03_0045__en-us_topic_0215567060_ol529724221613"><li id="dli_03_0045__en-us_topic_0215567060_li1194333912504"><span>On the DLI management console, choose <strong id="dli_03_0045__en-us_topic_0215567060_b14804143719469">Global Configuration</strong> &gt; <strong id="dli_03_0045__en-us_topic_0215567060_b17313105020464">Service Authorization</strong> and select the <strong id="dli_03_0045__en-us_topic_0215567060_b9383616154713">Tenant Administrator (Global service)</strong> permission. Update the permission settings.</span></li><li id="dli_03_0045__en-us_topic_0215567060_li71520517167"><span>In the navigation pane, choose <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade105861714121718"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol18585914141715">Job Management</span></b> &gt; <b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol95903211480">Flink Jobs</span></b></span>.</span></li><li id="dli_03_0045__en-us_topic_0215567060_li13451135115172"><span>In the row where the target job resides, click <span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol1698330494144327"><b>Edit</b></span> under <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1825117119197"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol4251819196">Operation</span></b></span> to switch to the <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1620314105193"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol17203131051913">Edit</span></b></span> page.</span></li><li id="dli_03_0045__en-us_topic_0215567060_li766441552319"><span>Configure parameters under <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1851946243144422"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol1810455313144422">Running Parameters</span></b></span> on the <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1666513155235"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol19665415192313">Edit</span></b></span> page.</span><p><ol type="a" id="dli_03_0045__en-us_topic_0215567060_ol27719205232"><li id="dli_03_0045__en-us_topic_0215567060_li132971442101613">Select <span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol296416412208"><b>Enable Checkpointing</b></span> or <span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol36661159152010"><b>Save Job Log</b></span>.</li><li id="dli_03_0045__en-us_topic_0215567060_li1493119502119">Specify <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1229616177226"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol22967171221">OBS Bucket</span></b></span>.</li><li id="dli_03_0045__en-us_topic_0215567060_li179202394222">Select <span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol1262713513239"><b>Authorize OBS</b></span>.</li></ol>
<ol id="dli_03_0045__en-us_topic_0215567060_ol529724221613"><li id="dli_03_0045__en-us_topic_0215567060_li1194333912504"><span>On the DLI management console, choose <strong id="dli_03_0045__en-us_topic_0215567060_b14804143719469">Global Configuration</strong> &gt; <strong id="dli_03_0045__en-us_topic_0215567060_b17313105020464">Service Authorization</strong> and select the <strong id="dli_03_0045__en-us_topic_0215567060_b9383616154713">Tenant Administrator (Global service)</strong> permission.</span></li><li id="dli_03_0045__en-us_topic_0215567060_li71520517167"><span>In the navigation pane, choose <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade105861714121718"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol18585914141715">Job Management</span></b> &gt; <b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol95903211480">Flink Jobs</span></b></span>.</span></li><li id="dli_03_0045__en-us_topic_0215567060_li13451135115172"><span>In the row where the target job resides, click <span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol1698330494144327"><b>Edit</b></span> under <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1825117119197"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol4251819196">Operation</span></b></span> to switch to the <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1620314105193"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol17203131051913">Edit</span></b></span> page.</span></li><li id="dli_03_0045__en-us_topic_0215567060_li766441552319"><span>Configure parameters under <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1851946243144422"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol1810455313144422">Running Parameters</span></b></span> on the <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1666513155235"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol19665415192313">Edit</span></b></span> page.</span><p><ol type="a" id="dli_03_0045__en-us_topic_0215567060_ol27719205232"><li id="dli_03_0045__en-us_topic_0215567060_li132971442101613">Select <span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol296416412208"><b>Enable Checkpointing</b></span> or <span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol36661159152010"><b>Save Job Log</b></span>.</li><li id="dli_03_0045__en-us_topic_0215567060_li1493119502119">Specify <span class="menucascade" id="dli_03_0045__en-us_topic_0215567060_menucascade1229616177226"><b><span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol22967171221">OBS Bucket</span></b></span>.</li><li id="dli_03_0045__en-us_topic_0215567060_li179202394222">Select <span class="uicontrol" id="dli_03_0045__en-us_topic_0215567060_uicontrol1262713513239"><b>Authorize OBS</b></span>.</li></ol>
</p></li></ol>
</div>
<div>

2
docs/dli/umn/dli_03_0056.html
View File

@ -7,7 +7,7 @@
<ul class="ullinks">
<li class="ulchildlink"><strong><a href="dli_03_0060.html">Why Is Error "unsupported media Type" Reported When I Subimt a SQL Job?</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_03_0125.html">Are Project IDs of Different Accounts the Same When They Are Used to Call APIs?</a></strong><br>
<li class="ulchildlink"><strong><a href="dli_03_0125.html">Is the Project ID Fixed when Different IAM Users Call an API?</a></strong><br>
</li>
<li class="ulchildlink"><strong><a href="dli_03_0178.html">What Can I Do If an Error Is Reported When the Execution of the API for Creating a SQL Job Times Out?</a></strong><br>
</li>

4
docs/dli/umn/dli_03_0071.html
View File

@ -1,8 +1,8 @@
<a name="dli_03_0071"></a><a name="dli_03_0071"></a>
<h1 class="topictitle1">Why Is Error "The current account does not have permission to perform this operation,the current account was restricted. Restricted for no budget." Reported when a SQL Statement Is Executed?</h1>
<div id="body8662426"><p id="dli_03_0071__en-us_topic_0277546211_p3906617224">Check whether the account is in arrears. If yes, recharge the account.</p>
<p id="dli_03_0071__en-us_topic_0277546211_p8060118">If the error persists, log out and log in again.</p>
<div id="body8662426"><p id="dli_03_0071__en-us_topic_0277546211_p3906617224">Check if your account is in arrears and top it up if necessary.</p>
<p id="dli_03_0071__en-us_topic_0277546211_p8060118">If the same error message persists after the top-up, log out of your account and log back in.</p>
</div>
<div>
<div class="familylinks">

2
docs/dli/umn/dli_03_0072.html
View File

@ -2,7 +2,7 @@
<h1 class="topictitle1">How Do I Synchronize DLI Table Data from One Region to Another?</h1>
<div id="body8662426"><p id="dli_03_0072__en-us_topic_0277728030_p6136629763">You can use the cross-region replication function of OBS. The procedure is as follows:</p>
<ol id="dli_03_0072__en-us_topic_0277728030_ol519011371617"><li id="dli_03_0072__en-us_topic_0277728030_li18525174118610">Export the DLI table data in region 1 to the user-defined OBS bucket. For details, see "Exporting Data from DLI to OBS" in <em id="dli_03_0072__en-us_topic_0277728030_i2075575165014">Data Lake Insight User Guide</em>.</li><li id="dli_03_0072__en-us_topic_0277728030_li2019012371068">Use the OBS cross-region replication function to replicate data to the OBS bucket in region 2. For details, see "Configuring Cross-Region Replication" in <em id="dli_03_0072__en-us_topic_0277728030_i6777934192813">Data Lake Insight User Guide</em>.</li><li id="dli_03_0072__en-us_topic_0277728030_li1507347168">Import or use the corresponding data as required.</li></ol>
<ol id="dli_03_0072__en-us_topic_0277728030_ol519011371617"><li id="dli_03_0072__en-us_topic_0277728030_li143064390202">Export the DLI table data in region 1 to the user-defined OBS bucket.</li><li id="dli_03_0072__en-us_topic_0277728030_li158216404202">Use the OBS cross-region replication function to replicate data to the OBS bucket in region 2.</li><li id="dli_03_0072__en-us_topic_0277728030_li1507347168">Import or use the corresponding data as required.</li></ol>
</div>
<div>
<div class="familylinks">

2
docs/dli/umn/dli_03_0082.html
View File

@ -2,7 +2,7 @@
<h1 class="topictitle1">How Do I Run a Complex PySpark Program in DLI?</h1>
<div id="body8662426"><p id="dli_03_0082__en-us_topic_0296781189_p137711611151310">DLI natively supports PySpark.</p>
<p id="dli_03_0082__en-us_topic_0296781189_p09411456143214">For most cases, Python is preferred for data analysis, and PySpark is the best choice for big data analysis. Generally, JVM programs are packed into JAR files and depend on third-party JAR files. Similarly, Python programs also depend on third-party libraries, especially big data analysis programs related to PySpark-based converged machine learning. Traditionally, the Python library is installed on the execution machine based on pip. For serverless services such as DLI, you do not need to and are unaware of the underlying compute resources. In this case, how does DLI ensure that you run their programs perfectly?</p>
<p id="dli_03_0082__en-us_topic_0296781189_p09411456143214">For most cases, Python is preferred for data analysis, and PySpark is the best choice for big data analysis. Generally, JVM programs are packed into JAR files and depend on third-party JAR files. Similarly, Python programs also depend on third-party libraries, especially big data analysis programs related to PySpark-based converged machine learning. Traditionally, Python libraries are installed directly on the execution machine using pip. However, for serverless services like DLI, users do not need to and are not aware of the underlying compute resources. How can we ensure that users can run their programs more effectively?</p>
<p id="dli_03_0082__en-us_topic_0296781189_p1970594033110">DLI has built-in algorithm libraries for machine learning in its compute resources. These common algorithm libraries meet the requirements of most users. What if a user's PySpark program depends on a program library that is not provided by the built-in algorithm library? Actually, the dependency of PySpark is specified based on PyFiles. On the DLI Spark job page, you can directly select the Python third-party program library (such as ZIP and EGG) stored on OBS.</p>
<p id="dli_03_0082__en-us_topic_0296781189_p756318461067">The compressed package of the dependent third-party Python library has structure requirements. For example, if the PySpark program depends on moduleA (import moduleA), the compressed package must meet the following structure requirement:</p>
<div class="fignone" id="dli_03_0082__en-us_topic_0296781189_fig17718221978"><span class="figcap"><b>Figure 1 </b>Compressed package structure requirement</span><br><span><img id="dli_03_0082__en-us_topic_0296781189_image147762216717" src="en-us_image_0296823520.png"></span></div>

56
docs/dli/umn/dli_03_0100.html
View File

@ -2,52 +2,62 @@
<h1 class="topictitle1">How Do I Manage Fine-Grained DLI Permissions?</h1>
<div id="body8662426"><p id="dli_03_0100__en-us_topic_0000001103929830_p8060118">DLI has a comprehensive permission control mechanism and supports fine-grained authentication through Identity and Access Management (IAM). You can create policies in IAM to manage DLI permissions.</p>
<p id="dli_03_0100__en-us_topic_0000001103929830_p188991750134717">With IAM, you can use your account to create IAM users for your employees, and assign permissions to the users to control their access to specific resource types. For example, some software developers in your enterprise need to use DLI resources but must not delete them or perform any high-risk operations. To achieve this result, you can create IAM users for the software developers and grant them only the permissions required for using DLI resources.</p>
<p id="dli_03_0100__en-us_topic_0000001103929830_p188991750134717">IAM allows you to create IAM users for your employees using your cloud account, and assign permissions to control their access to specific types of resources. For example, some software developers in your enterprise need to use DLI resources but must not delete them or perform any high-risk operations. To achieve this result, you can create IAM users for the software developers and grant them only the permissions required for using DLI resources.</p>
<div class="note" id="dli_03_0100__en-us_topic_0000001103929830_note17965103310292"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_03_0100__en-us_topic_0000001103929830_p2966183352919">For a new user, you need to log in for the system to record the metadata before using DLI.</p>
</div></div>
<p id="dli_03_0100__en-us_topic_0000001103929830_p446382154810">IAM can be used free of charge. You pay only for the resources in your account. </p>
<p id="dli_03_0100__en-us_topic_0000001103929830_p8642155123119">If the account has met your requirements, you do not need to create an independent IAM user for permission management. Then you can skip this section. This will not affect other functions of DLI.</p>
<div class="section" id="dli_03_0100__en-us_topic_0000001103929830_section0826035122314"><h4 class="sectiontitle">DLI System Permissions</h4><p id="dli_03_0100__en-us_topic_0000001103929830_p81616496240"><a href="#dli_03_0100__en-us_topic_0000001103929830_table6578220217">Table 1</a> lists all the system-defined roles and policies supported by DLI.</p>
<div class="p" id="dli_03_0100__en-us_topic_0000001103929830_p934171813107">You can grant users permissions by using roles and policies.<ul id="dli_03_0100__en-us_topic_0000001103929830_ul2804101433117"><li id="dli_03_0100__en-us_topic_0000001103929830_li28043143315">Roles: A type of coarse-grained authorization that defines permissions related to user responsibilities. This mechanism provides only a limited number of service-level roles for authorization. When using roles to grant permissions, you need to also assign other roles on which the permissions depend to take effect. Roles are not an ideal choice for fine-grained authorization and secure access control.</li><li id="dli_03_0100__en-us_topic_0000001103929830_li480481493115">Policies: A type of fine-grained authorization that defines permissions required to perform operations on specific cloud resources under certain conditions. This type of authorization is more flexible and ideal for secure access control. For example, you can grant DLI users only the permissions for managing a certain type of cloud servers.</li></ul>
<p id="dli_03_0100__en-us_topic_0000001103929830_p446382154810">IAM is free to use, and you only need to pay for the resources in your account. </p>
<p id="dli_03_0100__en-us_topic_0000001103929830_p8642155123119">If your cloud account does not need individual IAM users for permissions management, skip over this section.</p>
<div class="section" id="dli_03_0100__en-us_topic_0000001103929830_section0826035122314"><h4 class="sectiontitle">DLI System Permissions</h4><div class="p" id="dli_03_0100__en-us_topic_0000001103929830_p934171813107">You can grant users permissions by using roles and policies.<ul id="dli_03_0100__en-us_topic_0000001103929830_ul2804101433117"><li id="dli_03_0100__en-us_topic_0000001103929830_li28043143315">Roles: A type of coarse-grained authorization that defines permissions related to user responsibilities. This mechanism provides only a limited number of service-level roles for authorization. When using roles to grant permissions, you need to also assign other roles on which the permissions depend to take effect. Roles are not an ideal choice for fine-grained authorization and secure access control.</li><li id="dli_03_0100__en-us_topic_0000001103929830_li480481493115">Policies: A type of fine-grained authorization that defines permissions required to perform operations on specific cloud resources under certain conditions. This type of authorization is more flexible and ideal for secure access control. For example, you can grant DLI users only the permissions for managing a certain type of cloud servers.</li></ul>
</div>
<div class="tablenoborder"><a name="dli_03_0100__en-us_topic_0000001103929830_table6578220217"></a><a name="en-us_topic_0000001103929830_table6578220217"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_03_0100__en-us_topic_0000001103929830_table6578220217" frame="border" border="1" rules="all"><caption><b>Table 1 </b>DLI system permissions</caption><thead align="left"><tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_row65805262114"><th align="left" class="cellrowborder" valign="top" width="22.650000000000002%" id="mcps1.3.6.4.2.4.1.1"><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p458015212216">Role/Policy Name</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_03_0100__table6578220217" frame="border" border="1" rules="all"><caption><b>Table 1 </b>DLI system permissions</caption><thead align="left"><tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_row1916124942412"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.6.3.2.5.1.1"><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p1616154914249">Role/Policy Name</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="53.290000000000006%" id="mcps1.3.6.4.2.4.1.2"><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p205811524219">Description</p>
<th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.6.3.2.5.1.2"><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p616194952410">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="24.060000000000002%" id="mcps1.3.6.4.2.4.1.3"><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p115811821211">Category</p>
<th align="left" class="cellrowborder" valign="top" width="15%" id="mcps1.3.6.3.2.5.1.3"><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p116204916243">Category</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="35%" id="mcps1.3.6.3.2.5.1.4"><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_p1176119524410">Dependency</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_row6581172142115"><td class="cellrowborder" valign="top" width="22.650000000000002%" headers="mcps1.3.6.4.2.4.1.1 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p85811726215">DLI FullAccess</p>
<tbody><tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_row9615165162914"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.3.2.5.1.1 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p05661101252">DLI FullAccess</p>
</td>
<td class="cellrowborder" valign="top" width="53.290000000000006%" headers="mcps1.3.6.4.2.4.1.2 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p358110262111">Full permissions for DLI.</p>
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.3.2.5.1.2 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p8566410102511">Full permissions for DLI.</p>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.6.4.2.4.1.3 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p17581142102119">System-defined policy</p>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.6.3.2.5.1.3 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p165664105256">System-defined policy</p>
</td>
<td class="cellrowborder" valign="top" width="35%" headers="mcps1.3.6.3.2.5.1.4 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_p18800101284715">This role depends on other roles in the same project.</p>
<ul id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_ul1091313219455"><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_li891493219452">Creating a datasource connection: <strong id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_b1788683318268">VPC ReadOnlyAccess</strong></li><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_li29147324457">Creating a tag: <strong id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_b18233181519277">TMS FullAccess</strong> and <strong id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_b1355392014275">EPS EPS FullAccess</strong></li><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_li189141032174519">Using OBS for storage: <strong id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_b72961547142720">OBS OperateAccess</strong></li><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_li891415323457">Creating an agency: <strong id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_b1673095210279">Security Administrator</strong></li></ul>
</td>
</tr>
<tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_row14583162182113"><td class="cellrowborder" valign="top" width="22.650000000000002%" headers="mcps1.3.6.4.2.4.1.1 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p2584823217">DLI ReadOnlyAccess</p>
<tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_row9786108162910"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.3.2.5.1.1 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p145301015112510">DLI ReadOnlyAccess</p>
</td>
<td class="cellrowborder" valign="top" width="53.290000000000006%" headers="mcps1.3.6.4.2.4.1.2 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p125841218213">Read-only permissions for DLI.</p>
<p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_en-us_topic_0206791772_p275775023011">With read-only permissions, you can use DLI resources and perform operations that do not require fine-grained permissions. For example, create global variables, create packages and package groups, submit jobs to the default queue, create tables in the default database, create datasource connections, and delete datasource connections.</p>
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.3.2.5.1.2 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p2530915172514">Read-only permissions for DLI.</p>
<p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p275775023011">With read-only permissions, you can use DLI resources and perform operations that do not require fine-grained permissions. For example, create global variables, create packages and package groups, submit jobs to the default queue, create tables in the default database, create datasource connections, and delete datasource connections.</p>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.6.4.2.4.1.3 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p55853242118">System-defined policy</p>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.6.3.2.5.1.3 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p18530915162514">System-defined policy</p>
</td>
<td class="cellrowborder" valign="top" width="35%" headers="mcps1.3.6.3.2.5.1.4 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_p776118517446">None</p>
</td>
</tr>
<tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_row145857219214"><td class="cellrowborder" valign="top" width="22.650000000000002%" headers="mcps1.3.6.4.2.4.1.1 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p658519213213">Tenant Administrator</p>
<tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_row12852521175716"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.3.2.5.1.1 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p1285232111571">Tenant Administrator</p>
</td>
<td class="cellrowborder" valign="top" width="53.290000000000006%" headers="mcps1.3.6.4.2.4.1.2 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p1558511202116">Tenant administrator</p>
<ul id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_ul65851021215"><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_li558612252112">Administer permissions for managing and accessing all cloud services. After a database or a queue is created, the user can use the ACL to assign rights to other users.</li><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_li10586728212">Scope: project-level service</li></ul>
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.3.2.5.1.2 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p18853152175711">Tenant administrator</p>
<ul id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_ul1231319105816"><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_li17311619195817">Job execution permissions for DLI resources. After a database or a queue is created, the user can use the ACL to assign rights to other users.</li><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_li1032519125814">Scope: project-level service</li></ul>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.6.4.2.4.1.3 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p195864262119">System-defined role</p>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.6.3.2.5.1.3 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p12853142111579">System-defined role</p>
</td>
<td class="cellrowborder" valign="top" width="35%" headers="mcps1.3.6.3.2.5.1.4 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_p2076114514415">None</p>
</td>
</tr>
<tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_row195865282116"><td class="cellrowborder" valign="top" width="22.650000000000002%" headers="mcps1.3.6.4.2.4.1.1 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p758610252116">DLI Service Admin</p>
<tr id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_row1917749132415"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.6.3.2.5.1.1 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p79511462325">DLI Service Administrator</p>
</td>
<td class="cellrowborder" valign="top" width="53.290000000000006%" headers="mcps1.3.6.4.2.4.1.2 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p75869213212">DLI administrator</p>
<ul id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_ul1858617219211"><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_li85869272111">Administer permissions for managing and accessing the queues and data of DLI. After a database or a queue is created, the user can use the ACL to assign rights to other users.</li><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_li15586828211">Scope: project-level service</li></ul>
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.6.3.2.5.1.2 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p49787271109">DLI administrator.</p>
<ul id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_ul973112019319"><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_li147319201035">Job execution permissions for DLI resources. After a database or a queue is created, the user can use the ACL to assign rights to other users.</li><li id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_li167321520337">Scope: project-level service</li></ul>
</td>
<td class="cellrowborder" valign="top" width="24.060000000000002%" headers="mcps1.3.6.4.2.4.1.3 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_p6586142182114">System-defined role</p>
<td class="cellrowborder" valign="top" width="15%" headers="mcps1.3.6.3.2.5.1.3 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_en-us_topic_0206791772_p5263816561">System-defined role</p>
</td>
<td class="cellrowborder" valign="top" width="35%" headers="mcps1.3.6.3.2.5.1.4 "><p id="dli_03_0100__en-us_topic_0000001103929830_dli_01_0440_dli_07_0006_p177619574410">None</p>
</td>
</tr>
</tbody>

Some files were not shown because too many files have changed in this diff Show More