vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_1732.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

346 lines
42 KiB
HTML
Raw Blame History

<a name="mrs_01_1732"></a><a name="mrs_01_1732"></a>
<h1 class="topictitle1">Configuring Resource Groups</h1>
<div id="body32001227"><div class="section" id="mrs_01_1732__en-us_topic_0000001173631190_section1741811610132"><h4 class="sectiontitle">Resource Group Introduction</h4><p id="mrs_01_1732__en-us_topic_0000001173631190_p1442582041313">The resource group mechanism controls the overall query load of the instance from the perspective of resource allocation and implements queuing policies for queries. Multiple resource groups can be created under a compute instance resource, and each submitted query is assigned to a specific resource group for execution. Before a resource group executes a new query, it checks whether the resource load of the current resource group exceeds the amount of resources allocated to it by the instance. If it is exceeded, new incoming queries are blocked, placed in a queue, or even rejected directly. However, the resource component does not cause the running query to fail.</p>
</div>
<div class="section" id="mrs_01_1732__en-us_topic_0000001173631190_section10485144051315"><h4 class="sectiontitle">Application Scenarios of Resource Groups</h4><p id="mrs_01_1732__en-us_topic_0000001173631190_p7399142514488">Resource groups are used to manage resources in compute instances. Different resource groups are allocated to different users and queries to isolate resources. This prevents a single user or query from exclusively occupying resources in the compute instance. In addition, the weight and priority of resource components can be configured to ensure that important tasks are executed first. <a href="#mrs_01_1732__en-us_topic_0000001173631190_table14837640204918">Table 1</a> describes the typical application scenarios of resource groups.</p>
<div class="tablenoborder"><a name="mrs_01_1732__en-us_topic_0000001173631190_table14837640204918"></a><a name="en-us_topic_0000001173631190_table14837640204918"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1732__en-us_topic_0000001173631190_table14837640204918" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Typical application scenarios of resource groups</caption><thead align="left"><tr id="mrs_01_1732__en-us_topic_0000001173631190_row1837114034914"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.2.3.2.3.1.1"><p id="mrs_01_1732__en-us_topic_0000001173631190_p15837164024920">Typical Scenarios</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.2.3.2.3.1.2"><p id="mrs_01_1732__en-us_topic_0000001173631190_p28373409492">Solution</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1732__en-us_topic_0000001173631190_row138371409494"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1783734064919">As the number of business teams using the compute instance increases, there is no resource when a team's task becomes more important and does not want to execute a query.</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p18372040164914">Allocate a specified resource group to each team. Important tasks are assigned to resource groups with more resources. When the sum of the proportions of sub-resource groups is less than or equal to 100%, the resources of a queue cannot be preempted by other resource groups. This is similar to static resource allocation.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row208371040144917"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p15837144024920">When the instance resource load is high, two users submit a query at the same time. At the beginning, both queries are queuing. When there are idle resources, the query of a specific user can be scheduled to obtain resources first.</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1383720402494">Two users are allocated with different resource groups. Important tasks can be allocated to resource groups with higher weights or priorities. The scheduling policy is configured by schedulingPolicy. Different scheduling policies have different resource allocation sequences.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row283794020497"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p18837840154916">For ad hoc queries and batch queries, resources can be allocated more properly based on different SQL types.</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p22371421202420">You can match different resource groups for different query types, such as EXPLAIN, INSERT, SELECT, and DATA_DEFINITION, and allocate different resources to execute the query.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="mrs_01_1732__en-us_topic_0000001173631190_section83605362220"><h4 class="sectiontitle">Enabling a Resource Group</h4><p id="mrs_01_1732__en-us_topic_0000001173631190_p59183616255">When creating a compute instance, add custom configuration parameters to the <span class="filepath" id="mrs_01_1732__en-us_topic_0000001173631190_filepath38111117214"><b>resource-groups.json</b></span> file. For details, see <a href="mrs_01_1731.html#mrs_01_1731__en-us_topic_0000001173631202_li135621231135216">3.e</a> in <a href="mrs_01_1731.html">Creating HetuEngine Compute Instances</a>.</p>
</div>
<div class="section" id="mrs_01_1732__en-us_topic_0000001173631190_section112695622518"><a name="mrs_01_1732__en-us_topic_0000001173631190_section112695622518"></a><a name="en-us_topic_0000001173631190_section112695622518"></a><h4 class="sectiontitle">Resource Group Properties</h4><p id="mrs_01_1732__en-us_topic_0000001173631190_p10551175818252">For details about how to configure resource group attributes, see <a href="#mrs_01_1732__en-us_topic_0000001173631190_table1961772415811">Table 2</a>.</p>
</div>
<div class="tablenoborder"><a name="mrs_01_1732__en-us_topic_0000001173631190_table1961772415811"></a><a name="en-us_topic_0000001173631190_table1961772415811"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1732__en-us_topic_0000001173631190_table1961772415811" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Resource group properties</caption><thead align="left"><tr id="mrs_01_1732__en-us_topic_0000001173631190_row1261812435810"><th align="left" class="cellrowborder" valign="top" width="22.472247224722473%" id="mcps1.3.5.2.4.1.1"><p id="mrs_01_1732__en-us_topic_0000001173631190_p4618162405820">Configuration Item</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.080908090809082%" id="mcps1.3.5.2.4.1.2"><p id="mrs_01_1732__en-us_topic_0000001173631190_p16618724135818">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="68.44684468446845%" id="mcps1.3.5.2.4.1.3"><p id="mrs_01_1732__en-us_topic_0000001173631190_p1861812435816">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1732__en-us_topic_0000001173631190_row1561852485811"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p861862416584">name</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1361832455813">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p7618112425811">Resource group name</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row1361832416584"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p9618924175817">maxQueued</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p25774144013">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p116188243589">Maximum number of queued queries. When this threshold is reached, new queries will be rejected.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row10618122419581"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1461802414582">hardConcurrencyLimit</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p9618102419581">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1661819245586">Maximum number of running queries.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row961822455814"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p06189247585">softMemoryLimit</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p061852425815">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p3618102417586">Maximum memory usage of a resource group. When the memory usage reaches this threshold, new tasks are queued. The value can be an absolute value (for example, 10 GB) or a percentage (for example, 10% of the cluster memory).</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row136181024125811"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1961802419583">softCpuLimit</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p36181224155814">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p661862415813">The CPU time that can be used in a period (see the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b330416313577">cpuQuotaPeriod </strong>parameter in <a href="#mrs_01_1732__en-us_topic_0000001173631190_section6811173510322">Global Attributes</a>). You must also specify the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b1967221435819">hardCpuLimit </strong>parameter. When the threshold is reached, the CPU resources occupied by the query that occupies the maximum CPU resources in the resource group are reduced.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row1361882415815"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p161822445812">hardCpuLimit</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1661818244584">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p12618152413582">Maximum CPU time that can be used in a period.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row3636164718586"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p463615479585">schedulingPolicy</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p26369472584">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p156368477584">The scheduling policy for a specific query from the queuing state to the running state</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul11700121141313"><li id="mrs_01_1732__en-us_topic_0000001173631190_li8700141113136">fair (default)</li></ul>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p19751102821212">When multiple sub-resource groups in a resource group have queuing queries, the sub-resource groups obtain resources in turn based on the defined sequence. The query of the same sub-resource group obtains resources based on the first-come-first-executed rule.</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul10669217171314"><li id="mrs_01_1732__en-us_topic_0000001173631190_li18669181717139">weighted_fair</li></ul>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p521234813300">The <strong id="mrs_01_1732__en-us_topic_0000001173631190_b11504141715118">schedulingWeight </strong>attribute is configured for each resource group that uses this policy. Each sub-resource group calculates a ratio: <em id="mrs_01_1732__en-us_topic_0000001173631190_i12547113418116">Number of queried sub-resource groups</em>/Scheduling weight. A sub-resource group with a smaller ratio obtains resources first.</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul3272122220134"><li id="mrs_01_1732__en-us_topic_0000001173631190_li192721322131312">weighted</li></ul>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p97525289121">The default value is <strong id="mrs_01_1732__en-us_topic_0000001173631190_b3806346812">1</strong>. A larger value of <strong id="mrs_01_1732__en-us_topic_0000001173631190_b4464538112">schedulingWeight </strong>indicates that resources are obtained earlier.</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul23151043133"><li id="mrs_01_1732__en-us_topic_0000001173631190_li14315184131314">query_priority</li></ul>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p87525284123">All sub-resource groups must be set with <strong id="mrs_01_1732__en-us_topic_0000001173631190_b941183924">query_priority</strong>. Resources are obtained in the sequence specified by <strong id="mrs_01_1732__en-us_topic_0000001173631190_b169421311524">query_priority</strong>.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row61174546581"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1311711547583">schedulingWeight</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p13117185418589">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p101171654115815">Weight of the group. For details, see <strong id="mrs_01_1732__en-us_topic_0000001173631190_b0585831621">schedulingPolicy</strong>. The default value is <strong id="mrs_01_1732__en-us_topic_0000001173631190_b1352043714210">1</strong>.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row9119205965813"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p7119115985817">jmxExport</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p15119125975817">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1111905919587">If this parameter is set to <strong id="mrs_01_1732__en-us_topic_0000001173631190_b16306426212">true</strong>, group statistics are exported to the JMX for monitoring. The default value is <strong id="mrs_01_1732__en-us_topic_0000001173631190_b772102916318">false</strong>.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row2618152416580"><td class="cellrowborder" valign="top" width="22.472247224722473%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p261862417581">subGroups</p>
</td>
<td class="cellrowborder" valign="top" width="9.080908090809082%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p8618524165819">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.44684468446845%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p18618192419580">Subgroup list</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="mrs_01_1732__en-us_topic_0000001173631190_section510615136297"><h4 class="sectiontitle">Selector Rules</h4><p id="mrs_01_1732__en-us_topic_0000001173631190_p81511159290">The selector matches resource groups in sequence. The first matched resource group is used. Generally, you are advised to configure a default resource group. If no default resource group is configured and other resource group selector conditions are not met, the query will be rejected. For details about how to set selector rule parameters, see <a href="#mrs_01_1732__en-us_topic_0000001173631190_table6836551111718">Table 3</a>.</p>
</div>
<div class="tablenoborder"><a name="mrs_01_1732__en-us_topic_0000001173631190_table6836551111718"></a><a name="en-us_topic_0000001173631190_table6836551111718"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1732__en-us_topic_0000001173631190_table6836551111718" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Selector rules</caption><thead align="left"><tr id="mrs_01_1732__en-us_topic_0000001173631190_row183685121715"><th align="left" class="cellrowborder" valign="top" width="22.24222422242224%" id="mcps1.3.7.2.4.1.1"><p id="mrs_01_1732__en-us_topic_0000001173631190_p17836105191711">Configuration Item</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.43094309430943%" id="mcps1.3.7.2.4.1.2"><p id="mrs_01_1732__en-us_topic_0000001173631190_p1836135181713">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="68.32683268326832%" id="mcps1.3.7.2.4.1.3"><p id="mrs_01_1732__en-us_topic_0000001173631190_p16836175131716">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1732__en-us_topic_0000001173631190_row183610518178"><td class="cellrowborder" valign="top" width="22.24222422242224%" headers="mcps1.3.7.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p12836651101716">user</p>
</td>
<td class="cellrowborder" valign="top" width="9.43094309430943%" headers="mcps1.3.7.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p63127550196">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.32683268326832%" headers="mcps1.3.7.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p7292115511916">Regular expression for matching the user name.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row11836751181711"><td class="cellrowborder" valign="top" width="22.24222422242224%" headers="mcps1.3.7.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p52781723162018">source</p>
</td>
<td class="cellrowborder" valign="top" width="9.43094309430943%" headers="mcps1.3.7.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p17691448132014">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.32683268326832%" headers="mcps1.3.7.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p158361651161717">Data source to be matched with, such as JDBC, HBase, and Hive. For details, see the value of <strong id="mrs_01_1732__en-us_topic_0000001173631190_b1591233213225">--source</strong> in <a href="#mrs_01_1732__en-us_topic_0000001173631190_section1347672818362">Configuration of Selector Attributes</a>.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row283614516178"><td class="cellrowborder" valign="top" width="22.24222422242224%" headers="mcps1.3.7.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p118361451191713">queryType</p>
</td>
<td class="cellrowborder" valign="top" width="9.43094309430943%" headers="mcps1.3.7.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p136451248152015">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.32683268326832%" headers="mcps1.3.7.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p5836165181718">Task types:</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul353455675910"><li id="mrs_01_1732__en-us_topic_0000001173631190_li1534256185919"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b134952417851451">DATA_DEFINITION</strong>: indicates that you can modify, create, or delete the metadata of schemas, tables, and views, and manage the query of prepared statements, permissions, sessions, and transactions.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li13534165655913"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b51595627751451">DELETE</strong>: indicates the DELETE queries.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li953445645920"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b108255348351451">DESCRIBE</strong>: indicates the DESCRIBE, DESCRIBE INPUT, DESCRIBE OUTPUT, and SHOW queries.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li165341556185918"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b198659100551451">EXPLAIN</strong>: indicates the EXPLAIN queries.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li0534165665918"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b761417176617">INSERT</strong>: indicates the INSERT and CREATE TABLE AS queries.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li175341656175911"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b137243848051451">SELECT</strong>: indicates the SELECT queries.</li></ul>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row4836105181715"><td class="cellrowborder" valign="top" width="22.24222422242224%" headers="mcps1.3.7.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1183614519170">clientTags</p>
</td>
<td class="cellrowborder" valign="top" width="9.43094309430943%" headers="mcps1.3.7.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p41638492205">No</p>
</td>
<td class="cellrowborder" valign="top" width="68.32683268326832%" headers="mcps1.3.7.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p128363513171">Match client tag to be matched with. Each tag must be in the tag list of the task submitted by the user. For details, see the value of <strong id="mrs_01_1732__en-us_topic_0000001173631190_b1562819112519">--client-tags</strong> in <a href="#mrs_01_1732__en-us_topic_0000001173631190_section1347672818362">Configuration of Selector Attributes</a>.</p>
</td>
</tr>
<tr id="mrs_01_1732__en-us_topic_0000001173631190_row18367512179"><td class="cellrowborder" valign="top" width="22.24222422242224%" headers="mcps1.3.7.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p3836175112170">group</p>
</td>
<td class="cellrowborder" valign="top" width="9.43094309430943%" headers="mcps1.3.7.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p14836155110178">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="68.32683268326832%" headers="mcps1.3.7.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p1083625191716">The resource group with running queries</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="mrs_01_1732__en-us_topic_0000001173631190_section6811173510322"><a name="mrs_01_1732__en-us_topic_0000001173631190_section6811173510322"></a><a name="en-us_topic_0000001173631190_section6811173510322"></a><h4 class="sectiontitle">Global Attributes</h4><p id="mrs_01_1732__en-us_topic_0000001173631190_p1218311012331">For details about how to configure global attributes, see <a href="#mrs_01_1732__en-us_topic_0000001173631190_table159281244203010">Table 4</a>.</p>
<div class="tablenoborder"><a name="mrs_01_1732__en-us_topic_0000001173631190_table159281244203010"></a><a name="en-us_topic_0000001173631190_table159281244203010"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1732__en-us_topic_0000001173631190_table159281244203010" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Global attributes</caption><thead align="left"><tr id="mrs_01_1732__en-us_topic_0000001173631190_row159291644113011"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.8.3.2.4.1.1"><p id="mrs_01_1732__en-us_topic_0000001173631190_p1392904419302">Configuration Item</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="19.901990199019902%" id="mcps1.3.8.3.2.4.1.2"><p id="mrs_01_1732__en-us_topic_0000001173631190_p2929844143014">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="46.764676467646765%" id="mcps1.3.8.3.2.4.1.3"><p id="mrs_01_1732__en-us_topic_0000001173631190_p199295443307">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1732__en-us_topic_0000001173631190_row392912440306"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.8.3.2.4.1.1 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p16929444153017">cpuQuotaPeriod</p>
</td>
<td class="cellrowborder" valign="top" width="19.901990199019902%" headers="mcps1.3.8.3.2.4.1.2 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p092954443013">No</p>
</td>
<td class="cellrowborder" valign="top" width="46.764676467646765%" headers="mcps1.3.8.3.2.4.1.3 "><p id="mrs_01_1732__en-us_topic_0000001173631190_p11929044203010">Time range during which the CPU quota takes effect. This parameter is used together with <strong id="mrs_01_1732__en-us_topic_0000001173631190_b124170591598">softCpuLimit </strong>and <strong id="mrs_01_1732__en-us_topic_0000001173631190_b1776170191012">hardCpuLimit </strong>in <a href="#mrs_01_1732__en-us_topic_0000001173631190_section112695622518">Resource Group Properties</a>.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="mrs_01_1732__en-us_topic_0000001173631190_section1347672818362"><a name="mrs_01_1732__en-us_topic_0000001173631190_section1347672818362"></a><a name="en-us_topic_0000001173631190_section1347672818362"></a><h4 class="sectiontitle">Configuration of Selector Attributes</h4><p id="mrs_01_1732__en-us_topic_0000001173631190_p216093344019">The data source name (<strong id="mrs_01_1732__en-us_topic_0000001173631190_b184971184100">source</strong>) can be set as follows:</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul659841144013"><li id="mrs_01_1732__en-us_topic_0000001173631190_li10470194704010"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b201280771751451">CLI</strong>: Use the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b109451879951451">--source</strong> option.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li65912418405"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b49252581451451">JDBC</strong>: Set the ApplicationName client information property on the Connection instance.</li></ul>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p17206166184113">The client tag (<strong id="mrs_01_1732__en-us_topic_0000001173631190_b71971538201013">clientTags</strong>) can be configured as follows:</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul439114816411"><li id="mrs_01_1732__en-us_topic_0000001173631190_li1590561374113"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b113265029451451">CLI</strong>: Use the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b133950613951451">--client-tags</strong> option.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li039218817417"><strong id="mrs_01_1732__en-us_topic_0000001173631190_b117819481851451">JDBC</strong>: Set the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b203784524751451">ClientTags client info</strong> property on the Connection instance.</li></ul>
</div>
<div class="section" id="mrs_01_1732__en-us_topic_0000001173631190_section8376165718367"><h4 class="sectiontitle">Configuration Example</h4><div class="fignone" id="mrs_01_1732__en-us_topic_0000001173631190_fig89011425373"><a name="mrs_01_1732__en-us_topic_0000001173631190_fig89011425373"></a><a name="en-us_topic_0000001173631190_fig89011425373"></a><span class="figcap"><b>Figure 1 </b>Configuration example</span><br><span><img id="mrs_01_1732__en-us_topic_0000001173631190_image4187168184315" src="en-us_image_0000001296219396.png"></span></div>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p15250133944615">As shown in <a href="#mrs_01_1732__en-us_topic_0000001173631190_fig89011425373">Figure 1</a>.</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul1414835017399"><li id="mrs_01_1732__en-us_topic_0000001173631190_li6148650103912">For the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b107711313151315">global </strong>resource group, a maximum of 100 queries can be executed at the same time. 1000 queries are in the queuing state. The <strong id="mrs_01_1732__en-us_topic_0000001173631190_b143812561317">global </strong>resource group has three sub-resource groups: <strong id="mrs_01_1732__en-us_topic_0000001173631190_b20497142911316">data_definition</strong>, <strong id="mrs_01_1732__en-us_topic_0000001173631190_b187531230141320">adhoc</strong>, and <strong id="mrs_01_1732__en-us_topic_0000001173631190_b14825323134">pipeline</strong>.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li0410153203917">Each user in the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b147001638101317">pipeline </strong>resource group can run a maximum of five queries at the same time, which occupy 50% of the memory resources of the pipeline resource group. By default, the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b432491191414">fair </strong>scheduling policy is used in the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b1170553191415">pipeline </strong>resource group. Therefore, the query is executed in the sequence of "first come, first served".</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li15816175613916">To make full use of instance resources, the total memory quota of all child resource groups can be greater than that of the parent resource group. For example, the sum of the memory quota of the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b1493518438148">global </strong>resource group (80%) and that of the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b155541546151415">admin </strong>resource group (100%) is 180%, which is greater than 100%.</li></ul>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p1353505645913">In the following example configuration, there are multiple resource groups, some of which are templates. HetuEngine administrators can use templates to dynamically build a resource group tree. For example, in the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b81565065851451">pipeline_</strong><em id="mrs_01_1732__en-us_topic_0000001173631190_i140309843751451">${USER}</em> group, <em id="mrs_01_1732__en-us_topic_0000001173631190_i4945341551451">${USER}</em> is the name of the user who submits a query. <em id="mrs_01_1732__en-us_topic_0000001173631190_i53242578251451">${SOURCE}</em> is also supported, which will be the source where a query is submitted later. You can also use custom variables in <strong id="mrs_01_1732__en-us_topic_0000001173631190_b86765845151451">source</strong> expressions and <strong id="mrs_01_1732__en-us_topic_0000001173631190_b174305797551451">user</strong> regular expressions.</p>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p18388171215433">The following is an example of a resource group selector:</p>
<pre class="screen" id="mrs_01_1732__en-us_topic_0000001173631190_screen51501312164417">"selectors": [{
"user": "bob",
"group": "admin"
},
{
"source": ".*pipeline.*",
"queryType": "DATA_DEFINITION",
"group": "global.data_definition"
},
{
"source": ".*pipeline.*",
"group": "global.pipeline.pipeline_${USER}"
},
{
"source": "jdbc#(?&lt;toolname&gt;.*)",
"clientTags": ["hipri"],
"group": "global.adhoc.bi-${toolname}.${USER}"
},
{
"group": "global.adhoc.other.${USER}"
}]</pre>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p1853511561595">There are four selectors that define which resource group to run the query:</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul2157412164510"><li id="mrs_01_1732__en-us_topic_0000001173631190_li290313181457">The first selector matches queries from <strong id="mrs_01_1732__en-us_topic_0000001173631190_b37283998251451">bob</strong> and places them in the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b170771122451451">admin</strong> group.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li2577173354512">The second selector matches all data definition language (DDL) queries from the source name that includes the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b195828868751451">pipeline</strong> and places them in the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b100700487851451">global.data_definition</strong> group. This helps reduce the queuing time of such queries.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li49681828104713">The third selector matches queries from sources that include the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b205191139551451">pipeline</strong> and places them in a single-user pipe group that is dynamically created under the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b10396264751451">global.pipeline</strong> group.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li138018373476">The fourth selector matches queries from BI tools whose sources match the regular expression <strong id="mrs_01_1732__en-us_topic_0000001173631190_b188579736651451">jdbc#(?&lt;toolname&gt;.*)</strong>, and the tags provided by the client are the superset of <strong id="mrs_01_1732__en-us_topic_0000001173631190_b96642590751451">hi-pri</strong>. These files are placed in the subgroups that are dynamically created under the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b12058062151451">global.pipeline.tools</strong> group. Dynamic subgroups are created based on the naming variable <strong id="mrs_01_1732__en-us_topic_0000001173631190_b18801950951451">toolname</strong>, which is extracted from the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b32392439551451">source</strong> regular expression. Consider queries that use the query containing the source <strong id="mrs_01_1732__en-us_topic_0000001173631190_b61658271451451">jdbc#powerfulbi</strong>, user <strong id="mrs_01_1732__en-us_topic_0000001173631190_b211094951351451">kayla</strong>, and client tags, <strong id="mrs_01_1732__en-us_topic_0000001173631190_b125138764651451">hipri</strong> and <strong id="mrs_01_1732__en-us_topic_0000001173631190_b177297010851451">fast</strong>. This query will be routed to the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b106905765351451">global.pipeline.bi-powerfulbi.kayla</strong> resource group.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li7157712104514">The last selector is a default selector that puts all the unmatched queries into the resource group.</li></ul>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p14535195665910">These selectors work together to implement the following policies:</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul4535656175915"><li id="mrs_01_1732__en-us_topic_0000001173631190_li205351356165910">HetuEngine administrator <strong id="mrs_01_1732__en-us_topic_0000001173631190_b191202648051451">bob</strong> can run 50 queries at the same time. The query will run according to the user-supplied priority.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li121056526477">For the remaining users:<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul20535135611592"><li id="mrs_01_1732__en-us_topic_0000001173631190_li5535125685917">The total number of concurrent queries cannot exceed 100.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li1253515568597">You can use the source pipeline to run a maximum of five concurrent DDL queries. The query is performed in the FIFO sequence.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li153575615916">Non-DDL queries are executed in the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b141871276951451">global.pipeline</strong> group. The total number of concurrent queries is 45, and each user can run 5 queries concurrently. The query is performed in the FIFO sequence.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li55351856185912">Each BI tool can run a maximum of 10 concurrent queries, and each user can run a maximum of three concurrent queries. If the total number of concurrent queries exceeds 10, the user who runs the least queries gets the next concurrency slot. This policy makes it fairer to compete for resources.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li18535856115912">All the remaining queries are placed in each of the user groups under <strong id="mrs_01_1732__en-us_topic_0000001173631190_b171416126851451">global.adhoc.other</strong>.</li></ul>
</li></ul>
<p id="mrs_01_1732__en-us_topic_0000001173631190_p8602834818">The description of the query match selector is as follows:</p>
<ul id="mrs_01_1732__en-us_topic_0000001173631190_ul1590586484"><li id="mrs_01_1732__en-us_topic_0000001173631190_li198913816489">Each pair of braces represents a selector that matches the resource group. Five selectors are configured to match the five resource groups.<pre class="screen" id="mrs_01_1732__en-us_topic_0000001173631190_screen198918164820">admin
global.data_definition
global.pipeline.pipeline_${USER}
global.adhoc.bi-${toolname}.${USER
global.adhoc.other.${USER}</pre>
</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li8899814811">Only when all the conditions of the selector are met, the task can be put into the current queue for execution. For example, if user <strong id="mrs_01_1732__en-us_topic_0000001173631190_b773317455170">amy</strong> submits a query request in JDBC mode and <strong id="mrs_01_1732__en-us_topic_0000001173631190_b817135218171">clientTags </strong>is not configured, the query request cannot be allocated to the resource group<strong id="mrs_01_1732__en-us_topic_0000001173631190_b3808556181811"> global.adhoc.bi-${toolname}.${USER}</strong>.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li69018817481">When a query can meet the conditions of two selectors at the same time, the first selector that meets the requirements is matched. For example, if the <strong id="mrs_01_1732__en-us_topic_0000001173631190_b375710234197">bob </strong>user submits a DATA_DEFINITION job whose source is <strong id="mrs_01_1732__en-us_topic_0000001173631190_b126201228111920">pipeline</strong>, only the resource corresponding to the resource group <strong id="mrs_01_1732__en-us_topic_0000001173631190_b134141036191910">admin </strong>is matched, not the resource corresponding to <strong id="mrs_01_1732__en-us_topic_0000001173631190_b1299444131911">global.data_definition</strong>.</li><li id="mrs_01_1732__en-us_topic_0000001173631190_li589118497516">If none of the four selectors is matched, resources in the resource group <strong id="mrs_01_1732__en-us_topic_0000001173631190_b14444171042010">global.adhoc.other.${USER}</strong> specified by the last selector are used. This resource group functions as a default resource group. If the default resource group is not set and does not meet the conditions of other resource group selectors, the resource group will be rejected.<p id="mrs_01_1732__en-us_topic_0000001173631190_p86769545211">The following is a complete example:</p>
<pre class="screen" id="mrs_01_1732__en-us_topic_0000001173631190_screen145814610526">{
"rootGroups": [{
"name": "global",
"softMemoryLimit": "80%",
"hardConcurrencyLimit": 100,
"maxQueued": 1000,
"schedulingPolicy": "weighted",
"jmxExport": true,
"subGroups": [{
"name": "data_definition",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 5,
"maxQueued": 100,
"schedulingWeight": 1
},
{
"name": "adhoc",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 50,
"maxQueued": 1,
"schedulingWeight": 10,
"subGroups": [{
"name": "other",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 2,
"maxQueued": 1,
"schedulingWeight": 10,
"schedulingPolicy": "weighted_fair",
"subGroups": [{
"name": "${USER}",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 1,
"maxQueued": 100
}]
},
{
"name": "bi-${toolname}",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 10,
"maxQueued": 100,
"schedulingWeight": 10,
"schedulingPolicy": "weighted_fair",
"subGroups": [{
"name": "${USER}",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 3,
"maxQueued": 10
}]
}]
},
{
"name": "pipeline",
"softMemoryLimit": "80%",
"hardConcurrencyLimit": 45,
"maxQueued": 100,
"schedulingWeight": 1,
"jmxExport": true,
"subGroups": [{
"name": "pipeline_${USER}",
"softMemoryLimit": "50%",
"hardConcurrencyLimit": 5,
"maxQueued": 100
}]
}]
},
{
"name": "admin",
"softMemoryLimit": "100%",
"hardConcurrencyLimit": 50,
"maxQueued": 100,
"schedulingPolicy": "query_priority",
"jmxExport": true
}],
"selectors": [{
"user": "bob",
"group": "admin"
},
{
"source": ".*pipeline.*",
"queryType": "DATA_DEFINITION",
"group": "global.data_definition"
},
{
"source": ".*pipeline.*",
"group": "global.pipeline.pipeline_${USER}"
},
{
"source": "jdbc#(?&lt;toolname&gt;.*)",
"clientTags": ["hipri"],
"group": "global.adhoc.bi-${toolname}.${USER}"
},
{
"group": "global.adhoc.other.${USER}"
}],
"cpuQuotaPeriod": "1h"
}</pre>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1729.html">Managing Compute Instances</a></div>
</div>
</div>
View Git Blame Copy Permalink