Configuring Scaling Policies for Queues

Scenario

A company has multiple departments that perform data analysis in different periods during a day.

• Department A requires a large number of compute resources from 00:00 a.m. to 09:00 a.m. In other time segments, only small tasks are running.
• Department B requires a large number of compute resources from 10:00 a.m. to 10:00 p.m. Some periodical tasks are running in other time segments during a day.

In the preceding scenario, you can add two queues to an elastic resource pool: queue test_a for department A, and queue test_b for department B. You can add scaling policies for 00:00-09:00 and 10:00-23:00 respectively to the test_a and test_b queues. For jobs in other periods, you can modify the default scaling policy.

Precautions

• In any time segment of a day, the total minimum CUs of all queues in an elastic resource pool cannot be more than the minimum CUs of the pool.
• In any time segment of a day, the maximum CUs of any queue in an elastic resource pool cannot be more than the maximum CUs of the pool.
• The periods of scaling policies cannot overlap.
• The period of a scaling policy can only be set by hour and specified by the start time and end time. For example, if you set the period to 00-09, the period when the policy takes effect is [00:00, 09:00). The period of the default scaling policy cannot be modified.
• In any period, compute resources are preferentially allocated to meet the minimum number of CUs of all queues. The remaining CUs (maximum CUs of the elastic resource pool – total minimum CUs of all queues) are allocated in accordance with the scaling policy priorities.
  • A scaling policy with a smaller priority value (for example, 1) is prior to those with a bigger priority value (for example, 100).
  • If the scaling policies of two queues have the same priority, resources are randomly allocated to a queue. If there are remaining resources, they are randomly allocated until there is no more left.

    Table 2 CU allocation

    Scenario	CUs
    An elastic resource pool has a maximum number of 256 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–9:00; minimum CU: 32; maximum CU: 128 Queue B: priority 10; time period: 00:00–9:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are 160 CUs remaining. The remaining CUs are allocated based on the priorities. Queue B is prior to queue A. Therefore, queue B gets 64 CUs, and queue A has 96 CUs.
    An elastic resource pool has a maximum number of 96 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–9:00; minimum CU: 32; maximum CU: 64 Queue B: priority 10; time period: 00:00–9:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are no remaining CUs. The allocation is complete.
    An elastic resource pool has a maximum number of 128 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–9:00; minimum CU: 32; maximum CU: 64 Queue B: priority 10; time period: 00:00–9:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are 32 CUs remaining. The remaining 32 CUs are all preferentially allocated to queue B.
    An elastic resource pool has a maximum number of 128 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–9:00; minimum CU: 32; maximum CU: 64 Queue B: priority 5; time period: 00:00–9:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are 32 CUs remaining. The two queues have the same priority, the remaining 32 CUs are randomly allocated to the two queues.

Setting a Scaling Policy

1. Log in to the DLI management console and create an elastic resource pool. Set the minimum and maximum number of CUs of the pool to 128 and 256 respectively. For details, see Creating an Elastic Resource Pool.
2. Choose Resources > Resource Pool. Locate the row that contains the created elastic resource pool, and click Queue MGMT in the Operation column.
3. Refer to Adding a Queue to create the test_a queue and set the scaling policy.
   1. Set the priority of the default scaling policy to 5, Min CU to 16, and Max CU to 32.
   2. Click create to add a scaling policy. Set the priority to 20, Period to 00--09, Min CU to 64, and Max CU to 128.

4. View the scaling policy on the Queue Management page of the specific elastic resource pool.

   Click to view graphical statistics of priorities and CU settings for all time segments.

5. Refer to Adding a Queue to create the test_b queue and set the scaling policy.
   1. Set the priority of the default scaling policy to 5, Min CU to 32, and Max CU to 64.
   2. Click create to add a scaling policy. Set the priority to 20, Period to 10--23, Min CU to 64, and Max CU to 128.

6. View the scaling policy on the Queue Management page of the specific elastic resource pool.

   Click to view graphical statistics on priorities and CU settings of the two queues for all time segments.