Upon activation of the managed scaling feature, the system persistently monitors the load of the Yarn cluster, calculating fluctuations in peak load over the recent ten-minute period, thereby automatically supplementing or reducing task nodes. Managed scaling is exclusively applicable to cluster types incorporating Yarn components.
Basic settings
The basic settings delineate the range of node numbers for the expansion and contraction of the managed scaling feature, as well as the minimum node count for pay-as-you-go billing.
Minimum Node Count: When the managed contraction strategy is triggered, the cluster retains at least a certain number of task nodes for elastic scaling.
Maximum Node Count: When the managed expansion strategy is triggered, the cluster retains at most a certain number of task nodes for elastic scaling. The cumulative expansion count of a single or multiple specifications cannot exceed the maximum node count.
Minimum Node Count for Pay-as-you-go Billing: This refers to the least number of pay-as-you-go nodes to be expanded after the expansion is triggered. It is used to set the proportion of pay-as-you-go nodes and bidding instances, defaulting to the maximum node count.
For instance
With the minimum node count set to 0, the maximum node count set to 100, and the minimum pay-as-you-go node count set to 10, upon triggering expansion, at least 10 pay-as-you-go nodes will be expanded. The remainder will be supplemented by bidding instances. If there are insufficient bidding instances, they will be supplemented by pay-as-you-go billing.
Scaling specification management
Scaling specifications refer to the node specifications that can be expanded through managed scaling. Each cluster can configure up to five scaling specifications. When the expansion rule is triggered, expansion will be carried out according to the priority of the specifications. If the quantity of high-priority specifications is insufficient, the next priority resource specifications will supplement the high-priority specifications for expansion to supplement computing resources. To maintain the linear change of cluster load, it is recommended to keep the CPU and memory of the scaling specifications consistent. The managed scaling function only supports the host resource type.
Nodes within the scaling specifications support addition, deletion, modification, and inquiry, allowing for the adjustment of scaling specification priority as needed.
The expansion order for the five specifications is as follows (the execution order for pay-as-you-go and bidding instances is the same):
When resources are abundant: 1 > 2 > 3 > 4 > 5
For instance
Given five preset specifications and sufficient resources, when the expansion rule is triggered and requires the expansion of 10 nodes, the expansion will proceed with 10 nodes according to the first specification, while the remaining preset specifications will not be selected.
When resources are insufficient: 1 + 2 > 1 + 2 + 3 > 1 + 2 + 3 + 4 > 1 + 2 + 3 + 4 + 5
For instance
Given that preset specification 1 has 8 nodes, specification 2 has 4 nodes, and specification 3 has 3 nodes, when the expansion rule is triggered and requires the expansion of 13 nodes, the expansion will proceed with 8 nodes according to specification 1, 4 nodes according to specification 2, and 1 node according to specification 3.
When a resource specification is out of stock, assuming specification 2 is unavailable: 1 + 3 > 1 + 3 + 4 > 1 + 3 + 4 + 5
For instance
Given that preset specification 1 has 8 nodes, specification 2 is out of stock with no nodes, and specification 3 has 3 nodes, when the expansion rule is triggered and requires the expansion of 10 nodes, the expansion will proceed with 8 nodes according to specification 1, specification 2 will not be selected, and 2 nodes will be expanded according to specification 3.
Given that preset specification 1 has 8 nodes and the remaining preset specifications are out of stock, when the expansion rule is triggered and requires the expansion of 10 nodes, the expansion rule will be activated, and 8 nodes will be expanded according to specification 1, resulting in a partially successful expansion.
Managed Scaling Monitoring Metrics
Managed scaling will monitor multiple metrics and calculate the recommended node count for each, subsequently making expansion and contraction decisions based on the provided node quantities.
Managed Scaling Monitoring Metrics | MetricsDescription |
AvailableMemPercentage | Percentage of Remaining Memory |
AvailableVCoresPercentage | Percentage of Available YARN Virtual Cores |
Statistical Rules: The cluster load metrics within a statistical cycle, according to the selected aggregation dimension, are set to handle the peak load within the last 10 minutes.
Statistical Cycle: The duration of the metric statistics, one minute.
The default principle of managed scaling is: rapid expansion, cautious contraction, with contraction being for graceful reduction.