When cluster resources are tight, tasks submitted to the engine may be queued within it. However, users are unaware of this and may continue submitting tasks, leading to severe task blocking.
Engine Usage Insights presents a unified view of the distribution of all tasks under an engine, from submission to execution, based on the engine dimension. This helps customers quickly analyze the general usage of the engine.
Prerequisites
1. SuperSQL SparkSQL and Spark Job Engine:
1.1 Engine Usage Insights is enabled by default for newly purchased engines after July 18, 2024.
1.2 For Spark kernel editions released before July 18, 2024, you must upgrade the engine kernel to enable Task Insights. For the upgrade procedure, see How to Enable the Insight Feature below.
2. Standard Spark Engine:
2.1 Engines purchased after December 20, 2024, support Engine Usage Insights by default.
2.2 For engines purchased before December 20, 2024, Engine Usage Insights cannot be manually enabled by users. To enable this feature, submit a ticket to contact after-sales support.
Engine Usage Insights is not currently supported for other engine types.
Operation Steps
Log in to the DLC console and select the Insight Management feature. Then, navigate to the Engine Usage Insights page, select the data engine you want to view, and hover your mouse over the task duration waterfall chart to view the details.
The gray progress bar represents queuing time.
The blue progress bar represents engine execution time.
The green progress bar represents the time taken to obtain results.
The overall bar chart shifts over time.
Note:
The SparkSQL engine of SuperSQL supports real-time engine execution duration and data scan volume. For other engine types, queuing time and engine execution duration data are available only after the insight process is complete.