DLC provides you with agile and efficient Serverless data lake analysis and computing services. As a distributed computing platform, the query performance of DLC is affected by multiple internal and external factors, such as: engine CU scale, the number of concurrently submitted and queued tasks, SQL writing patterns, and Spark parameter configurations. DLC insight management offers a visual and intuitive interface to help you quickly understand current query performance and the potential factors affecting it, and to obtain performance optimization recommendations.
DLC provides the insight management feature, which includes task insight, engine usage insight, and intelligent storage features, to help users better adjust resources or optimize task logic.
Applicable Business Scenarios:
1. There is a need for insight into the overall runtime status of the Spark engine. For example: metrics such as resource contention during task execution under the engine, resource usage within the engine, engine execution duration, data scan size, and data shuffle size are all displayed and analyzed intuitively.
2. There is a need for convenient self-service troubleshooting and analysis of task runtime status. For example: you can filter and sort numerous tasks by duration to quickly identify problematic large tasks, and pinpoint the causes of slow or failed Spark tasks, such as resource contention, shuffle exceptions, or insufficient disk space, all with clear identification.
3. There is a need for insight into table storage distribution. For example: it enables observation of storage distribution, rankings, and usage trends, intelligent diagnosis of risks, and helps identify tables that require storage optimization.