The content of this page has been automatically translated by AI. If you encounter any problems while reading, you can view the corresponding content in Chinese.
Help & Documentation>Data Lake Compute

Using DLC to Analyze CLS Logs

Last updated: 2026-05-20 17:04:35

On this page:

Overview

When you need to deliver logs from CLS to DLC for OLAP computation, you can refer to this document for practice. You can use the data analysis and computing service provided by Tencent Cloud DLC (Data Lake Compute, DLC) to perform offline computation and analysis on logs. The schematic diagram is as follows:




Operation Steps

Shipping CLS Logs to COS

Creating a Shipping Task

1. Log in to the CLS console. Choose Ship Task > Ship to COS in the left sidebar.
2. On the "Ship to COS" page, click Add Ship Configuration. In the "Ship to COS" window that pops up, configure and create a ship task. Note the following configuration items:
Configuration Item
Must-Knows
COS Bucket
Log files will be shipped to this catalog in the Cloud Object Storage (COS) bucket. In a data warehouse model, this path generally corresponds to the address of the Table Location.
COS Path
Specify according to the partitioned table format. For example, a daily partition can be set to /dt=%Y%m%d/test, where dt= represents the partition field, %Y%m%d represents the year, month, and day, and test represents the log file prefix.
File Naming
Include shipping time in file names.
Shipping Interval
Select a value within the range of 5–15 minutes. It is recommended to choose 15 minutes and 250 MB, as this will result in fewer files and better query performance.
Shipping Format
JSON format.
Click Next to go to Advanced Configuration. Select JSON and the fields to be processed.

Viewing Shipping Task Results

Typically, you can view the log data in the COS (Cloud Object Storage) console about 15 minutes after starting a ship task. The directory structure is similar to the figure below, with specific log files contained under the partition directories.




Querying Log Data in DLC

Creating an External Table in DLC and Mapping It to the COS Log Catalog

After log data is shipped to COS, you can create an external table using the Data Explore feature on the DLC console. For the table creation statement, see the following SQL statement example. Special attention should be paid to ensuring that the partition field and Location field are consistent with the catalog structure.
The DLC Create External Table Wizard provides advanced options to help you infer the table structure of data files and automatically generate SQL statements quickly. Since this process is sampling-based, you need to further judge whether the table fields are reasonable based on the SQL statement. For example, in the following case, the TIMESTAMP field is inferred as int, but bigint might be more appropriate.
CREATE EXTERNAL TABLE IF NOT EXISTS `DataLakeCatalog`.`test`.`log_data` (
  `__FILENAME__` string,
  `__SOURCE__` string,
  `__TIMESTAMP__` bigint,
  `appId` string,
  `caller` string,
  `consumeTime` string,
  `data` string,
  `datacontenttype` string,
  `deliveryStatus` string,
  `errorResponse` string,
  `eventRuleId` string,
  `eventbusId` string,
  `eventbusType` string,
  `id` string,
  `logTime` string,
  `region` string,
  `requestId` string,
  `retryNum` string,
  `source` string,
  `sourceType` string,
  `specversion` string,
  `status` string,
  `subject` string,
  `tags` string,
  `targetId` string,
  `targetSource` string,
  `time` string,
  `type` string,
  `uin` string
) PARTITIONED BY (`dt` string) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' STORED AS TEXTFILE LOCATION 'cosn://coreywei-1253240642/log_data/'
For partition shipping, Location should point to the cosn://coreywei-1253240642/log_data/ catalog, rather than the cosn://coreywei-1253240642/log_data/20220423/ catalog.
To use the inference feature, you need to point the catalog to the subcatalog where the data file is located, namely, the cosn://coreywei-1253240642/log_data/20220423/ catalog. After the inference is completed, modify Location in SQL statement back to the cosn://coreywei-1253240642/log_data/ catalog.
Proper partitioning can improve performance, but it is recommended that the total number of partitions not exceed 10,000.

Adding Partitions

A partition table can only obtain data via the SELECT statement after partition data is added. You can add partitions using the following 2 methods:
Adding Historical Partitions
Adding Incremental Partitions
You can load all partition data at once. Since the loading is slow, this method is suitable for scenarios where a large number of partitions are loaded for the first time.
msck repair table DataLakeCatalog.test.log_data;
After historical partitions are loaded, incremental partitions can be added periodically. For example, a partition can be added daily using this incremental addition method.
alter table DataLakeCatalog.test.log_data add partition(dt='20220424')

Analyzing Data

After partitions are added, you can perform data development or analysis using DLC.
select dt,count(1) from `DataLakeCatalog`.`test`.`log_data` group by dt;
中国站 - English