Overview
DLC Native Table is a table format built on the Iceberg lake format, offering high performance, ease of use, and simple operation. Users can perform data exploration and build applications such as Lakehouse on this basis. For first-time use of DLC Native Table, users need to follow these five main steps:
1. Enable DLC managed storage.
2. Purchase an engine.
3. Create a database and table. Select to create a native table according to the usage scenario and carry optimization parameters.
4. Configure data optimization. Select an independent optimization engine and configure optimization options according to the table type.
5. Import data into the DLC Native Table. Data Lake Compute (DLC) supports various data write modes, including insert into/merge into/upsert, and supports multiple import methods, including spark/presto/flink/inlong/oceanus.
Native Table Types
DLC Native Table is the recommended format of DLC Lakehouse. It provides two types of table formats: Iceberg and TC-Iceberg. Their characteristics and usage scenarios are as follows:
Note:
TC-Iceberg is currently in public beta. The current version only supports primary key update scenarios and has some limitations. For more details, see native table (TC-Iceberg) format description.
Table Type | Description | Scenarios |
Iceberg | Uses the Apache Iceberg table format, including ACID transactions, hidden partitions, data version control, and other features, and provides two table version selections: V1/V2. | 1. Traditional batch processing scenario (Append): The Iceberg V1 table version is used. In this scenario, tables only support Append, Overwrite, and Merge into write modes. 2. Real-time writing scenario (Upsert): The Iceberg V2 table version is used. Compared with the Append scenario, this scenario additionally supports the Upsert write mode and supports real-time writing modes such as InLong, Oceanus, or self-managed Flink. |
TC-Iceberg | Tencent Cloud's batch-stream integrated table format, expanded based on Iceberg. It is compatible with and includes all the strengths of Apache Iceberg, and provides performance enhancement and near-real-time lakehouse construction capabilities. | The current version only supports primary key update scenarios. It can meet customer business requirements in scenarios with real-time writing (Upsert) and near-real-time lakehouse construction (CDC streaming consumption). |
Advantages of Native Tables
DLC Native Table adopts the managed data storage mode. After users use an Iceberg table, DLC will uniformly manage the metadata and data files of the Iceberg table. This brings the following benefits to users:
Higher Data Security
Iceberg table data consists of two parts: metadata and data. If certain files are corrupted, the entire table query will fail. In contrast, with Hive, only the data from the corrupted files might be unqueryable. Storing data in DLC under managed storage can reduce the risk of users inadvertently damaging files due to a lack of understanding of Iceberg.
Enhanced Storage Performance
DLC managed storage uses chdfs as storage by defaults. Compared with ordinary Cloud Object Storage (COS), the performance is greatly enhanced. Based on DLC's Smart Optimizer service for managed tables, regularly merging small files and purging orphan snapshots can further improve query performance.
Reduced Ops Costs
The COS service is enabled automatically. Storage resource assignment is automatically performed. Built-in table data optimization and lifecycle management features are provided, significantly reducing Ops costs.