The content of this page has been automatically translated by AI. If you encounter any problems while reading, you can view the corresponding content in Chinese.
Help & Documentation>Data Lake Compute

Querying DLC Internal Storage Directly with StarRocks

Last updated: 2026-05-20 17:05:21

On this page:

What Is StarRocks

StarRocks is a next-generation, high-performance, full-scenario MPP database. It fully incorporates the excellent research achievements of relational OLAP databases and distributed storage systems from the big data era. Building on industry practices, it has been further improved, optimized, and upgraded in architecture, with numerous new features added, culminating in a new enterprise-grade product. StarRocks is dedicated to delivering an ultra-fast unified analytics experience to meet the diverse data analysis scenarios of enterprise users. It supports multiple data models (the duplicate key model, aggregate key model, and unique key model), various data ingestion methods (batch and real-time), and can import data with up to 10,000 columns. It is also capable of integrating and connecting with various existing systems (Spark, Flink, Hive, ElasticSearch).
In Tencent Cloud EMR, we provide a fully out-of-the-box StarRocks service. For details, refer to StarRocks Introduction.

Accelerating Queries in a Lakehouse with StarRocks+DLC

Tencent Cloud Data Lake Compute (DLC) supports a lakehouse query acceleration mode based on EMR StarRocks. This mode enables direct execution of complex SQL queries on DLC data sources without the need to import data into StarRocks or create external tables. Leveraging StarRocks' MPP vectorized query capability, it enhances data analysis efficiency and reduces Ops complexity and cost.
Next, this document describes how to enable the DLC + (EMR) StarRocks lakehouse query acceleration.

Prerequisites

1. You have purchased an EMR StarRocks cluster.
2. The DLC service has been activated.
Note:
1. Cross-region federated DLC is not currently supported. Plan your environment accordingly to ensure that both EMR StarRocks and DLC are in the same region.
2. Querying DLC existing native tables created before June 12, 2024 (the storage path for DLC existing native tables is lakefs://***, which is not currently compatible) is not supported. However, querying DLC new native tables created on or after June 12, 2024 (storage path: cosn://***) and all DLC external tables is supported.
3. StarRocks only supports querying DLC data and does not support write or delete operations on DLC data.
To enable the DLC+StarRocks lakehouse query acceleration, you first need to enable DLC external access, allowing the StarRocks cluster to access data stored in DLC's internal managed storage. Then, create a DLC External Catalog in the StarRocks cluster. You can then use the StarRocks compute engine to directly analyze DLC-stored data.

Enabling External Access to DLC

Go to the DLC console, click the Storage Configuration menu, and select Enable External Access.
Step 1: Click "Enable External Access to Managed Storage". After it is enabled, the EMR StarRocks clusters under your same Tencent Cloud account can access the internally managed storage of DLC. Subsequently, you can enable or disable the external access service for DLC's internally managed storage by toggling the switch here.

The account used to enable external access must have DLC management permissions. Use a root account (or a sub-account with DLC administrator permissions) to perform this operation.
Note:
1. If you already have DLC administrator permissions, you can skip this step.
2. If you are using a sub-account and do not have DLC administrator permissions, you can refer to the document Sub-account Permission Management to have an account with DLC administrator permissions grant you permissions.
Step 2: To ensure that your EMR StarRocks cluster can correctly access the DLC metadata Catalog service, you need to bind the EMR StarRocks VPC to the DLC network.
1. Click "Bind VPC". In the dialog box, select EMR StarRocks as the type. Then, in the EMR instance drop-down list, select the EMR StarRocks cluster instance ID to be bound.
2. The VPC where the StarRocks cluster resides will be automatically populated. You can enter an easily identifiable alias in the remarks name.
Step 3: After the VPC binding is completed, you can connect to the DLC metadata service in your StarRocks cluster using the URI connection string displayed in the Catalog access address, for example:
URI Connection String Example:
Catalog access address: thrift://172.17.1.18:7004.
Your EMR StarRocks cluster can now directly analyze data stored in DLC's internal managed storage. Before you start the analysis, you also need to create a DLC External Catalog in StarRocks.

Creating a DLC External Catalog

Log in to StarRocks and create a DLC Catalog in StarRocks. For details about External Catalog, see CREATE EXTERNAL CATALOG | Overview.

Syntax

CREATE EXTERNAL CATALOG dlc_iceberg_cos_catalog
PROPERTIES
(
    "type" = "iceberg",
	"iceberg.catalog.type" = "hive",
    "iceberg.catalog.hive.metastore.uris" = "thrift://169.254.0.171:8007",
    "aws.s3.endpoint" = "cos.ap-chongqing.myqcloud.com",
    "aws.s3.access_key" = "Tencent Cloud secret id",
    "aws.s3.secret_key" = "Tencent Cloud secret_key"
);

Parameter Description

Parameter
Description
type
The type of the data source, which is set to Iceberg by default.
iceberg.catalog.type
The type of the metadata service used by the Iceberg cluster. Set it to hive.
iceberg.catalog.hive.metastore.uris
The URI of the DLC metadata.
aws.s3.endpoint
The Endpoint for accessing S3-compatible object storage. The format for Tencent Cloud COS is cos.<region>.myqcloud.com, where <region> can be ap-beijing, ap-shanghai, ap-guangzhou, and so on.
aws.s3.access_key
The SecretID in the Tencent Cloud account key.
aws.s3.secret_key
The SecretKey in the Tencent Cloud account key.
Note:
1. For security reasons, you must use the SecretID and SecretKey of a root account in the above SecretID and SecretKey configuration to correctly access DLC internal storage. Specifically, you can log in to Tencent Cloud with a root account and obtain the SecretID and SecretKey on the CAM console > Login - Tencent Cloud page.
2. If you need to access DLC internal managed storage using the SecretID and SecretKey of a sub-account, you can refer to the last part of this document: Accessing with Sub-account SecretID/SecretKey.

Example

The following example demonstrates how to create a DLC Catalog named DLC_catalog:
CREATE EXTERNAL CATALOG dlc_hive_cos_catalog
PROPERTIES
(
    "type" = "Iceberg",
    "iceberg.catalog.type" = "hive",
    "hive.metastore.uris" = "thrift://169.254.0.171:8007",
    "aws.s3.endpoint" = "cos.ap-chongqing.myqcloud.com",
	"aws.s3.access_key" = "**********",
    "aws.s3.secret_key" = "**********"
)
If you need to directly query Hive tables in DLC managed storage through StarRocks, you need to create another DLC Hive catalog. Specifically, set the type to hive. The example is as follows:
CREATE EXTERNAL CATALOG dlc_hive_cos_catalog
PROPERTIES
(
    "type" = "hive",
    "iceberg.catalog.type" = "hive",
    "hive.metastore.uris" = "thrift://169.254.0.171:8007",
    "aws.s3.endpoint" = "cos.ap-chongqing.myqcloud.com",
	"aws.s3.access_key" = "**********",
    "aws.s3.secret_key" = "**********"
)

Querying DLC Data

DLC Tables and Data Preparation

The example for creating an Iceberg table is as follows:
CREATE TABLE test_sr_ofs.`customer`(
    `c_custkey` bigint,
    `c_name` string,
    `c_address` string,
    `c_nationkey` int,
    `c_phone` string,
    `c_acctbal` double,
    `c_mktsegment` string,
    `c_comment` string) using iceberg;
)
Query data in StarRocks:
#Log in to the StarRocks node
mysql -h 172.30.0.xxx -P9030 -u root -p

#Specify the iceberg catalog
set catalog dlc_iceberg_cos_catalog;

#Specify the database
use test_sr_ofs;

#Query customer table data
select * from customer limit 5;
The query result is as follows:




Accessing with Sub-account SecretID/SecretKey (Optional)

For data security considerations, after DLC external access is enabled, you must use the SecretID and SecretKey of a root account by default to initiate access to DLC internal storage from EMR StarRocks. If you need to use the SecretID and SecretKey of a sub-account for access due to business requirements, you must use a root account to create a custom policy in CAM and bind it to the corresponding sub-account.

Principle

After you enable DLC external access by following the instructions above in this document, the system essentially grants your root account permissions to access DLC internal managed storage. However, for data security reasons, other sub-accounts under your root account do not have this access permission by default. You need to create a custom policy in Tencent Cloud CAM that grants sub-accounts the ability to access, and bind it to the sub-accounts that need to access DLC internal managed storage. The specific steps are as follows:

Step 1: Generating a Custom Policy

Log in to Tencent Cloud with a root account, go to the DLC console, click the Storage Configuration menu, select Enable External Access, and in the Sub-account Authorization section, click "Click to Process". In the pop-up dialog box, click Copy to obtain the custom CAM policy you need to create.




Step 2: Creating a Custom Policy to Allow Sub-accounts to Access DLC Internal Storage

1. Log in to the CAM console with a root account and go to the Policy page.
2. Choose Create Custom Policy > Create by Policy Syntax, then select Blank Template and click Next.
Note:
A root account can grant data access permissions to a sub-account. Currently, this can only be done through a custom policy and is not supported via a preset policy.
3. Enter the following form:
Policy Name: Define a unique and meaningful policy name yourself, for example, cos-child-account.
Remarks: This is optional, and you can add custom remarks.
Policy Content: Paste the custom policy copied in Step 1, for example:
{
  "statement": [
    {
      "action": [
        "cos:*"
      ],
      "effect": "allow",
      "resource": [
        "qcs::cos:ap-shanghai:uid/1305424723:dlc03ff-100018379117-1647867281-100017307912-1304028854/*",
        "qcs::cos:ap-shanghai:uid/1305424723:dlc0a65-100018379117-1680005779-100017307912-1304028854/*"
      ]
    }
  ],
  "version": "2.0"
}
Note:
The above policy grants permissions for all DLC managed storage that the root account is authorized to operate to the sub-account. Here, 1305424723 in uid/1305424723 is the APPID of root account A, and dlc0a65-100018379117-1680005779-100017307912-1304028854/* is the DLC internal managed storage that you are authorized to operate.
4. Click Finish, and then the policy is created.

Step 3: Granting Permissions to Sub-accounts That Need to Access DLC Internal Storage

1. Locate the policy you just created in the Policy list, and click Associate Users/User Groups/Roles on the right.
2. In the pop-up window, select the sub-accounts that need permissions to access the DLC internal managed storage, and click OK.
3. After the authorization is completed, you can use the SecretID and SecretKey of the sub-account to access the DLC internal managed storage under your name.
中国站 - English