Operations for Metadata Acceleration-Enabled Buckets

Last updated: 2023-09-13 14:43:10

Metadata Acceleration Overview

The metadata acceleration feature leverages the excellent metadata management capabilities of CHDFS for you to access COS through HDFS semantics. After the metadata accelerator is enabled for a bucket, you can access COS directly through native HDFS APIs. This not only saves the overheads of conversion from HDFS protocol to object protocol, but also provides some native HDFS features, such as efficient atomic directory renaming, file Atime/Mtime update, efficient directory DU statistics collection, and POSIX ACL. For more information, see Metadata Acceleration.
After metadata acceleration is enabled, the Java APIs for bucket/file operations (the COS Java SDK version must be at least v5.6.80) are basically the same as those for operations in general buckets. For more information on how to call them, see Getting Started. This document mainly describes some new operation features available after metadata acceleration is enabled:
Determine whether the metadata acceleration feature is enabled for a bucket.
Determine if an object is a directory using the HeadObject API.
Delete a directory recursively through the DeleteObject API.
Quickly rename a file through the Rename API.
Upload a file to a bucket with metadata acceleration enabled through the PutObject API.

Creating a COSClient instance

Before calling COS APIs, you must first create a COSClient instance. This document uses SECRETID and SECRETKEY as examples. If you need to create a COSClient using temporary keys, please refer to Generating Temporary Keys.
COSClient createCOSClient() {
// Set user identity information.
// To view and manage SECRETID and SECRETKEY, please log in to the Access Management Console at https://console.cloud.tencent.com/cam/capi
// 1. Pass the obtained credentials (secretId, secretKey, sessionToken)
String secretId = "SECRETID";
String secretKey = "SECRETKEY";
COSCredentials cred = new BasicCOSCredentials(secretId, secretKey);

// ClientConfig contains the client settings for subsequent requests to COS:
ClientConfig clientConfig = new ClientConfig();

// Set the bucket region
// For COS_REGION, please refer to https://cloud.tencent.com/document/product/436/6224
clientConfig.setRegion(new Region("COS_REGION"));

// Set the request protocol, either HTTP or HTTPS
// For versions 5.6.53 and below, it is recommended to use the HTTPS protocol.
// Version 5.6.54 and higher, HTTPS is used by default.
clientConfig.setHttpProtocol(HttpProtocol.https);

// The following settings are optional:

// Set socket read timeout, default is 30 seconds
clientConfig.setSocketTimeout(30 * 1000);
// Set connection establishment timeout, default is 30 seconds
clientConfig.setConnectionTimeout(30 * 1000);

// Set the HTTP proxy, IP, and port if needed.
clientConfig.setHttpProxyIp("httpProxyIp");
clientConfig.setHttpProxyPort(80);

// Create a COS client.
return new COSClient(cred, clientConfig);
}

Checking whether metadata acceleration is enabled for a bucket

Sample Request

String bucketName = "examplebucket-1250000000";
// Create a client
COSClient cosClient = createCOSClient();
HeadBucketRequest headBucketRequest = new HeadBucketRequest(bucketName);
HeadBucketResult headBucketResult = cosClient.headBucket(headBucketRequest);
bool isMetaAccBucket = headBucketResult.isMetaAccBucket(); // If metadata acceleration is enabled, isMetaAccBucket is true.

Description

Parameter name
Description
Local Disk Types
bucketName
The naming convention for buckets is BucketName-APPID. For more information, please refer to Bucket Overview.
String

Determining whether an object is a directory through HeadObject

You need to first query the metadata of the specified object through HEAD Object and then use the method in the following example for determination.

Sample Request

String bucketName = "examplebucket-1250000000";
String key = "test";
// Create a client
COSClient cosClient = createCOSClient();
GetObjectMetadataRequest getObjectMetadataRequest = new GetObjectMetadataRequest(bucketName, key);
ObjectMetadata objectMetadata = cosClient.getObjectMetadata(getObjectMetadataRequest);
boolean isDir = objectMetadata.isFileModeDir(); // If the object is a directory, isDir is true.

Description

Parameter name
Description
Local Disk Types
bucketName
The naming convention for buckets is BucketName-APPID. For more information, please refer to Bucket Overview.
String
key
The object key is the unique identifier of an object within a bucket. For example, in the object access domain examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com/doc/picture.jpg, the object key is doc/picture.jpg. For more information, see Object Key.
String

Recursively deleting directory through DeleteObject

Sample Request

String bucketName = "examplebucket-1250000000";
String key = "test";
// Create a client
COSClient cosClient = createCOSClient();
DeleteObjectRequest deleteObjectRequest = new DeleteObjectRequest(bucket, key);
deleteObjectRequest.setRecursive(true); // Recursively delete the directory
cosClient.deleteObject(deleteObjectRequest);

Description

Parameter name
Description
Local Disk Types
bucketName
The naming convention for buckets is BucketName-APPID. For more information, please refer to Bucket Overview.
String
key
The object key is the unique identifier of an object within a bucket. For example, in the object access domain examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com/doc/picture.jpg, the object key is doc/picture.jpg. For more information, see Object Key.
String

Quickly renaming file through Rename

In a general bucket, you cannot rename a directory with an intermediate path (i.e., a directory on a view). For example, if the key is /test/demo/src, you cannot rename /test/demo.
After enabling the metadata acceleration feature for a bucket, you can rename the directory. This API is newly added for this type of bucket.

Sample Request

String bucketName = "examplebucket-1250000000";
String srcKey = "/test/src";
String dstKey = "test/dst"
// Create a client
COSClient cosClient = createCOSClient();
RenameRequest renameRequest = new RenameRequest(bucketName, srcKey, dstKey);
cosClient.rename(renameRequest);

Description

Parameter name
Description
Local Disk Types
bucketName
The naming convention for buckets is BucketName-APPID. For more information, please refer to Bucket Overview.
String
srcKey
Original object key, which must start with /.
String
dstKey
New object key.
String

Uploading file to bucket with metadata acceleration enabled through PutObject

When uploading a file to a bucket with metadata acceleration enabled, note the following differences:
When uploading a file to a bucket with metadata acceleration enabled through PutObject, if there is an intermediate path, an intermediate subdirectory will be created recursively.
When uploading a file to a regular bucket using the PutObject API, even if there are intermediate paths, they will still be treated as a single key, causing ListObjects to handle them specially and display them as directories in the console. Therefore, in Hadoop-COS, the HeadObject and ListObjects APIs are used to determine whether intermediate directories exist.

Sample Request

// putObject has an intermediate path
String bucketName = "examplebucket-1250000000";
// An object key is the unique identifier of an object in a bucket.
String key = "/testdemo/test/README.md";

COSClient cosClient = createCOSClient();

// Local file
String localFilePath = "README.md";
File localFile = new File(localFilePath);
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, key, file);
PutObjectResult putObjectResult = cosClient.putObject(bucketName, key, file);

// Determine whether the intermediate path is a file or a directory
// Intermediate path
dirKey = "/testdemo/test";
GetObjectMetadataRequest getObjectMetadataRequest = new GetObjectMetadataRequest(bucketName, dirKey);
ObjectMetadata objectMetadata = cosClient.getObjectMetadata(getObjectMetadataRequest); // 404 will be returned for a general bucket here
boolean isDir = objectMetadata.isFileModeDir(); // isDir is true

Description

For a detailed explanation of Put Object parameters, see Java Upload Object.

Supports and Limits

If the above code samples fail, see Getting Started.
Buckets with metadata acceleration enabled currently don't support the following API operations: