Metadata Acceleration Overview
The metadata acceleration feature leverages the excellent metadata management capabilities of CHDFS for you to access COS through HDFS semantics. After the metadata accelerator is enabled for a bucket, you can access COS directly through native HDFS APIs. This not only saves the overheads of conversion from HDFS protocol to object protocol, but also provides some native HDFS features, such as efficient atomic directory renaming, file Atime/Mtime update, efficient directory DU statistics collection, and POSIX ACL. For more information, see Metadata Acceleration.
After metadata acceleration is enabled, the Java APIs for bucket/file operations (the COS Java SDK version must be at least v5.6.80) are basically the same as those for operations in general buckets. For more information on how to call them, see Getting Started. This document mainly describes some new operation features available after metadata acceleration is enabled:
Determine whether the metadata acceleration feature is enabled for a bucket.
Determine if an object is a directory using the
HeadObject API.Delete a directory recursively through the
DeleteObject API.Quickly rename a file through the
Rename API.Upload a file to a bucket with metadata acceleration enabled through the
PutObject API.Creating a COSClient instance
Before calling COS APIs, you must first create a COSClient instance. This document uses SECRETID and SECRETKEY as examples. If you need to create a COSClient using temporary keys, please refer to Generating Temporary Keys.
COSClient createCOSClient() {// Set user identity information.// To view and manage SECRETID and SECRETKEY, please log in to the Access Management Console at https://console.cloud.tencent.com/cam/capi// 1. Pass the obtained credentials (secretId, secretKey, sessionToken)String secretId = "SECRETID";String secretKey = "SECRETKEY";COSCredentials cred = new BasicCOSCredentials(secretId, secretKey);// ClientConfig contains the client settings for subsequent requests to COS:ClientConfig clientConfig = new ClientConfig();// Set the bucket region// For COS_REGION, please refer to https://cloud.tencent.com/document/product/436/6224clientConfig.setRegion(new Region("COS_REGION"));// Set the request protocol, either HTTP or HTTPS// For versions 5.6.53 and below, it is recommended to use the HTTPS protocol.// Version 5.6.54 and higher, HTTPS is used by default.clientConfig.setHttpProtocol(HttpProtocol.https);// The following settings are optional:// Set socket read timeout, default is 30 secondsclientConfig.setSocketTimeout(30 * 1000);// Set connection establishment timeout, default is 30 secondsclientConfig.setConnectionTimeout(30 * 1000);// Set the HTTP proxy, IP, and port if needed.clientConfig.setHttpProxyIp("httpProxyIp");clientConfig.setHttpProxyPort(80);// Create a COS client.return new COSClient(cred, clientConfig);}
Checking whether metadata acceleration is enabled for a bucket
Sample Request
String bucketName = "examplebucket-1250000000";// Create a clientCOSClient cosClient = createCOSClient();HeadBucketRequest headBucketRequest = new HeadBucketRequest(bucketName);HeadBucketResult headBucketResult = cosClient.headBucket(headBucketRequest);bool isMetaAccBucket = headBucketResult.isMetaAccBucket(); // If metadata acceleration is enabled,isMetaAccBucketistrue.
Description
Parameter name | Description | Local Disk Types |
bucketName | The naming convention for buckets is BucketName-APPID. For more information, please refer to Bucket Overview. | String |
Determining whether an object is a directory through HeadObject
You need to first query the metadata of the specified object through
HEAD Object and then use the method in the following example for determination.Sample Request
String bucketName = "examplebucket-1250000000";String key = "test";// Create a clientCOSClient cosClient = createCOSClient();GetObjectMetadataRequest getObjectMetadataRequest = new GetObjectMetadataRequest(bucketName, key);ObjectMetadata objectMetadata = cosClient.getObjectMetadata(getObjectMetadataRequest);boolean isDir = objectMetadata.isFileModeDir(); // If the object is a directory,isDiristrue.
Description
Parameter name | Description | Local Disk Types |
bucketName | The naming convention for buckets is BucketName-APPID. For more information, please refer to Bucket Overview. | String |
key | The object key is the unique identifier of an object within a bucket. For example, in the object access domain examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com/doc/picture.jpg, the object key is doc/picture.jpg. For more information, see Object Key. | String |
Recursively deleting directory through DeleteObject
Sample Request
String bucketName = "examplebucket-1250000000";String key = "test";// Create a clientCOSClient cosClient = createCOSClient();DeleteObjectRequest deleteObjectRequest = new DeleteObjectRequest(bucket, key);deleteObjectRequest.setRecursive(true); // Recursively delete the directorycosClient.deleteObject(deleteObjectRequest);
Description
Parameter name | Description | Local Disk Types |
bucketName | The naming convention for buckets is BucketName-APPID. For more information, please refer to Bucket Overview. | String |
key | The object key is the unique identifier of an object within a bucket. For example, in the object access domain examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com/doc/picture.jpg, the object key is doc/picture.jpg. For more information, see Object Key. | String |
Quickly renaming file through Rename
In a general bucket, you cannot rename a directory with an intermediate path (i.e., a directory on a view). For example, if the
key is /test/demo/src, you cannot rename /test/demo.After enabling the metadata acceleration feature for a bucket, you can rename the directory. This API is newly added for this type of bucket.
Sample Request
String bucketName = "examplebucket-1250000000";String srcKey = "/test/src";String dstKey = "test/dst"// Create a clientCOSClient cosClient = createCOSClient();RenameRequest renameRequest = new RenameRequest(bucketName, srcKey, dstKey);cosClient.rename(renameRequest);
Description
Parameter name | Description | Local Disk Types |
bucketName | The naming convention for buckets is BucketName-APPID. For more information, please refer to Bucket Overview. | String |
srcKey | Original object key, which must start with /. | String |
dstKey | New object key. | String |
Uploading file to bucket with metadata acceleration enabled through PutObject
When uploading a file to a bucket with metadata acceleration enabled, note the following differences:
When uploading a file to a bucket with metadata acceleration enabled through
PutObject, if there is an intermediate path, an intermediate subdirectory will be created recursively.When uploading a file to a regular bucket using the
PutObject API, even if there are intermediate paths, they will still be treated as a single key, causing ListObjects to handle them specially and display them as directories in the console. Therefore, in Hadoop-COS, the HeadObject and ListObjects APIs are used to determine whether intermediate directories exist.Sample Request
//putObjecthas an intermediate pathString bucketName = "examplebucket-1250000000";// An object key is the unique identifier of an object in a bucket.String key = "/testdemo/test/README.md";COSClient cosClient = createCOSClient();// Local fileString localFilePath = "README.md";File localFile = new File(localFilePath);PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, key, file);PutObjectResult putObjectResult = cosClient.putObject(bucketName, key, file);// Determine whether the intermediate path is a file or a directory// Intermediate pathdirKey = "/testdemo/test";GetObjectMetadataRequest getObjectMetadataRequest = new GetObjectMetadataRequest(bucketName, dirKey);ObjectMetadata objectMetadata = cosClient.getObjectMetadata(getObjectMetadataRequest); //404will be returned for a general bucket hereboolean isDir = objectMetadata.isFileModeDir(); //isDiristrue
Description
Supports and Limits
If the above code samples fail, see Getting Started.
Buckets with metadata acceleration enabled currently don't support the following API operations: