Feature Overview
This document provides an overview of APIs and SDK code samples for simple operations, multipart operations, and other object operations.
API | Operation | Description |
Downloading an object | Downloads an object to the local file system |
Advanced APIs (Recommended)
Downloading an object (checkpoint restart)
Note
This advanced download API can determine whether to download a file in whole or in parts based on the file size. It downloads files smaller than or equal to 20 MB in whole and files larger than 20 MB in parts. It also supports automatic checkpoint restart for suspended multipart downloads.
Method prototype
download_file(Bucket, Key, DestFilePath, PartSize=20, MAXThread=5, EnableCRC=False, **Kwargs)
Sample Request
# -*- coding=utf-8from qcloud_cos import CosConfigfrom qcloud_cos import CosS3Clientfrom qcloud_cos.cos_exception import CosClientError, CosServiceErrorimport sysimport osimport logging# Under normal circumstances, use the INFO log level. To locate issues, change it to DEBUG, and the SDK will print communication information with the server.logging.basicConfig(level=logging.INFO, stream=sys.stdout)# 1. Set user attributes, including secret_id, secret_key, region, etc. Appid has been removed from CosConfig, please include Appid in the Bucket parameter. Bucket is composed of BucketName-Appid.secret_id = os.environ['COS_SECRET_ID'] # User <1>SecretId</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.secret_key = os.environ['COS_SECRET_KEY'] # User <1>SecretKey</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.region = 'ap-beijing' # Replace it with the actual region, which can be viewed in the console at https://console.cloud.tencent.com/cos5/bucket.For a list of all regions supported by COS, please visit https://cloud.tencent.com/document/product/436/6224token = None # Token is required for temporary keys but not permanent keys. For more information about how to generate and use a temporary key, see https://cloud.tencent.com/document/product/436/14048.scheme = 'https' # Specify whether to use HTTP or HTTPS protocol to access COS. This is optional and is https by default.config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token, Scheme=scheme)client = CosS3Client(config)# Using the advanced interface to download once without retrying, without utilizing the checkpoint restart feature.response = client.download_file(Bucket='examplebucket-1250000000',Key='exampleobject',DestFilePath='local.txt')# Use the advanced interface for checkpoint restart; failed retries will not download already successful parts (retrying 10 times here)for i in range(0, 10):try:response = client.download_file(Bucket='examplebucket-1250000000',Key='exampleobject',DestFilePath='local.txt')breakexcept CosClientError or CosServiceError as e:print(e)
Sample request with all parameters
def upload_percentage(consumed_bytes, total_bytes):"""Progress bar callback function, calculating the current upload percentage:param consumed_bytes: Amount of data already uploaded:param total_bytes: Total data volume"""if total_bytes:rate = int(100 * (float(consumed_bytes) / float(total_bytes)))print('\r{0}% '.format(rate))sys.stdout.flush()response = client.download_file(Bucket='examplebucket-1250000000',Key='exampleobject',DestFilePath='local.txt',PartSize=1,MAXThread=5,EnableCRC=True,TrafficLimit='1048576'IfMatch='"9a4802d5c99dafe1c04da0a8e7e166bf"',IfModifiedSince='Wed, 28 Oct 2014 20:30:00 GMT',IfNoneMatch='"9a4802d5c99dafe1c04da0a8e7e166bf"',IfUnmodifiedSince='Wed, 28 Oct 2014 20:30:00 GMT',ResponseCacheControl='string',ResponseContentDisposition='string',ResponseContentEncoding='string',ResponseContentLanguage='string',ResponseContentType='string',ResponseExpires='string',VersionId='string',progress_callback=upload_percentage)
Description
Parameter name | ParameterDescription | Local Disk Types | Required |
Bucket | Bucket name in the format of BucketName-APPID | String | Required |
Key | ObjectKey is the unique identifier of the object in the bucket. For example, in the object's access domain name examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com/doc/pic.jpg, the ObjectKey is doc/pic.jpg | String | Required |
DestFilePath | Local path to save the downloaded file | String | Required |
PartSize | Part size for multipart download. Default value: 20 MB | Int | Not required |
MAXThread | Maximum number of concurrent threads for a multipart download. Default value: 5 | Int | Not required |
EnableCRC | Whether to enable CRC for local and remote files. Default: False | Bool | Not required |
TrafficLimit | The rate limit value for a single connection is in bits per second (bit/s), with a range of 819200 to 838860800, or 800Kb/s to 800Mb/s. The advanced interface restricts the speed of a single thread. | String | Not required |
IfMatch | Returns the object only if the Etag matches the specified content | String | Not required |
IfModifiedSince | Returns the object metadata only if the object is modified after the time specified in GMT format | String | Not required |
IfNoneMatch | Returns the object only if the Etag does not match the specified content | String | Not required |
IfUnmodifiedSince | Returns the object only if it is modified before or at the time specified in GMT format | String | Not required |
ResponseCacheControl | Sets Cache-Control in the response header | String | Not required |
ResponseContentDisposition | Sets Content-Disposition in the response header | String | Not required |
ResponseContentEncoding | Sets Content-Encoding in the response header | String | Not required |
ResponseContentLanguage | Sets Content-Language in the response header | String | Not required |
ResponseContentType | Sets Content-Type in the response header | String | Not required |
ResponseExpires | Expires of the response header | String | Not required |
VersionId | Version ID of the object to download | String | Not required |
progress_callback | Callback function for the upload progress. You can customize this function to query the upload progress. | Func | Not required |
Response description
None
Batch download (downloading a COS directory)
Note
The following sample uses the basic SDK APIs to download objects in a COS directory to the local file system.
Sample Request
# -*- coding=utf-8import osimport loggingimport sysimport jsonimport osfrom qcloud_cos import CosConfig, CosServiceErrorfrom qcloud_cos import CosS3Clientfrom qcloud_cos.cos_threadpool import SimpleThreadPool# Under normal circumstances, use the INFO log level. To locate issues, change it to DEBUG, and the SDK will print communication information with the server.logging.basicConfig(level=logging.INFO, stream=sys.stdout)# 1. Set user attributes, including secret_id, secret_key, region, etc. Appid has been removed from CosConfig, please include Appid in the Bucket parameter. Bucket is composed of BucketName-Appid.secret_id = os.environ['COS_SECRET_ID'] # User <1>SecretId</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.secret_key = os.environ['COS_SECRET_KEY'] # User <1>SecretKey</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.region = 'ap-beijing' # Replace it with the actual region, which can be viewed in the console at https://console.cloud.tencent.com/cos5/bucket.For a list of all regions supported by COS, please visit https://cloud.tencent.com/document/product/436/6224token = None # Token is required for temporary keys but not permanent keys. For more information about how to generate and use a temporary key, see https://cloud.tencent.com/document/product/436/14048.scheme = 'https' # Specify whether to use HTTP or HTTPS protocol to access COS. This is optional and is https by default.config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token, Scheme=scheme) # Obtain the object to configure.client = CosS3Client(config)# User's bucket informationtest_bucket = 'examplebucket-1250000000'start_prefix = 'data/'Object storage relies on the delimiter '/' to simulate directory semantics.# Using the default empty separator, you can list all child nodes under a directory, achieving a similar effect to local directory recursion.# If the delimiter is set to "/", you need to handle subdirectories recursively in the program.delimiter = ''# List the subnodes of the current directory and return information about all subnodes.def listCurrentDir(prefix):file_infos = []sub_dirs = []marker = ""count = 1while True:response = client.list_objects(test_bucket, prefix, delimiter, marker)Debug output# json_object = json.dumps(response, indent=4)# print(count, " =======================================")# print(json_object)count += 1if "CommonPrefixes" in response:common_prefixes = response.get("CommonPrefixes")sub_dirs.extend(common_prefixes)if "Contents" in response:contents = response.get("Contents")file_infos.extend(contents)if "NextMarker" in response.keys():marker = response["NextMarker"]else:breakprint("=======================================================")If the delimiter is set to "/", recursive processing of subdirectories is required.# sorted(sub_dirs, key=lambda sub_dir: sub_dir["Prefix"])# for sub_dir in sub_dirs:# print(sub_dir)# sub_dir_files = listCurrentDir(sub_dir["Prefix"])# file_infos.extend(sub_dir_files)print("=======================================================")sorted(file_infos, key=lambda file_info: file_info["Key"])for file in file_infos:print(file)return file_infos# Download the file to a local directory; if there is already a file with the same name in the local directory, it will be overwritten.# If the directory structure does not exist, it will create a directory structure identical to the object storage.def downLoadFiles(file_infos):localDir = "./download/"pool = SimpleThreadPool()for file in file_infos:# File download: Retrieve the file to the local systemfile_cos_key = file["Key"]localName = localDir + file_cos_key# Recursively create the local directory structure if it does not existif not os.path.exists(os.path.dirname(localName)):os.makedirs(os.path.dirname(localName))# skip dir, no need to download itif str(localName).endswith("/"):continue# Actual file download# Utilizing a thread pool methodpool.add_task(client.download_file, test_bucket, file_cos_key, localName)Simple download method# response = client.get_object(# Bucket=test_bucket,# Key=file_cos_key,# )# response['Body'].get_stream_to_file(localName)pool.wait_completion()return None# Function encapsulation: Download a directory from object storage to the local diskdef downLoadDirFromCos(prefix):global file_infostry:file_infos = listCurrentDir(prefix)except CosServiceError as e:print(e.get_origin_msg())print(e.get_digest_msg())print(e.get_status_code())print(e.get_error_code())print(e.get_error_msg())print(e.get_resource_location())print(e.get_trace_id())print(e.get_request_id())downLoadFiles(file_infos)return Noneif __name__ == "__main__":downLoadDirFromCos(start_prefix)
Simple Operations
Downloading an object
Note
This API (
GET Object) is used to download an object.Method prototype
get_object(Bucket, Key, **kwargs)
Sample 1: downloading an object
# -*- coding=utf-8from qcloud_cos import CosConfigfrom qcloud_cos import CosS3Clientimport sysimport osimport logging# Under normal circumstances, use the INFO log level. To locate issues, change it to DEBUG, and the SDK will print communication information with the server.logging.basicConfig(level=logging.INFO, stream=sys.stdout)# 1. Set user attributes, including secret_id, secret_key, region, etc. Appid has been removed from CosConfig, please include Appid in the Bucket parameter. Bucket is composed of BucketName-Appid.secret_id = os.environ['COS_SECRET_ID'] # User <1>SecretId</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.secret_key = os.environ['COS_SECRET_KEY'] # User <1>SecretKey</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.region = 'ap-beijing' # Replace it with the actual region, which can be viewed in the console at https://console.cloud.tencent.com/cos5/bucket.For a list of all regions supported by COS, please visit https://cloud.tencent.com/document/product/436/6224token = None # Token is required for temporary keys but not permanent keys. For more information about how to generate and use a temporary key, see https://cloud.tencent.com/document/product/436/14048.scheme = 'https' # Specify whether to use HTTP or HTTPS protocol to access COS. This is optional and is https by default.config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token, Scheme=scheme)client = CosS3Client(config)response = client.get_object(Bucket='examplebucket-1250000000',Key='exampleobject')response['Body'].get_stream_to_file('exampleobject')
Sample 2: Partially downloading an object
# -*- coding=utf-8from qcloud_cos import CosConfigfrom qcloud_cos import CosS3Clientimport sysimport osimport logging# Under normal circumstances, use the INFO log level. To locate issues, change it to DEBUG, and the SDK will print communication information with the server.logging.basicConfig(level=logging.INFO, stream=sys.stdout)# 1. Set user attributes, including secret_id, secret_key, region, etc. Appid has been removed from CosConfig, please include Appid in the Bucket parameter. Bucket is composed of BucketName-Appid.secret_id = os.environ['COS_SECRET_ID'] # User <1>SecretId</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.secret_key = os.environ['COS_SECRET_KEY'] # User <1>SecretKey</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.region = 'ap-beijing' # Replace it with the actual region, which can be viewed in the console at https://console.cloud.tencent.com/cos5/bucket.For a list of all regions supported by COS, please visit https://cloud.tencent.com/document/product/436/6224token = None # Token is required for temporary keys but not permanent keys. For more information about how to generate and use a temporary key, see https://cloud.tencent.com/document/product/436/14048.scheme = 'https' # Specify whether to use HTTP or HTTPS protocol to access COS. This is optional and is https by default.config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token, Scheme=scheme)client = CosS3Client(config)response = client.get_object(Bucket='examplebucket-1250000000',Key='exampleobject',Range='bytes=0-100')response['Body'].get_stream_to_file('exampleobject')
Sample request with all parameters
response = client.get_object(Bucket='examplebucket-1250000000',Key='exampleobject',Range='string',IfMatch='"9a4802d5c99dafe1c04da0a8e7e166bf"',IfModifiedSince='Wed, 28 Oct 2014 20:30:00 GMT',IfNoneMatch='"9a4802d5c99dafe1c04da0a8e7e166bf"',IfUnmodifiedSince='Wed, 28 Oct 2014 20:30:00 GMT',ResponseCacheControl='string',ResponseContentDisposition='string',ResponseContentEncoding='string',ResponseContentLanguage='string',ResponseContentType='string',ResponseExpires='string',VersionId='string',TrafficLimit='819200')
Description
Parameter name | ParameterDescription | Local Disk Types | Required |
Bucket | Bucket name in the format of BucketName-APPID | String | Required |
Key | ObjectKey is the unique identifier of the object in the bucket. For example, in the object's access domain name examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com/doc/pic.jpg, the ObjectKey is doc/pic.jpg | String | Required |
Range | Sets the byte range of the object to download in the format of bytes=first-last. | String | Not required |
IfMatch | Returns the object only if the Etag matches the specified content | String | Not required |
IfModifiedSince | Returns the object metadata only if the object is modified after the time specified in GMT format | String | Not required |
IfNoneMatch | Returns the object only if the Etag does not match the specified content | String | Not required |
IfUnmodifiedSince | Returns the object only if it is modified before or at the time specified in GMT format | String | Not required |
ResponseCacheControl | Sets Cache-Control in the response header | String | Not required |
ResponseContentDisposition | Sets Content-Disposition in the response header | String | Not required |
ResponseContentEncoding | Sets Content-Encoding in the response header | String | Not required |
ResponseContentLanguage | Sets Content-Language in the response header | String | Not required |
ResponseContentType | Sets Content-Type in the response header | String | Not required |
ResponseExpires | Expires of the response header | String | Not required |
VersionId | Version ID of the object to download | String | Not required |
TrafficLimit | The rate limit value for a single connection is in bits per second (bit/s), with a range of 819200 to 838860800, or 800Kb/s to 800Mb/s. The advanced interface restricts the speed of a single thread. | String | Not required |
Response description
The response contains the content and metadata of the downloaded object in dict format:
{'Body': StreamBody(),'ETag': '"9a4802d5c99dafe1c04da0a8e7e166bf"','Last-Modified': 'Wed, 28 Oct 2014 20:30:00 GMT','Accept-Ranges': 'bytes','Content-Range': 'bytes 0-16086/16087','Cache-Control': 'max-age=1000000','Content-Type': 'application/octet-stream','Content-Disposition': 'attachment; filename="filename.jpg"','Content-Encoding': 'gzip','Content-Language': 'zh-cn','Content-Length': '16807','Expires': 'Wed, 28 Oct 2019 20:30:00 GMT','x-cos-meta-test': 'test','x-cos-version-id': 'MTg0NDUxODMzMTMwMDM2Njc1ODA','x-cos-request-id': 'NTg3NzQ3ZmVfYmRjMzVfMzE5N182NzczMQ=='}
Parameter name | ParameterDescription | Local Disk Types |
Body | Content of the downloaded object. You can use the get_raw_stream() method to get a file stream and the get_stream_to_file(local_file_path) method to download the object’s content to a specified local file. | StreamBody |
ETag | MD5 checksum of the object | String |
Last-Modified | Time the object was last modified | String |
Accept-Ranges | Range unit. HTTP standard header | String |
Content-Range | Content range. HTTP standard header | String |
Cache-Control | Cache policy. Standard HTTP header | String |
Content-Type | Content type. Standard HTTP header | String |
Content-Disposition | File name. Standard HTTP header | String |
Content-Encoding | Encoding format. Standard HTTP header | String |
Content-Language | Language type. HTTP standard header | String |
Content-Length | Object size | String |
Expires | Cache expiration time. HTTP standard header | String |
x-cos-meta-* | User-defined object metadata. It must start with x-cos-meta; otherwise, it will be ignored | String |
x-cos-version-id | Version ID of the object if versioning is enabled | String |