Downloading Objects

Last updated: 2023-09-13 11:02:44

Feature Overview

This document provides an overview of APIs and SDK code samples for simple operations, multipart operations, and other object operations.
API
Operation
Description
Downloading an object
Downloads an object to the local file system

Advanced APIs (Recommended)

Downloading an object (checkpoint restart)

Note

This advanced download API can determine whether to download a file in whole or in parts based on the file size. It downloads files smaller than or equal to 20 MB in whole and files larger than 20 MB in parts. It also supports automatic checkpoint restart for suspended multipart downloads.

Method prototype

download_file(Bucket, Key, DestFilePath, PartSize=20, MAXThread=5, EnableCRC=False, **Kwargs)

Sample Request

# -*- coding=utf-8
from qcloud_cos import CosConfig
from qcloud_cos import CosS3Client
from qcloud_cos.cos_exception import CosClientError, CosServiceError
import sys
import os
import logging

# Under normal circumstances, use the INFO log level. To locate issues, change it to DEBUG, and the SDK will print communication information with the server.
logging.basicConfig(level=logging.INFO, stream=sys.stdout)

# 1. Set user attributes, including secret_id, secret_key, region, etc. Appid has been removed from CosConfig, please include Appid in the Bucket parameter. Bucket is composed of BucketName-Appid.
secret_id = os.environ['COS_SECRET_ID'] # User <1>SecretId</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.
secret_key = os.environ['COS_SECRET_KEY'] # User <1>SecretKey</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.
region = 'ap-beijing' # Replace it with the actual region, which can be viewed in the console at https://console.cloud.tencent.com/cos5/bucket.
For a list of all regions supported by COS, please visit https://cloud.tencent.com/document/product/436/6224
token = None # Token is required for temporary keys but not permanent keys. For more information about how to generate and use a temporary key, see https://cloud.tencent.com/document/product/436/14048.
scheme = 'https' # Specify whether to use HTTP or HTTPS protocol to access COS. This is optional and is https by default.

config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token, Scheme=scheme)
client = CosS3Client(config)

# Using the advanced interface to download once without retrying, without utilizing the checkpoint restart feature.
response = client.download_file(
Bucket='examplebucket-1250000000',
Key='exampleobject',
DestFilePath='local.txt'
)

# Use the advanced interface for checkpoint restart; failed retries will not download already successful parts (retrying 10 times here)
for i in range(0, 10):
try:
response = client.download_file(
Bucket='examplebucket-1250000000',
Key='exampleobject',
DestFilePath='local.txt')
break
except CosClientError or CosServiceError as e:
print(e)

Sample request with all parameters

def upload_percentage(consumed_bytes, total_bytes):
"""Progress bar callback function, calculating the current upload percentage

:param consumed_bytes: Amount of data already uploaded
:param total_bytes: Total data volume
"""
if total_bytes:
rate = int(100 * (float(consumed_bytes) / float(total_bytes)))
print('\r{0}% '.format(rate))
sys.stdout.flush()

response = client.download_file(
Bucket='examplebucket-1250000000',
Key='exampleobject',
DestFilePath='local.txt',
PartSize=1,
MAXThread=5,
EnableCRC=True,
TrafficLimit='1048576'
IfMatch='"9a4802d5c99dafe1c04da0a8e7e166bf"',
IfModifiedSince='Wed, 28 Oct 2014 20:30:00 GMT',
IfNoneMatch='"9a4802d5c99dafe1c04da0a8e7e166bf"',
IfUnmodifiedSince='Wed, 28 Oct 2014 20:30:00 GMT',
ResponseCacheControl='string',
ResponseContentDisposition='string',
ResponseContentEncoding='string',
ResponseContentLanguage='string',
ResponseContentType='string',
ResponseExpires='string',
VersionId='string',
progress_callback=upload_percentage
)

Description

Parameter name
ParameterDescription
Local Disk Types
Required
Bucket
Bucket name in the format of BucketName-APPID
String
Required
Key
ObjectKey is the unique identifier of the object in the bucket. For example, in the object's access domain name examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com/doc/pic.jpg, the ObjectKey is doc/pic.jpg
String
Required
DestFilePath
Local path to save the downloaded file
String
Required
PartSize
Part size for multipart download. Default value: 20 MB
Int
Not required
MAXThread
Maximum number of concurrent threads for a multipart download. Default value: 5
Int
Not required
EnableCRC
Whether to enable CRC for local and remote files. Default: False
Bool
Not required
TrafficLimit
The rate limit value for a single connection is in bits per second (bit/s), with a range of 819200 to 838860800, or 800Kb/s to 800Mb/s. The advanced interface restricts the speed of a single thread.
String
Not required
IfMatch
Returns the object only if the Etag matches the specified content
String
Not required
IfModifiedSince
Returns the object metadata only if the object is modified after the time specified in GMT format
String
Not required
IfNoneMatch
Returns the object only if the Etag does not match the specified content
String
Not required
IfUnmodifiedSince
Returns the object only if it is modified before or at the time specified in GMT format
String
Not required
ResponseCacheControl
Sets Cache-Control in the response header
String
Not required
ResponseContentDisposition
Sets Content-Disposition in the response header
String
Not required
ResponseContentEncoding
Sets Content-Encoding in the response header
String
Not required
ResponseContentLanguage
Sets Content-Language in the response header
String
Not required
ResponseContentType
Sets Content-Type in the response header
String
Not required
ResponseExpires
Expires of the response header
String
Not required
VersionId
Version ID of the object to download
String
Not required
progress_callback
Callback function for the upload progress. You can customize this function to query the upload progress.
Func
Not required

Response description

None

Batch download (downloading a COS directory)

Note

The following sample uses the basic SDK APIs to download objects in a COS directory to the local file system.

Sample Request

# -*- coding=utf-8
import os
import logging
import sys
import json
import os

from qcloud_cos import CosConfig, CosServiceError
from qcloud_cos import CosS3Client
from qcloud_cos.cos_threadpool import SimpleThreadPool

# Under normal circumstances, use the INFO log level. To locate issues, change it to DEBUG, and the SDK will print communication information with the server.
logging.basicConfig(level=logging.INFO, stream=sys.stdout)

# 1. Set user attributes, including secret_id, secret_key, region, etc. Appid has been removed from CosConfig, please include Appid in the Bucket parameter. Bucket is composed of BucketName-Appid.
secret_id = os.environ['COS_SECRET_ID'] # User <1>SecretId</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.
secret_key = os.environ['COS_SECRET_KEY'] # User <1>SecretKey</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.
region = 'ap-beijing' # Replace it with the actual region, which can be viewed in the console at https://console.cloud.tencent.com/cos5/bucket.
For a list of all regions supported by COS, please visit https://cloud.tencent.com/document/product/436/6224
token = None # Token is required for temporary keys but not permanent keys. For more information about how to generate and use a temporary key, see https://cloud.tencent.com/document/product/436/14048.
scheme = 'https' # Specify whether to use HTTP or HTTPS protocol to access COS. This is optional and is https by default.

config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token, Scheme=scheme) # Obtain the object to configure.
client = CosS3Client(config)

# User's bucket information
test_bucket = 'examplebucket-1250000000'
start_prefix = 'data/'
Object storage relies on the delimiter '/' to simulate directory semantics.
# Using the default empty separator, you can list all child nodes under a directory, achieving a similar effect to local directory recursion.
# If the delimiter is set to "/", you need to handle subdirectories recursively in the program.
delimiter = ''


# List the subnodes of the current directory and return information about all subnodes.
def listCurrentDir(prefix):
file_infos = []
sub_dirs = []
marker = ""
count = 1
while True:
response = client.list_objects(test_bucket, prefix, delimiter, marker)
Debug output
# json_object = json.dumps(response, indent=4)
# print(count, " =======================================")
# print(json_object)
count += 1

if "CommonPrefixes" in response:
common_prefixes = response.get("CommonPrefixes")
sub_dirs.extend(common_prefixes)

if "Contents" in response:
contents = response.get("Contents")
file_infos.extend(contents)

if "NextMarker" in response.keys():
marker = response["NextMarker"]
else:
break

print("=======================================================")

If the delimiter is set to "/", recursive processing of subdirectories is required.
# sorted(sub_dirs, key=lambda sub_dir: sub_dir["Prefix"])
# for sub_dir in sub_dirs:
# print(sub_dir)
# sub_dir_files = listCurrentDir(sub_dir["Prefix"])
# file_infos.extend(sub_dir_files)

print("=======================================================")

sorted(file_infos, key=lambda file_info: file_info["Key"])
for file in file_infos:
print(file)
return file_infos


# Download the file to a local directory; if there is already a file with the same name in the local directory, it will be overwritten.
# If the directory structure does not exist, it will create a directory structure identical to the object storage.
def downLoadFiles(file_infos):
localDir = "./download/"

pool = SimpleThreadPool()
for file in file_infos:
# File download: Retrieve the file to the local system
file_cos_key = file["Key"]
localName = localDir + file_cos_key

# Recursively create the local directory structure if it does not exist
if not os.path.exists(os.path.dirname(localName)):
os.makedirs(os.path.dirname(localName))

# skip dir, no need to download it
if str(localName).endswith("/"):
continue

# Actual file download
# Utilizing a thread pool method
pool.add_task(client.download_file, test_bucket, file_cos_key, localName)

Simple download method
# response = client.get_object(
# Bucket=test_bucket,
# Key=file_cos_key,
# )
# response['Body'].get_stream_to_file(localName)

pool.wait_completion()
return None


# Function encapsulation: Download a directory from object storage to the local disk
def downLoadDirFromCos(prefix):
global file_infos

try:
file_infos = listCurrentDir(prefix)

except CosServiceError as e:
print(e.get_origin_msg())
print(e.get_digest_msg())
print(e.get_status_code())
print(e.get_error_code())
print(e.get_error_msg())
print(e.get_resource_location())
print(e.get_trace_id())
print(e.get_request_id())

downLoadFiles(file_infos)
return None


if __name__ == "__main__":
downLoadDirFromCos(start_prefix)

Simple Operations

Downloading an object

Note

This API (GET Object) is used to download an object.

Method prototype

get_object(Bucket, Key, **kwargs)

Sample 1: downloading an object

# -*- coding=utf-8
from qcloud_cos import CosConfig
from qcloud_cos import CosS3Client
import sys
import os
import logging

# Under normal circumstances, use the INFO log level. To locate issues, change it to DEBUG, and the SDK will print communication information with the server.
logging.basicConfig(level=logging.INFO, stream=sys.stdout)

# 1. Set user attributes, including secret_id, secret_key, region, etc. Appid has been removed from CosConfig, please include Appid in the Bucket parameter. Bucket is composed of BucketName-Appid.
secret_id = os.environ['COS_SECRET_ID'] # User <1>SecretId</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.
secret_key = os.environ['COS_SECRET_KEY'] # User <1>SecretKey</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.
region = 'ap-beijing' # Replace it with the actual region, which can be viewed in the console at https://console.cloud.tencent.com/cos5/bucket.
For a list of all regions supported by COS, please visit https://cloud.tencent.com/document/product/436/6224
token = None # Token is required for temporary keys but not permanent keys. For more information about how to generate and use a temporary key, see https://cloud.tencent.com/document/product/436/14048.
scheme = 'https' # Specify whether to use HTTP or HTTPS protocol to access COS. This is optional and is https by default.

config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token, Scheme=scheme)
client = CosS3Client(config)

response = client.get_object(
Bucket='examplebucket-1250000000',
Key='exampleobject'
)
response['Body'].get_stream_to_file('exampleobject')

Sample 2: Partially downloading an object

# -*- coding=utf-8
from qcloud_cos import CosConfig
from qcloud_cos import CosS3Client
import sys
import os
import logging

# Under normal circumstances, use the INFO log level. To locate issues, change it to DEBUG, and the SDK will print communication information with the server.
logging.basicConfig(level=logging.INFO, stream=sys.stdout)

# 1. Set user attributes, including secret_id, secret_key, region, etc. Appid has been removed from CosConfig, please include Appid in the Bucket parameter. Bucket is composed of BucketName-Appid.
secret_id = os.environ['COS_SECRET_ID'] # User <1>SecretId</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.
secret_key = os.environ['COS_SECRET_KEY'] # User <1>SecretKey</1>. We recommend that you use a sub-account key and follow the principle of least privilege to reduce risks. For more information on how to obtain a sub-account key, visit https://cloud.tencent.com/document/product/598/37140.
region = 'ap-beijing' # Replace it with the actual region, which can be viewed in the console at https://console.cloud.tencent.com/cos5/bucket.
For a list of all regions supported by COS, please visit https://cloud.tencent.com/document/product/436/6224
token = None # Token is required for temporary keys but not permanent keys. For more information about how to generate and use a temporary key, see https://cloud.tencent.com/document/product/436/14048.
scheme = 'https' # Specify whether to use HTTP or HTTPS protocol to access COS. This is optional and is https by default.

config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token, Scheme=scheme)
client = CosS3Client(config)

response = client.get_object(
Bucket='examplebucket-1250000000',
Key='exampleobject',
Range='bytes=0-100'
)
response['Body'].get_stream_to_file('exampleobject')

Sample request with all parameters

response = client.get_object(
Bucket='examplebucket-1250000000',
Key='exampleobject',
Range='string',
IfMatch='"9a4802d5c99dafe1c04da0a8e7e166bf"',
IfModifiedSince='Wed, 28 Oct 2014 20:30:00 GMT',
IfNoneMatch='"9a4802d5c99dafe1c04da0a8e7e166bf"',
IfUnmodifiedSince='Wed, 28 Oct 2014 20:30:00 GMT',
ResponseCacheControl='string',
ResponseContentDisposition='string',
ResponseContentEncoding='string',
ResponseContentLanguage='string',
ResponseContentType='string',
ResponseExpires='string',
VersionId='string',
TrafficLimit='819200'
)

Description

Parameter name
ParameterDescription
Local Disk Types
Required
Bucket
Bucket name in the format of BucketName-APPID
String
Required
Key
ObjectKey is the unique identifier of the object in the bucket. For example, in the object's access domain name examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com/doc/pic.jpg, the ObjectKey is doc/pic.jpg
String
Required
Range
Sets the byte range of the object to download in the format of bytes=first-last.
String
Not required
IfMatch
Returns the object only if the Etag matches the specified content
String
Not required
IfModifiedSince
Returns the object metadata only if the object is modified after the time specified in GMT format
String
Not required
IfNoneMatch
Returns the object only if the Etag does not match the specified content
String
Not required
IfUnmodifiedSince
Returns the object only if it is modified before or at the time specified in GMT format
String
Not required
ResponseCacheControl
Sets Cache-Control in the response header
String
Not required
ResponseContentDisposition
Sets Content-Disposition in the response header
String
Not required
ResponseContentEncoding
Sets Content-Encoding in the response header
String
Not required
ResponseContentLanguage
Sets Content-Language in the response header
String
Not required
ResponseContentType
Sets Content-Type in the response header
String
Not required
ResponseExpires
Expires of the response header
String
Not required
VersionId
Version ID of the object to download
String
Not required
TrafficLimit
The rate limit value for a single connection is in bits per second (bit/s), with a range of 819200 to 838860800, or 800Kb/s to 800Mb/s. The advanced interface restricts the speed of a single thread.
String
Not required

Response description

The response contains the content and metadata of the downloaded object in dict format:
{
'Body': StreamBody(),
'ETag': '"9a4802d5c99dafe1c04da0a8e7e166bf"',
'Last-Modified': 'Wed, 28 Oct 2014 20:30:00 GMT',
'Accept-Ranges': 'bytes',
'Content-Range': 'bytes 0-16086/16087',
'Cache-Control': 'max-age=1000000',
'Content-Type': 'application/octet-stream',
'Content-Disposition': 'attachment; filename="filename.jpg"',
'Content-Encoding': 'gzip',
'Content-Language': 'zh-cn',
'Content-Length': '16807',
'Expires': 'Wed, 28 Oct 2019 20:30:00 GMT',
'x-cos-meta-test': 'test',
'x-cos-version-id': 'MTg0NDUxODMzMTMwMDM2Njc1ODA',
'x-cos-request-id': 'NTg3NzQ3ZmVfYmRjMzVfMzE5N182NzczMQ=='
}
Parameter name
ParameterDescription
Local Disk Types
Body
Content of the downloaded object. You can use the get_raw_stream() method to get a file stream and the get_stream_to_file(local_file_path) method to download the object’s content to a specified local file.
StreamBody
ETag
MD5 checksum of the object
String
Last-Modified
Time the object was last modified
String
Accept-Ranges
Range unit. HTTP standard header
String
Content-Range
Content range. HTTP standard header
String
Cache-Control
Cache policy. Standard HTTP header
String
Content-Type
Content type. Standard HTTP header
String
Content-Disposition
File name. Standard HTTP header
String
Content-Encoding
Encoding format. Standard HTTP header
String
Content-Language
Language type. HTTP standard header
String
Content-Length
Object size
String
Expires
Cache expiration time. HTTP standard header
String
x-cos-meta-*
User-defined object metadata. It must start with x-cos-meta; otherwise, it will be ignored
String
x-cos-version-id
Version ID of the object if versioning is enabled
String