As more enterprises migrate to the cloud, cost concerns regarding cloud usage are increasingly emphasized. The growth of businesses generates massive storage demands, so how can cost optimization be achieved when storing data in the cloud to alleviate the burden on businesses?
Before optimizing costs, it is essential to understand the cost structure of Tencent Cloud Object Storage (COS). The primary billing components for object storage include storage capacity fees, traffic fees, request fees, data retrieval fees, and management feature fees.
For most enterprises, storage capacity fees and traffic fees are the main components of their cloud storage costs. In terms of storage fees, Tencent Cloud Object Storage offers various storage types, including Standard Storage, Infrequent Access Storage, Archive Storage, and Deep Archive Storage. Each storage type has different product specifications and prices, allowing enterprises to choose the most cost-effective storage type based on their business model. Regarding traffic fees, there are several types of traffic, such as public network outbound traffic, CDN back-to-origin traffic, cross-region replication traffic, and global acceleration traffic. Different business models may result in varying traffic fee structures. For example, if an enterprise primarily operates an e-commerce website with large-scale image distribution, it will have a higher amount of CDN back-to-origin traffic, and thus, its traffic fees will mainly consist of CDN back-to-origin traffic fees.
Optimization 1: Select the appropriate storage type and business region.
Choosing the appropriate storage type and business region based on the business model can significantly optimize an enterprise's storage costs.
Object storage offers a diverse range of storage types for enterprises, allowing them to choose different storage types based on their performance, data durability, and business availability requirements, while incurring varying costs. Standard Storage has relatively higher storage fees but provides the lowest read latency. Infrequent Access Storage, Archive Storage, and Deep Archive Storage have lower storage capacity fees, but when downloading data, additional data retrieval fees are incurred, and longer retrieval times are required. Therefore, these storage types are more suitable for scenarios with infrequent data access.
The following table illustrates the storage costs for storing 100 TB of business data in the Guangzhou region for one month using different storage types:
Comparison Item | STANDARD | Infrequent Access Storage | ARCHIVE | DEEP ARCHIVE | MAZ_STANDARD | MAZ_STANDARD_IA |
Storage unit price (CNY/GB/month) | 0.118 | 0.08 | 0.033 | 0.01 | 0.15 | 0.1 |
Traffic unit price (CNY/GB) | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 |
Request Unit Price (CNY/10k times, taking 1 million times as an example) | 0.01 | 0.05 | 0.01 | Read-write Requests: 0.5 Standard retrieval request: 7 | 0.01 | 0.05 |
Retrieval price (CNY/GB) | 0 | 0.02 | Standard Retrieval: 0.06 | Standard retrieval: 0.14 | 0 | 0.02 |
Total cost (100TB storage + no downloads) | 12083.20 | 8192.00 | 3379.20 | 1024 | 15360.00 | 10240.00 |
Total cost (100TB storage + 100TB download + 1 million requests + 100TB retrieval) | 62084.20 | 60245.00 | 59524.20 | 66110.00 | 65361.00 | 62293.00 |
Total cost (100TB storage + 500TB download + 1 million requests + 500TB retrieval) | 262084.20 | 268437.00 | 284100.20 | 323454.00 | 265361.00 | 270485.00 |
Note
To learn about the unit prices for other regions and storage types, please refer to the COS Pricing page.
As seen in the table, if the business data download volume is low, choosing Archive Storage or even Deep Archive Storage can effectively reduce storage costs, with the coldest Deep Archive Storage saving up to 90% of storage fees compared to Standard Storage. However, if the business data requires frequent downloads, the retrieval fees of Infrequent Access Storage, Archive Storage, and Deep Archive Storage will result in additional cost overheads, leading to higher overall expenses.
In specific business scenarios, we recommend:
1. Frequent read-write scenarios: For businesses with more reads than writes, such as UGC scenarios and e-commerce images, Standard Storage can be used. If the business requires high availability and data durability, consider using Multi-AZ Standard Storage.
2. Low-frequency read scenarios (once a month): For businesses such as log data analysis and cloud storage data, where the read frequency is low but high performance is required during reads, Infrequent Access Storage can be used. For businesses with high requirements for availability and data durability, Infrequent Access Storage (multi-AZ) is recommended.
3. Very infrequent read scenarios (once every three months): For businesses such as video surveillance and log data archiving, where the read frequency is extremely low and read performance requirements are minimal, Archive Storage can be used.
4. Infrequent access scenarios (accessed once every six months): For businesses such as medical imaging and archival materials that only require long-term backups and have virtually no demand for read performance, Deep Archive Storage can be used.
Additionally, when selecting different storage types, we recommend that enterprises pay attention to the minimum storage duration and minimum storage unit restrictions for some storage types, as well as the performance differences between them. The following table provides a simple comparison.
Comparison Item | STANDARD | Infrequent Access Storage | ARCHIVE | DEEP ARCHIVE | MAZ_STANDARD | MAZ_STANDARD_IA |
Time to First Byte (TTFB) | Milliseconds | Milliseconds | Minimum 1-minute recovery time. | Restore within a minimum of 12 hours. | Milliseconds | Milliseconds |
Minimum Storage Unit | No limit | 64KB | 64KB | 64KB | No limit | No limit |
Minimum Storage Time | No limit | 30 days | 90 days | 180 days | No limit | 30 days |
Data durability | Eleven nines | Eleven nines | Eleven nines | Eleven nines | Twelve nines | Twelve nines |
Designing for availability in business operations. | 99.95% | 99.95% | 99.95% | 99.95% | 99.995% | 99.995% |
Note
Minimum storage duration: The shortest time a file must be stored in a specific storage type. If the actual storage time is shorter than the minimum duration, the fee will be calculated based on the minimum duration. For example, if a file in Infrequent Access Storage is stored for only 1 day and then deleted, the fee will still be calculated based on a 30-day storage period, as the minimum storage duration for Infrequent Access Storage is 30 days.
Minimum Storage Unit: The minimum file capacity required when a file is stored in a specific storage type. If the file capacity does not meet the minimum requirement, it will be calculated based on the minimum file capacity. For example, Infrequent Access Storage requires a minimum of 64KB. If a file in this storage type only occupies 1KB, the fee will still be calculated based on 64KB.
Optimization 2: Analyze access patterns and implement data tiering.
1. Regularly analyze data access patterns using inventory and access log features.
Analyzing data access patterns can provide data support for selecting a reasonable storage type. Tencent Cloud Object Storage (COS) offers inventory and access log features, which record file metadata information and file access records, respectively, and transfer this information to the user's storage bucket.
Note
For a detailed introduction to the inventory feature, see Inventory Feature Overview.
For a detailed introduction to the access log management feature, please refer to Log Management Overview.
Object Storage offers the COS Select feature, which allows you to search the contents of files. If you have generated a large number of inventory files or log records, you can also purchase an Elastic Map Reduce cluster and set up a Presto cluster for data analysis.
Note
For an overview of the COS Select feature, see Select Overview.
For information on using EMR for analysis, please refer to Analyzing Data in COS with Presto.
Taking the example of retrieving and analyzing data from an inventory file, once the inventory report is delivered to the specified bucket, you can access the console to analyze the designated report. The analysis operation guide is as follows:
Note
For instructions on generating inventory reports, please refer to Enabling Inventory Feature.
The console only supports searching for files smaller than 128MB. If the inventory report has a large capacity or a high number of reports, you can opt for using tools, SDKs, or APIs to make the call.
1. Log in to the COS Console.
2. Click the name of the bucket you wish to configure to enter the bucket list page.
3. Locate the corresponding report list, and in the Operation column on the right, select More > Retrieve.
4. On the object retrieval page, configure the corresponding input parameters, enter the retrieval statement, and click Run SQL to view the retrieval results on the results card page.
Here are some common retrieval statements for inventory reports:
Query the number of files for a specific storage type on a particular day:
select count(*) from cosobject s where s._7 = <storage_class>select count(*) from cosobject s where s._7 = 'Standard'
Query the storage capacity in MB for a specific storage type on a particular day:
select SUM(CAST(s._4 AS FLOAT))/1024/1024 from cosobject s where s._7 = <storage_class>select SUM(CAST(s._4 AS FLOAT))/1024/1024 from cosobject s where s._7 = 'Standard'
Query the number of files smaller than 64KB for a specific storage type:
select count(*) from cosobject s where s._7 = <storage_class> and CAST(s._4 AS FLOAT) < <SIZE>select count(*) from cosobject s where s._7 = 'Standard_IA' and s._4 < 64*1024
Query the number of files with cross-region replication failures within the bucket:
select count(*) from cosobject s where s._9 = 'Failed'
Note
The inventory report does not include header information, so you can only search by entering the corresponding field's serial number. The header and serial number correspondence information for the inventory report is as follows:
Appid | Bucket | Key | Size | LastModifiedDate | ETag | StorageClass | IsMultipartUploaded | ReplicationStatus |
s._1 | s._2 | s._3 | s._4 | s._5 | s._6 | s._7 | s._8 | s._9 |
2. Transition data using lifecycle policies and batch processing.
During business development, data access patterns are constantly changing. By periodically analyzing data access patterns using inventory and access log features, you can "cool down" business data based on the analysis report.
For most data, the access frequency tends to decrease as the storage duration increases. Therefore, enterprises need to adjust their data storage types based on the changing access patterns of their business data to achieve optimal cost control.
Object storage offers a lifecycle feature that helps enterprises periodically transition storage types. Enterprises can analyze their business data access patterns using inventory and access logs, and establish reasonable lifecycle transition rules based on these access patterns.
Taking a customer from a community platform as an example, they use object storage services to store user-uploaded image data. Generally, image data is frequently accessed shortly after being uploaded, and after some time, most data gradually "cools down," with access frequency decreasing. Assuming that for this customer, the access frequency of most image data drops below once per month after 90 days and is virtually unaccessed after 365 days, we can compare the cost scenarios when setting a lifecycle policy and not setting a lifecycle policy:
Comparison Item | Use Standard Storage only. | Use Standard Storage for hot data, and transition to Infrequent Access Storage after 90 days. | Utilize Standard Storage for hot data, transition to Infrequent Access Storage after 90 days, and move to Archive Storage after 365 days. |
Storage unit price (CNY/GB/month, using Guangzhou region as an example) | Standard Storage: 0.118 | Standard Storage: 0.118 Infrequent Access Storage: 0.08 | Standard Storage: 0.118 Infrequent Access Storage: 0.08 Archive Storage: 0.033 |
Storage Time | Twenty-four months | Twenty-four months (3 months of Standard Storage + 21 months of Infrequent Access Storage) | Twenty-four months (3 months of Standard Storage + 9 months of Infrequent Access Storage + 12 months of Archive Storage) |
Total Storage Fees (CNY) | 289996.80 | 208281.60 | 150528.00 |
As observed, using lifecycle rules to manage objects in storage buckets can substantially reduce data storage costs. For long-term data storage, properly configuring lifecycle rules can help businesses reduce storage costs by more than 50%.
In addition to managing the storage type of business data, the lifecycle feature can also be used to manage file fragments and historical version files in storage buckets. File fragments are incomplete file chunks generated when the transmission of large files fails due to unexpected interruptions, such as network disconnections. If there are numerous file fragments in the business data, you can use lifecycle rules to delete expired fragments. Historical version files are old file information generated after enabling version control. These files can be used for data recovery and rollback in case of accidental deletion but will occupy storage space. Businesses can also set an expiration time for deleting historical version files that are no longer in use, achieving a balance between data security and cost.
Refer to Setting Lifecycle to add rules and enable Manage Historical Version Files for downgrading or deleting historical version files. Select Delete Fragments and set an expiration time to clean up any potential file fragments.
For specific businesses that require a one-time conversion of a large number of files to colder storage types without fixed rules (such as specified prefixes or tags), users can utilize the COS Batch Processing (Batch) feature. This feature allows for batch replication of data to other storage types. Additionally, you can add object tags to your business data to set lifecycle rules for batch deletion. The operation steps are as follows:
1. Export the list of files pending processing and consolidate them into a CSV format file.
2. Create a COSBatch bulk processing task and import the file list.
3. Initiate a batch processing task and wait for its completion.
Optimization 3: Reduce storage capacity by compressing files.
For image data, object storage also provides data compression features to help users reduce image size and storage costs. Currently, the available compression features are as follows:
1. Guetzli Compression: Guetzli compression is a visually lossless compression technique that takes advantage of the human eye's insensitivity to certain color ranges and image details. By selectively discarding detail information without affecting visual quality, it can save approximately 35% - 50% of image size compared to the original image under the same quality conditions.
2. TPG Compression: TPG is a proprietary image format developed by Tencent, which can convert JPG, PNG, GIF, and WEBP images into TPG format, significantly reducing image size with a compression ratio of over 35%.
3. HEIF Compression: For image usage scenarios in iOS environments, images in JPG, PNG, GIF, and WEBP formats can be converted to HEIF format. HEIF format boasts an ultra-high compression ratio, with a compression rate generally above 45%.
Taking a customer's image storage business as an example, the following table compares the cost expenditures for 100 TB of image storage using different compression methods:
Comparison Item | Guetzli compression | TPG compression | WebP compression | HEIF compression |
Original storage fees | 12083.20 | 12083.20 | 12083.20 | 12083.20 |
Compression Ratio Example | 40% | 35% | 20% | 45% |
Compressed Unit Price | 1 CNY/1,000 times | 0.1 USD/1,000 times | ￥0.025 CNY/GB | 0.1 USD/1,000 times |
Storage fees after compression | 4833.28 | 4229.12 | 2416.64 | 5437.44 |
Total cost before compression (100T storage + 500T download) | 262083.20 | 262083.20 | 262083.20 | 262083.20 |
Total fees after compression | 117233.28 | 94829.12 | 56176.64 | 121637.44 |
Compression Ratio Range | 30% - 50%, the larger the image, the better the effect. | Over 35%, the compression ratio remains stable. | Approximately 20%. | Above 45% |
Visual loss | Visually lossless | Almost lossless | Some color gamut loss is noticeable. | Almost lossless |
As seen in the table, using the image compression feature can significantly reduce storage costs, despite incurring some compression costs. Moreover, a large amount of traffic will be generated during subsequent business operations, and compression can greatly save traffic costs.
Conduct a cost review.
Cost optimization should be implemented throughout the entire business process, not just during the initial cloud migration planning. Businesses need to periodically review their costs. On the one hand, as the business grows, storage demands and data access patterns are constantly changing. On the other hand, Tencent Cloud Object Storage is committed to providing users with lower-cost storage services to help reduce costs and increase efficiency. Therefore, conducting regular cost reviews and planning the cloud storage architecture according to business needs is beneficial for reducing storage costs.
In addition, you can also:
1. Go to the Bill Download page in the Billing Center to download your Tencent Cloud billing statement and understand your cloud storage usage details. Analyze your cloud storage consumption and optimize it accordingly.
2. Follow the Tencent Cloud Storage WeChat Official Account or visit the Object Storage Console Overview page to stay informed about new product releases and cost optimization-related updates.