Performance Test on Cloud-native Databases
Performance Test on Cloud-native Databases
Hello，I'm binghe~~
In the past few years, we have seen explosive growth in the cloud database market, with fierce competition between service providers aiming for the top. Notable contenders in this space include AWS Aurora, Alibaba Cloud PolarDB, and Huawei GaussDB.
Having been a MySQL practitioner for many years and published A Practical Guide for MySQL Development, Optimization, and O&M, I frequently receive messages from database administrators and architects at companies who are ready to make the leap to cloud, asking for advice on selecting cloud database services. To make things simple, we can make reliable choices around two major indicators – stability and performance. In this article, I'll test and compare the performance of five database services: PolarDB, ApsaraDB for OceanBase, Aurora, GaussDB, and TDSQL-C. The tests will be performed on these databases with the default configurations to provide a baseline reference.
Abstract
In the low-specification category (8 cores, 64 GB), PolarDB outperforms its contenders across the board in both CPU-bound and I/O-bound tests.
In the high-specification category (64 cores, 512 GB), PolarDB also leads with considerable margin. Largely outperforming its contenders in both CPU-bound and I/O-bound tests, PolarDB has demonstrated its edge in today's cloud database offerings.
It is also worth mentioning that in both categories, we can see an obvious deterioration in performance on Aurora databases in read-write and write-only scenarios.
Figure 1. Test results of the instances in the high-specification category (64 cores, 512 GB)
Figure 2. Test results of the instances in the low-specification category (8 cores, 64 GB)
Scope
The following MySQL-compatible database services are tested:
- PolarDB by Alibaba Cloud
- Aurora by Amazon Web Services
- ApsaraDB for OceanBase by Ant Group
- GaussDB by HUAWEI CLOUD
- TDSQL-C by Tencent Cloud
Controlled Variables
Public cloud database instances are purchased from the cloud service providers and tested out-of-the-box, with no post-configuration. For this test, database instances are divided into two categories, the low-specification category (8 cores, 64 GB), and the high-specification category (64 cores, 512 GB). All instances run MySQL 8.0 and are deployed in the primary/secondary architecture.
Note: ApsaraDB for OceanBase does not offer a 64-core, 512 GB instance type. The 62-core, 400 GB instance type is used instead.
Testing Configurations
CPU-bound and I/O-bound tests are performed on the database services by using the same scripts for each test.
The test results are returned in the following format:
Case Database name, number of tables, table sizes, number of threads, cloud server name, QPS
Where:
- Case: the name of the test script used, which is indicative of the read/write mode. Test scripts used: oltp_read_only, oltp_read_write, and oltp_write_only.
- Database name: the name of the database. Databases used in CPU-bound tests are named tpk_s, while those in I/O-bound tests are named tpk_l.
- Number of tables: the number of tables in each database. Ten tables are specified for each database.
- Table sizes: the number of rows in each table. In CPU-bound tests, each table contains 10 million rows of data. In I/O-bound tests, the table sizes vary based on the category. Tables in the low-specification category contain 40 million rows each, and tables in the high-specification category contain 300 million rows each.
- Number of threads: the number of threads used to perform the test. The number of threads used vary based on the category of the tests. Tests on the low-specification category involve a minimum of 1 thread and a maximum of 128 threads. Tests on the high-specification category involve a minimum of 1 thread and a maximum of 300 threads.
- Cloud server name: the name of the cloud server instance where the database is deployed. Test scripts are not stored in this cloud server.
- QPS: the QPS value, grouped by the number of threads.
The Comparative Analysis section provides side-to-side comparisons of the performance metrics outlined in the preceding section. Raw results from the tests are included in the Tests results section, and are grouped by services.
Comparative Analysis
Low-specification category (8 cores, 64 GB)
The tables in this section provide side-to-side comparisons for databases in the low-specification category.
CPU-bound tests
1）oltp_read_only
Database | QPS (128 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 95,863.56 | 5,184.38 |
ApsaraDB for OceanBase | 61,068.59 | 1,953.32 |
Aurora | 69,933.45 | 3,183.13 |
GaussDB | 85,244.76 | 5,638.32 |
TDSQL-C | 94,343.16 | 4,091.835 |
2）oltp_read_write
Database | QPS (128 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 82,701.53 | 4,809.97 |
ApsaraDB for OceanBase | 39,874.51 | 1,801.23 |
Aurora | 42,649.67 | 2,465.01 |
GaussDB | 58,522.77 | 4,896.80 |
TDSQL-C | 61,997.33 | 2,661.6075 |
3）oltp_write_only
Database | QPS (128 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 96,784.14 | 4,617.05 |
ApsaraDB for OceanBase | 31,767.13 | 1,648.32 |
Aurora | 35,598.10 | 1,484.75 |
GaussDB | 58,697.92 | 2,216.81 |
TDSQL-C | 53,867.95 | 1,799.2725 |
The CPU-bound test results show that PolarDB outperforms its contenders in every test metric.
I/O-bound tests
1）oltp_read_only
Database | QPS (128 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 67,105.08 | 2,943.15 |
ApsaraDB for OceanBase | 33,997.1 | 1,684.21 |
Aurora | 30,695.90 | 1,056.77 |
GaussDB | 24,423.91 | 1,927.34 |
TDSQL-C | 48,069.14 | 2,032.22 |
2）oltp_read_write
Database | QPS (128 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 61,093.90 | 2,891.23 |
ApsaraDB for OceanBase | 29,325.76 | 1,582.34 |
Aurora | 21,751.73 | 683.03 |
GaussDB | 90,774.04 | 4,896.80 |
TDSQL-C | 38,388.89 | 1,667.94 |
3）oltp_write_only
Database | QPS (128 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 61,438.08 | 2,602.82 |
ApsaraDB for OceanBase | 29,876.21 | 1,503.22 |
Aurora | 23,290.39 | 1,001.32 |
GaussDB | 41,209.88 | 1,638.43 |
TDSQL-C | 40,414.65 | 1,559.45 |
In the I/O-bound tests, PolarDB also emerged as the clear winner in every test metric.
High-specification category (64 cores, 512 GB)
The tables in this section provide side-to-side comparisons for databases in the high-specification category.
CPU-bound tests
1）oltp_read_only
Database | QPS (300 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 583,481.91 | 5,299.95 |
ApsaraDB for OceanBase | 210,599.67 | 1,979.00 |
Aurora | 460,661.52 | 3,953.16 |
GaussDB | 324,943.02 | 5,238.16 |
TDSQL-C | 345,769.06 | 4,102.13 |
2）oltp_read_write
Database | QPS (300 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 459,306.28 | 4,998.68 |
ApsaraDB for OceanBase | 161,787.02 | 1,725.00 |
Aurora | 161,193.67 | 2,330.50 |
GaussDB | 231,511.89 | 3,528.90 |
TDSQL-C | 259,070.85 | 2,648.34 |
3）oltp_write_only
Database | QPS (300 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 415,477.31 | 5,025.55 |
ApsaraDB for OceanBase | 102,735.32 | 1,636.21 |
Aurora | 52,484.71 | 1,282.10 |
GaussDB | 195,454.50 | 2,077.85 |
TDSQL-C | 122,732.32 | 1,799.2725 |
The CPU-bound test results show that PolarDB outperforms its contenders in every test metric.
I/O-bound tests
1）oltp_read_only
Database | QPS (300 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 379,448.39 | 3,205.95 |
ApsaraDB for OceanBase | 186,231.85 | 1,663.59 |
Aurora | 162,073.88 | 908.75 |
GaussDB | 202,102.54 | 2,438.24 |
TDSQL-C | 130,938.23 | 2,203.23 |
2）oltp_read_write
Database | QPS (300 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 323,182.93 | 3,147.55 |
ApsaraDB for OceanBase | 142,723.88 | 1,592.08 |
Aurora | 75,080.03 | 807.78 |
GaussDB | 164,403.19 | 2,243.14 |
TDSQL-C | 112,711.66 | 1,782.34 |
3）oltp_write_only
Database | QPS (300 threads) | QPS (1 thread) |
|---|---|---|
PolarDB | 335,549.53 | 3,751.35 |
ApsaraDB for OceanBase | 99,543.00 | 1,447.24 |
Aurora | 26,616.15 | 748.74 |
GaussDB | 152,426.14 | 2,254.31 |
TDSQL-C | 72,023.23 | 1,582.23 |
In the I/O-bound tests, PolarDB also emerged as the clear winner in every test metric, with obvious performance gains compared to its contenders.
Test Results
This section contains the raw results of the tests performed in this article. To ensure the fairness of the tests, all tests are performed on out-of-the-box offerings of different service providers. Controlled variables and testing configurations used for these tests can be found in their corresponding sections.
Low-specification category (8 cores, 64GB)
PolarDB
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 128 | 1 | 95,863.56 | 5,184.38 |
oltp_read_write | 10 | 10,000,000 | 128 | 1 | 82,701.53 | 4,809.97 |
oltp_write_only | 10 | 10,000,000 | 128 | 1 | 96,784.14 | 4,617.05 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 40,000,000 | 128 | 1 | 67,105.08 | 2,943.15 |
oltp_read_write | 10 | 40,000,000 | 128 | 1 | 61,093.90 | 2,891.23 |
oltp_write_only | 10 | 40,000,000 | 128 | 1 | 61,438.08 | 2,602.82 |
ApsaraDB for OceanBase
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 128 | 1 | 61,068.59 | 1,953.32 |
oltp_read_write | 10 | 10,000,000 | 128 | 1 | 39,874.51 | 1,801.23 |
oltp_write_only | 10 | 10,000,000 | 128 | 1 | 31,767.13 | 1,648.32 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 40,000,000 | 128 | 1 | 33,997.1 | 1,684.21 |
oltp_read_write | 10 | 40,000,000 | 128 | 1 | 29,325.76 | 1,582.34 |
oltp_write_only | 10 | 40,000,000 | 128 | 1 | 29,876.21 | 1,503.22 |
Aurora
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 128 | 1 | 69,933.45 | 3,183.13 |
oltp_read_write | 10 | 10,000,000 | 128 | 1 | 42,649.67 | 2,465.01 |
oltp_write_only | 10 | 10,000,000 | 128 | 1 | 35,598.10 | 1,484.75 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 40,000,000 | 128 | 1 | 30,695.90 | 1,056.77 |
oltp_read_write | 10 | 40,000,000 | 128 | 1 | 21,751.73 | 683.03 |
oltp_write_only | 10 | 40,000,000 | 128 | 1 | 23,290.39 | 1,001.32 |
GaussDB
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 128 | 1 | 85,244.76 | 5,638.32 |
oltp_read_write | 10 | 10,000,000 | 128 | 1 | 58,522.77 | 4,896.80 |
oltp_write_only | 10 | 10,000,000 | 128 | 1 | 58,697.92 | 2,216.81 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 40,000,000 | 128 | 1 | 24,423.91 | 1,927.34 |
oltp_read_write | 10 | 40,000,000 | 128 | 1 | 23,178.85 | 1,703.34 |
oltp_write_only | 10 | 40,000,000 | 128 | 1 | 41,209.88 | 1,638.43 |
TDSQL-C
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 128 | 1 | 94,343.16 | 4,091.835 |
oltp_read_write | 10 | 10,000,000 | 128 | 1 | 61,997.33 | 2,661.6075 |
oltp_write_only | 10 | 10,000,000 | 128 | 1 | 53,867.95 | 1,799.2725 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (128 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 40,000,000 | 128 | 1 | 48,069.14 | 2,032.22 |
oltp_read_write | 10 | 40,000,000 | 128 | 1 | 38,388.89 | 1,667.94 |
oltp_write_only | 10 | 40,000,000 | 128 | 1 | 40,414.65 | 1,559.45 |
High-specification category (64 cores, 512 GB)
PolarDB
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 300 | 1 | 583,481.91 | 5,299.95 |
oltp_read_write | 10 | 10,000,000 | 300 | 1 | 459,306.28 | 4,998.68 |
oltp_write_only | 10 | 10,000,000 | 300 | 1 | 415,477.31 | 5,025.55 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 300,000,000 | 300 | 1 | 379,448.39 | 3,205.95 |
oltp_read_write | 10 | 300,000,000 | 300 | 1 | 323,182.93 | 3,147.55 |
oltp_write_only | 10 | 300,000,000 | 300 | 1 | 335,549.53 | 3,751.35 |
ApsaraDB for OceanBase
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 300 | 1 | 210,599.67 | 1,979.00 |
oltp_read_write | 10 | 10,000,000 | 300 | 1 | 161,787.02 | 1,725.00 |
oltp_write_only | 10 | 10,000,000 | 300 | 1 | 102,735.32 | 1,636.21 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 300,000,000 | 30 | 1 | 186,231.85 | 1,663.59 |
oltp_read_write | 10 | 300,000,000 | 30 | 1 | 142,723.88 | 1,592.08 |
oltp_write_only | 10 | 300,000,000 | 30 | 1 | 99,543.00 | 1,447.24 |
Aurora
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 300 | 1 | 460,661.52 | 3,953.16 |
oltp_read_write | 10 | 10,000,000 | 300 | 1 | 161,193.67 | 2,330.50 |
oltp_write_only | 10 | 10,000,000 | 300 | 1 | 52,484.71 | 1,282.10 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 300,000,000 | 300 | 1 | 162,073.88 | 908.75 |
oltp_read_write | 10 | 300,000,000 | 300 | 1 | 75,080.03 | 807.78 |
oltp_write_only | 10 | 300,000,000 | 300 | 1 | 26,616.15 | 748.74 |
GaussDB
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 300 | 1 | 324,943.02 | 5,238.16 |
oltp_read_write | 10 | 10,000,000 | 300 | 1 | 231,511.89 | 3,528.90 |
oltp_write_only | 10 | 10,000,000 | 300 | 1 | 195,454.50 | 2,077.85 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 300,000,000 | 300 | 1 | 202,102.54 | 2,438.24 |
oltp_read_write | 10 | 300,000,000 | 300 | 1 | 164,403.19 | 2,243.14 |
oltp_write_only | 10 | 300,000,000 | 300 | 1 | 152,426.14 | 2,254.31 |
TDSQL-C
1）CPU-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 10,000,000 | 300 | 1 | 345,769.06 | 4,102.13 |
oltp_read_write | 10 | 10,000,000 | 300 | 1 | 259,070.85 | 2,648.34 |
oltp_write_only | 10 | 10,000,000 | 300 | 1 | 122,732.32 | 1,799.2725 |
2）I/O-bound tests
Read or write mode | Number of tables | Table size | Threads (Max) | Threads (Min) | QPS (300 threads) | QPS (1 thread) |
|---|---|---|---|---|---|---|
oltp_read_only | 10 | 300,000,000 | 300 | 1 | 130,938.23 | 2,203.23 |
oltp_read_write | 10 | 300,000,000 | 300 | 1 | 112,711.66 | 1,782.34 |
oltp_write_only | 10 | 300,000,000 | 300 | 1 | 72,023.23 | 1,582.23 |
Well, let's call it a day. I'm binghe. See you next time~~