Applicable Scenario
To meet user needs in different scenarios, TDMQ Pulsar offers two product forms: Pro Cluster and Virtual Cluster.
Due to the stability risk of virtual clusters, we stopped adding new ones in 2023. Pro Clusters have stronger product capabilities and a more comprehensive console (manage, renew, scale-out, etc.). To provide better service, for your existing Virtual Clusters, we offer smooth migration between clusters, supporting the smooth migration of Virtual Clusters to Pro Clusters.
Capability Description
The data plane migration process of clusters is almost transparent to users, meaning the migration process is smooth (no need to adjust access points, no need to modify user business code).
Migration Process:
1. The system will deduct based on the Pro Cluster specification. When the migration starts, you can see the deduction status as processing on the order page; during the migration, if any issue occurs and a rollback happens, the order will be automatically refunded; after the migration is complete, the order status will be complete and the timing will officially start.
2. On the console Pro Cluster list page, you can see the information of the migrated cluster and perform subsequent operations such as management, upgrade, and renewal.
Smooth Prerequisite
Smooth migration has a prerequisite: some access point addresses, due to early cluster creation or special network connections, cannot be smoothly migrated to the new cluster.
Therefore, users need to provide the access point address information in use to confirm feasibility.
Possible Issues
1. Message repetition issue
Progress synchronization through individual ack has been implemented to minimize the number of duplicate messages during migration. However, duplicates during migration may not be completely avoided, usually within 1 minute. Users should prepare for idempotent processing in advance if needed.
2. Message disordered
A common issue with cluster migration is that disorder during the migration process cannot be completely avoided, regardless of the solution. Users need to be informed in advance.
3. Monitoring Data
Switching clusters may cause momentary inaccuracies in monitoring data, usually recoverable within 1-2 minutes.
4. Generation time jitter
Switching clusters may cause brief production latency fluctuations, similar to those during cluster upgrades, usually recoverable within 1 minute.
5. Message trace
During data synchronization, consumption progress messages will be generated. Users will see these messages when using message query. Querying message details during the migration process may be affected, and it may be impossible to view them.
6. Migration duration
The entire migration time is related to the number of namespaces, production flow, and the data volume of message storage. For one namespace with 1000 TPS and 100G message storage, cluster migration can usually be completed within 1 hour. For larger data volumes, such as 1T storage, it takes about 2 hours.
7. Old cluster retention time
After the migration is completed, Tencent Cloud's R&D side will wait for 1-3 days before clearing the resources on the old physical cluster. After clearing, rollback will no longer be possible.
8. Message backlog issue
The synchronization of consumption progress during the migration process is done through the user's topic, so there will be some internal messages in the user's topic. These messages will be filtered out on the server-side during consumption, and the business will not actually consume these messages. For subscriptions without consumers, message backlog will occur.
9. Message replication scope
During the message replication process, due to Pulsar's implementation mechanism, only messages within the TTL range of the old cluster can be replicated to the new cluster. If your message retention time is relatively large and you need to synchronize data within the retention time range, you need to adjust the TTL configuration first.
Migration Principle
Technical Solution
Using the Pulsar Geo Replication solution, enabling cross-cluster bidirectional replication can achieve data and message progress synchronization to meet the needs of cluster migration.
Main Migration Process
The main migration process is shown in the figure below. After each process is successful, it proceeds to the next step. If the process fails, it can also be rolled back.
Before resource cleanup, the entire cluster migration process can be rolled back, and the old cluster can continue to provide services.
1. You agree on a migration plan with us, the platform initiates the shipment, and you start building the new professional cluster after the payment order.
2. Cluster metadata synchronization, including namespace, topic, subscription, role, and namespace policies.
3. Enable cross-cluster data sync.
4. Switch clusters, the operation platform issues a tenant cluster switch command, actively triggers the unload of topics on the old cluster, triggers client lookup, and the lookup phase returns the new cluster address information.
5. After confirming the user data migration is successful, disable cross-cluster data sync and clear the old cluster resources.
6. You can see the new cluster in the console.