Overview
Through the Metadata Migration Task, customers can synchronize the metadata of their self-built RocketMQ cluster to TDMQ for RocketMQ. After the metadata migration is completed, customers need to modify the access information of the producer cluster and consumer cluster, and migrate them from the old self-built RocketMQ to TDMQ for RocketMQ. This will ultimately enable all message sending and receiving operations to be conducted on the new RocketMQ cluster.
Note
The migration of the messaging service only migrates the message production and consumption links, and does not migrate the message data on the original RocketMQ cluster. Currently, it is only applicable for migration to dedicated clusters. Shared clusters will be supported after the public beta testing is completed.
Migration Directions
This document primarily discusses the methods of migrating messaging services using the Dual Read and Write and Batch Release strategies. During the migration process, the producer and consumer clusters can simultaneously produce or consume messages on both the original and new RocketMQ clusters. This ensures that there is no data backlog due to migration, allowing for a smooth transition. For specific strategies and operational procedures, please refer to the following diagram.
The specific steps are as follows:
1. Upon completion of the TDMQ for RocketMQ cluster creation and the relevant metadata migration, obtain the necessary client information, such as the AccessKey and SecretKey of the new cluster, from the console.
2. Alter the access information for a portion of the nodes in the consumer cluster, connecting these consumers to the new version of the RocketMQ cluster. The switched consumers will consume messages from the new RocketMQ cluster, while the remaining consumers will continue to consume messages from the original RocketMQ cluster.
3. Alter the access information of some nodes in the producer cluster, and connect these producers to the new version of RocketMQ. The switched producers will send messages to the new RocketMQ cluster, while the remaining producers will continue to send messages to the old RocketMQ cluster. To prevent message duplication or loss, idempotent logic for message consumption can be prepared in advance.
4. Connect all remaining producers to the new RocketMQ cluster. At this point, all messages will be sent to the new RocketMQ cluster.
5. Check whether there are any unconsumed messages piled up in the original RocketMQ cluster. Once you have confirmed that there are no piled up or unprocessed messages in the original RocketMQ cluster, connect all remaining consumers to the new RocketMQ cluster. This completes the entire data flow migration operation.
Note
If you don't follow the above steps strictly, for example, if you switch producers first and then consumers, message loss may occur.
Before switching the remaining consumers, please ensure that all messages in the original RocketMQ cluster have been consumed to avoid potential consumption omissions. You can determine whether the messages have been consumed by checking the message backlog in the original RocketMQ cluster.
Migration process diagram
Possible Issues
Order Issue
Message sequence cannot be guaranteed due to cluster switch.
Message Duplication
In theory, duplication should not occur. However, in extreme cases, it might happen. For instance, during the switch, if a consumer consumes a message but has not yet acknowledged it to the server (the original RocketMQ cluster), this could cause the message to enter the retry queue and be consumed again. Implementing idempotent processing logic for messages can circumvent this issue.
Consumption latency
During the read switch process, due to partition reallocation, a rebalance between the queue and consumer clients is required, which may cause a brief consumption delay. No additional action is required in this case, as normal operation will resume after the switch is completed.