Help & Documentation>TDMQ for RocketMQ

Metadata Migration

Last updated: 2024-10-14 10:57:01

On this page:

Scenario

When using a TDMQ for RocketMQ cluster, you may need to migrate your existing business, for example, from a self-built or third-party RocketMQ cluster to the TDMQ for RocketMQ cluster.
This document describes how to migrate metadata from self-built open-source RocketMQ to TDMQ for RocketMQ.
Note
Both dedicated clusters and virtual clusters support metadata migration, and the migration steps are the same.

Preparations

You have already created a cluster and a namespace in the console.

Instructions

Step 1. Export the metadata file

If you are using self-built open-source RocketMQ, you can export metadata in the following two ways:

Method A

If your RocketMQ server can access the public network, run the following script on your server directly (if there are multiple servers in your RocketMQ cluster, you can run the script on any server as long as the network is interconnected in the cluster).
/bin/bash -c "$(curl -fsSL https://rocketmq-1306598660.cos.ap-guangzhou.myqcloud.com/rocketmq-export.sh)"

Method B

If your RocketMQ server has no permission to access the public network, follow the steps below:
1. Download the Cloud Migration Tool to your local machine.
2. Upload the tool to your self-built RocketMQ cluster (if there are multiple servers in your RocketMQ cluster, you can run the tool on any server as long as the network is interconnected in the cluster).
3. Decompress the tool and enter the directory.
unzip rocketmq-migration.zip
cd rocketmq-migration
4. Run the following command for migration.
// Execute this command primarily to use the exported tool file as the executable file.
ROCKETMQ_HOME=pwd
//Initiating the export process
./bin/export.sh
// Enter the open-source RocketMQ address, such as localhost:9876
Enter name server address list:localhost:9876
// Select a cluster to export, such as DefaultCluster
Choose a cluster to export:DefaultCluster
// Enter a directory for saving the exported metadata, which is /tmp/rocketmq/config/rocketmq-metadata-export.json by default.
Enter file path to export [default /tmp/rocketmq/export]:

Step 2. Create a migration task

1. Log in to the TDMQ for RocketMQ console, navigate to the Migration to Cloud page, access the migration task list page, and click Create Task to establish a corresponding migration task.

2. Select a migration task type:
Cluster Migration: This involves migrating the metadata from the original self-built cluster to the newly created cluster. During the "Cluster Migration" task, the migration tool will by default parse the part before "%" in the topic name in the open-source RocketMQ Dashboard as the namespace name, thereby creating multiple logically isolated namespaces for you. If the original self-built cluster did not distinguish namespaces, the system will generate a default namespace for you.
Import to a specified namespace: Migrate the metadata from the original self-built cluster to a specified namespace. If the original topics were not differentiated by namespace or business category, you can take this opportunity to migrate to the cloud. During the import process, select specific topics and groups and assign them to different namespaces to distinguish between business types or environments.

3. Upload the metadata file obtained in step 1 and select the topics and groups you want to import.
Note
Up to 1,000 topics and 1,000 groups can be imported in a single task. Excess data will fail to be imported.


Step 3. Check the task status

Upon successful creation of the task, navigate to the task list to monitor its progress. If the data volume is substantial, the task may require some time to load. By clicking on View Details, you can inspect the specific operational status of the task.

If the task status is "Some failed" or "All failed", you can filter causes of the failures in the Task Status column.


中国站 - English