Scenario
This document describes how to use MirrorMaker to migrate data from a self-built Kafka cluster to a CKafka cluster.
Kafka's MirrorMaker tool enables data backup from a self-built Kafka cluster to a CKafka cluster, with the following principles: MirrorMaker consumes messages from the self-built Kafka cluster using a Consumer, then sends the data to the CKafka cluster through a Producer. Finally, you can switch the client's production and consumption configurations to the cloud instance's access network to complete the data migration from the self-built Kafka cluster to the CKafka cluster.
Preparations
You have purchased a CKafka instance.
You have migrated the Topic to the cloud.
Instructions
1. Download the MirrorMaker tool and extract it to your local machine.
Note
2. Configure the
consumer.properties file.# list of brokers used for bootstrapping knowledge about the rest of the cluster# format: host1:port1,host2:port2 ...bootstrap.servers=localhost:9092# consumer group idgroup.id=test-consumer-grouppartition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor# What to do when there is no initial offset in Kafka or if the current# offset does not exist any more on the server: latest, earliest, none#auto.offset.reset=
Category | Note |
bootstrap.servers | Broker access point list of the self-built instance. |
group.id | When migrating data, please ensure that the consumer group ID used does not conflict with the existing consumer names in the self-built instance. |
partition.assignment.strategy | Partition allocation strategy, for example, partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor. |
3. Configure the
producer.properties file.# list of brokers used for bootstrapping knowledge about the rest of the cluster# format: host1:port1,host2:port2 ...bootstrap.servers=localhost:9092# specify the compression codec for all data generated: none, gzip, snappy, lz4compression.type=none
Category | Note |
bootstrap.servers | Copy the access network of the cloud instance from the Access Mode section's Network column on the instance details page in the console.  |
compression.type | Compression type: CKafka does not support GZip compression format. |
4. Start the mirrormaker migration tool in the
.bin directory to begin the migration process.sh bin/kafka-mirror-maker.sh --consumer.config config/consumer.properties --producer.config config/producer.properties --whitelist topicName
Note
whitelist is a regular expression, and topics whose names hit it will be migrated.5. In the .bin directory, run
kafka-consumer-groups.sh to check the consumption progress of the self-built cluster.bin/kafka-consumer-groups.sh --new-consumer --describe --bootstrap-server self-built cluster access point --group test-consumer-group
Note
group is the ID of the consumer group used during data migration.
Subsequent Operations
After data migration, transfer the production and consumption configurations of the client to the access point of the CKafka cluster.