CKafka Pro Edition supports Multi-AZ deployment. When purchasing CKafka instances in regions with three or more AZs, you can select up to four AZs for cross-AZ instances. Partition replicas of the instance will be distributed across nodes in different AZs, ensuring that your instance remains operational even if a single AZ becomes unavailable.
Note
Only the Pro Edition supports multi-AZ deployment.
How multi-AZ deployment of CKafka works
The multi-AZ deployment feature of CKafka involves three layers: network, data, and control.
Network Layer
CKafka exposes a VIP to the client. After the client is connected to the VIP, it will get the metadata information (which is generally addresses mapped one to one at different ports on the same VIP) of the topic partitions .
It is a VIP that can failover to another AZ at any time. When an AZ becomes unavailable, the VIP automatically migrates to another available node in the region, achieving cross-AZ disaster recovery.
Data Layer
CKafka's data layer and native Kafka adopt the same distributed deployment method, with multiple data replicas distributed across different broker nodes, each deployed in a different availability zone. When processing a partition, there is a leader-follower relationship between different nodes. If the leader is offline due to an anomaly, the cluster control node (Controller) will elect a new partition leader to handle requests for that partition.
For clients, if an AZ becomes unavailable and the leader of a topic partition is located on a broker node in the unavailable AZ, the established connections may experience timeouts or be closed. After the leader node becomes unavailable, the Controller (if the Controller node is unavailable, the remaining nodes will elect a new Controller) will elect a new leader node to provide services. The leader switch time is in seconds (the specific switch time is proportional to the number of cluster nodes and the size of metadata). Clients will periodically refresh the topic partition metadata information and connect to the new leader node for production and consumption.
Control Layer
CKafka's control layer and native Kafka utilize the same technology, relying on Zookeeper for broker node service discovery and cluster Controller election. For CKafka instances that support multi-AZ deployment, the Zookeeper cluster nodes (hereinafter referred to as zk nodes) are deployed across three availability zones (or data centers). If a zk node in any of these AZs fails or disconnects, the entire Zookeeper cluster can still function normally.
Pros and Cons of Multi-AZ Deployment
Advantages
Multi-AZ deployment can significantly improve the disaster recovery capability of the cluster. When a single AZ experiences force majeure events such as network instability and restart after power outage, the client can resume message production and consumption after waiting a short while to reconnect.
Drawbacks
When using multi-AZ deployment, partition replicas are distributed across multiple AZs, resulting in additional cross-AZ network latency for message replication compared to single-AZ deployment. This latency directly affects the write duration for producer clients with an Ack parameter greater than 1 or equal to -1 (all). Currently, the cross-AZ latency in major regions such as Guangzhou, Shanghai, and Beijing ranges from 10ms to 40ms.
Multi-AZ Deployment Scenarios
Unavailability of one single AZ
After a single AZ becomes unavailable, as explained above, the client will disconnect, reconnect, and then provide services normally after reconnection.
As multi-AZ deployment is currently not supported for the management of API services, after a single AZ becomes unavailable, you may not be able to create topics, configure ACL policies, or view monitoring data in the console, but this will not affect the production and consumption of existing businesses.
Network isolation between two AZs
If network isolation occurs between two AZs (that is, they cannot communicate), a cluster "split-brain" event may occur; in other words, the nodes in both AZs provide services, but data written in one of the AZs will be treated as dirty data after the cluster is recovered.
Consider the following scenario: the cluster Controller node and one zk node become network-isolated from the other nodes. In this case, the other nodes will elect a new Controller (since the majority of zk nodes have normal network communication, the Controller can be successfully elected). However, the network-isolated Controller still believes it is the Controller node, leading to a split-brain situation in the cluster.
In this scenario, client write operations need to be considered based on different cases. For example, when the client's Ack policy is set to -1 or "all" and the number of replicas is 2, assuming a 3-node cluster, the distribution after a split-brain situation would be 2:1. If the original leader was in the region of node 1, the partition write operation would fail, while the other side would write successfully. However, if there are 3 replicas and the Ack policy is set to -1 or "all", neither side will succeed in writing. In this case, the further handling plan should be determined based on the specific parameter configuration.
After the cluster network is back to normal, the client can resume production and consumption without performing any operations. As the server will normalize the data again, the data on one of the split nodes will be truncated directly, which, however, will not result in data loss in case of multi-replica multi-AZ data storage.
Region: select a region close to the resource for client deployment.
AZ: select an AZ according to your actual needs.
Standard Edition: it does not support multi-AZ deployment.
Pro Edition: If the current region supports multi-AZ deployment, you can select up to four AZs for deployment. For more information on cross-AZ deployment, see Cross-AZ Deployment.
Product Specs: select an appropriate model based on the peak bandwidth and disk capacity.
Message Retention Period: The value range is 24–2160 hours.
When the disk capacity is insufficient (that is, the disk utilization reaches 90%), old messages will be deleted in advance to ensure the service availability.
VPC: If you need to access other VPCs, you can modify the routing access rules as instructed in Adding Routing Policy.
Tag: It is optional. For more information, see Tag Overview.
Instance Name: When purchasing multiple instances, you can batch create instances by its numeric suffix (which is numbered in an ascending order) or its designated pattern string. For detailed directions, see Naming with Consecutive Numeric Suffixes or Designated Pattern String.
4. Click Buy Now to complete the instance creation process.
