zookeeper学习系列：四、Paxos算法和zookeeper的关系

一、问题起源

淘宝搜索的博客 http://www.searchtb.com/2011/01/zookeeper-research.html  提到Paxos是zookeeper的灵魂

有一篇文章标题更是以“Zookeeper全解析——Paxos作为灵魂” 作为标题，认为是zookeeper的基础：

“

Google的Chubby，Apache的Zookeeper都是基于它的理论来实现的，Paxos还被认为是到目前为止唯一的分布式一致性算法，其它的算法都是Paxos的改进或简化。有个问题要提一下，Paxos有一个前提：没有拜占庭将军问题。就是说Paxos只有在一个可信的计算环境中才能成立，这个环境是不会被入侵所破坏的。

”

Paxos 这个算法是Leslie Lamport在1990年提出的一种基于消息传递的一致性算法.Paxos 算法解决的问题是一个分布式系统如何就某个值（决议）达成一致。

part-time parliament Paxos Made Simple里这样描述Paxos算法执行过程：

1. prepare 阶段：
   1. proposer 选择一个提案编号 n 并将 prepare 请求发送给 acceptors 中的一个多数派；
   2. acceptor 收到 prepare 消息后，如果提案的编号大于它已经回复的所有 prepare 消息，则 acceptor 将自己上次接受的提案回复给 proposer，并承诺不再回复小于 n 的提案；
2. 批准阶段：
   1. 当一个 proposer 收到了多数 acceptors 对 prepare 的回复后，就进入批准阶段。它要向回复 prepare 请求的 acceptors 发送 accept 请求，包括编号 n 和根据 P2c 决定的 value（如果根据 P2c 没有已经接受的 value，那么它可以自由决定 value）。
   2. 在不违背自己向其他 proposer 的承诺的前提下，acceptor 收到 accept 请求后即接受这个请求。

wiki上是两个阶段，论文里却是说三阶段，而且默认就有了个proposer相当于leader。查资料有大侠列出了第三个阶段(http://www.wuzesheng.com/?p=2724)：

3. Learn阶段:

• 当各个Acceptor达到一致之后，需要将达到一致的结果通知给所有的Learner.

zookeeper采用org.apache.zookeeper.server.quorum.FastLeaderElection作为其缺省选举算法

这篇文章http://csrd.aliapp.com/?p=162&replytocom=782 直接用paxos实现作为标题，提到  zookeeper在选举leader的时候采用了paxos算法(主要是fast paxos)

偶然看到下边有人反驳：

魏讲文：“

FastLeaderElection根本不是Paxos，也不是Fast Paxos的实现。 FastLeaderElection源码与Paxos的论文相去甚远。

Paxos与 FastPaxos算法中也有一个leader选举的问题。

FastLeaderElection对于zookeeper来讲，只是相当于Paxos中的leader选举。

”

 二、资料证实

好的，查查资料，分析源码开始调研

首先是魏讲文的反驳 ：

There is a very common misunderstanding that the leader election algorithm in zookeeper is paxos or fast paxos. The leader election algorithm is not paxos or fast paxos, please consider the following facts:

1. There is no the concept of proposal number in the leader election in zookeeper. the proposal number is a key concept to paxos. Some one think the epoch is the proposal number, but different followers may produce proposal with the same epoch which is not allowed in paxos.
2. Fast paxos requires at least 3t + 1 acceptors, where t is the number of servers which are allowed to fail [3]. This is conflict with the fact that a zookeeper cluster with 3 servers works well even if one server failed.
3. The leader election algorithm must make sure P1. However paxos does provide such guarantee.
4. In paxos, a leader is also required to achieve progress. There are some similarities between the leader in paxos and the leader in zookeeper. Even if more than one servers believe they are the leader, the consistency is preserved both in zookeeper and in paxos. this is very clearly discussed in [1] and [2]. 

然后是作者三次对比

1）邮件列表

Our protocol instead, has only two phases, just like a two-phase 
commit protocol. Of course, for Paxos, we can ignore the first phase in runs in 
which we have a single proposer as we can run phase 1 for multiple instances at 
a time, which is what Ben called previously Multi-Paxos, I believe. The trick 
with skipping phase 1 is to deal with leader switching. 

2）出书的访谈

We made a few interesting observations about Paxos when contrasting it to Zab, like problems you could run into if you just implemented Paxos alone. Not that Paxos is broken or anything, just that in our setting, there were some properties it was not giving us. Some people still like to map Zab to Paxos, and they are not completely off, but the way we see it, Zab matches a service like ZooKeeper well.

zk的分布式一致性算法有了个名称叫Zab

3）论文

We use an algorithm that shares some of the character- istics of Paxos, but that combines transaction logging needed for consensus with write-ahead logging needed for data tree recovery to enable an efficient implementa- tion. 

 三、leader选举分析

在我理解首先在选举时，并不能用到paxos算法，paxos里选总统也好，zk选leader也好，跟搞个提案让大部分人同意是有区别的。选主才能保证不会出现多proposer的并发提案冲突

谁去作为proposer发提案？是paxos算法进行下去的前提。而提出提案让大部分follower同意则可用到类似paxos的算法实现一致性。zookeeper使用的是Zab算法实现一致性。

zk的选主策略：

there can be at most one leader (proposer) at any time, and we guarantee this by making sure 
that a quorum of replicas recognize the leader as a leader by committing to an 
epoch change. This change in epoch also allows us to get unique zxids since the 
epoch forms part of the zxid. 

每个server有一个id，收到提交的事务时则有一个zxid，随更新数据的变动，事务编号递增，server id各不同。首先选zxid最大的作为leader，如果zxid比较不出来，则选server id最大的为leader

zxid包含一个epoch数字，epoch指示一个server作为leader的时期，随新的leader诞生而递增。

再看代码：

四、zookeeper数据更新原理分析

了解完选主的做法后，来了解下同步数据的做法，同步数据则采用Zab协议：Zookeeper Atomic broadcast protocol，是个类似两阶段提交的协议：

1. The leader sends a PROPOSAL message, p, to all followers.
2. Upon receiving p, a follower responds to the leader with an ACK, informing the leader that it has accepted the proposal.
3. Uponreceivingacknowledgmentsfromaquorum(thequorumincludestheleader itself), the leader sends a message informing the followers to COMMIT it. 

跟paxos的区别是leaer发送给所有follower，而不是大多数，所有follower都要确认并通知leader，而不是大多数。

保证机制：按顺序广播的两个事务， T 和 Tʹ ，T在前则Tʹ 生效前必须提交T。如果有一个server 提交了T 和 Tʹ ，则所有其他server必须也在Tʹ前提交T。

五、leader的探活

为解决leader crash的问题，避免出现多个leader导致事务混乱，Zab算法保证：

1、新事务开启时，leader必须提交上个epoch期间提交的所有事务

2、任何时候都不会有两个leader同时获得足够多的支持者。

一个新leader的起始状态需要大多数server同意

六、observer

zk里的第三种角色，观察者和follower的区别就是没有选举权。它主要是1、为系统的读请求扩展性存在 2、满足多机房部署需求，中心机房部署leader、follower，其他机房部署observer，读取配置优先读本地。

七、总结

我认为zookeeper只能说是受paxos算法影响，角色划分类似，提案通过方式类似，实现更为简单直观。选主基于voteid(server-id)和zxid做大小优先级排序，信息同步则使用两阶段提交，leader获取follower的全部同意后才提交事务，更新状态。observer角色则是为了增加系统吞吐和满足跨机房部署。

参考文献

[1] Reed, B., & Junqueira, F. P. (2008). A simple totally ordered broadcast protocol. Second Workshop on Large-Scale Distributed Systems and Middleware (LADIS 2008). Yorktown Heights, NY: ACM. ISBN: 978-1-60558-296-2. [2] Lamport, L. Paxos made simple. ACM SIGACT News 32, 4 (Dec. 2001), 1825. [3] F. Junqueira, Y. Mao, and K. Marzullo. Classic paxos vs. fast paxos: caveat emptor. In Proceedings of the 3rd USENIX/IEEE/IFIP Workshop on Hot Topics in System Dependability (HotDep.07). Citeseer, 2007.

[4]O'Reilly.ZooKeeper.Distributed process coordination.2013

[5] http://agapple.iteye.com/blog/1184023  zookeeper项目使用几点小结