mongodb 复制集 运维 遇到的问题
root couse： 对MongoDB复制集的认识不足
机器环境：
192.168.12.6 master状态
192.168.12.4 secondary状态
192.168.12.5 secondary状态
192.168.2.1 dump节点 ，之前因为磁盘不足，mongodb进程已宕机，这个实例也配置有vote投票权！
过程：
1、DBA在 192.168.12.5 这个 secondary节点上，执行了关闭实例命令
2、集群剩余的2台主机：192.168.12.4(secondary) 、192.168.12.6(master) ，都变成了secondary状态
3、业务反馈大量报错
4、DBA恢复 192.168.12.5 上面的mongodb进程，集群状态恢复
复盘：
下面的日志，是在 192.168.12.6 主节点上面看到的：
2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 192.168.12.5:27017 - HostUnreachable: Connection reset by peer
2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 192.168.12.5:27017 due to failed operation on a connection
2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to close stream: Transport endpoint is not connected
2019-04-16T15:47:14.196+0800 I REPL [ReplicationExecutor] Error in heartbeat request to 192.168.12.5:27017; HostUnreachable: Connection reset by peer
2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 192.168.12.5:27017
2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 192.168.12.5:27017 - HostUnreachable: Connection refused
2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 192.168.12.5:27017 due to failed operation on a connection
2019-04-16T15:47:14.196+0800 I REPL [ReplicationExecutor] Error in heartbeat request to 192.168.12.5:27017; HostUnreachable: Connection refused
2019-04-16T15:47:14.197+0800 I REPL [ReplicationExecutor] can't see a majority of the set, relinquishing primary
2019-04-16T15:47:14.197+0800 I REPL [ReplicationExecutor] Stepping down from primary in response to heartbeat
2019-04-16T15:47:14.198+0800 I REPL [replExecDBWorker-0] transition to SECONDARY
2019-04-16T15:47:14.274+0800 I NETWORK [conn476944080] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [192.168.3.11:38712]集群的配置如下：
set01:SECONDARY> rs.conf()
{
"_id" : "set01",
"version" : 130099,
"members" : [
{
"_id" : 6,
"host" : "192.168.2.1:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : true,
"priority" : 0,
"tags" : {
"dc" : "IDC1",
"role" : "dump"
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 7,
"host" : "192.168.12.4:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
"dc" : "IDC1"
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 8,
"host" : "192.168.12.5:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
"dc" : "IDC1"
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 9,
"host" : "192.168.12.6:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
"dc" : "IDC1"
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
}
}
}根据上面的内容，可以判断出 192.168.2.1宕机后，我们再次关闭一台192.168.12.5后，集群就剩2个投票，少于一半节点，整个集群无法选举出Primary，集群退化为只读状态【表现为rs.status()看到的都是secondary角色】 因此，通常建议将复制集成员数量设置为奇数。
解决措施：
将dump节点的投票属性去掉。
参考：
本文参与 腾讯云自媒体分享计划，分享自作者个人站点/博客。
原始发表：2019-04-16 ，如有侵权请联系 cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读
相关产品与服务
云数据库 MongoDB
腾讯云数据库 MongoDB（TencentDB for MongoDB）是腾讯云基于全球广受欢迎的 MongoDB 打造的高性能 NoSQL 数据库，100%完全兼容 MongoDB 协议，支持跨文档事务，提供稳定丰富的监控管理，弹性可扩展、自动容灾，适用于文档型数据库场景，您无需自建灾备体系及控制管理系统。