MHA 手动故障转移

        MHA提供了3种方式用于实现故障转移，分别自动故障转移，需要启用MHA监控；在无监控的情况下的手动故障转移以及基于在线手动切换。三种方式可以应对MySQL主从故障的任意场景。本文主要描述在无监控的情形是手动实现故障转移。供大家参考。

      有关MHA的其他两种切换方式，可以参考： MHA 在线切换过程 MHA 自动故障转移步骤及过程剖析

1、手动故障转移的特点     a、在监控节点未启用masterha_manager     b、master库已经宕机或者转移到高性能服务器     c、手动故障转移支持交互或非交互两种模式     d、切换样例：$ masterha_master_switch --master_state=dead --conf=/etc/app1.cnf --dead_master_host=host1

2、masterha_master_switch切换的几个参数 --master_state=dead       强制参数为"dead" 或者 "alive". dead为手动故障转移，alive为在线切换。 --dead_master_host=(hostname)       强制参数为主机名，另2个--dead_master_ip --dead_master_port(缺省3306)可选。 --new_master_host=(hostname)       可选参数，用于指定新master，如果未指定则按candidate_master参数设定值。 --interactive=(0|1)       可选参数，指定是否交互。缺省为1，表明交互。

3、演示手动故障转移 ###环境   [root@vdbsrv1 ~]# more /etc/hosts 127.0.0.1    localhost.localdomain localhost 192.168.1.6  vdbsrv1  #master 192.168.1.7  vdbsrv2  #slave1 192.168.1.8  vdbsrv3  #slave2 192.168.1.12 vdbsrv4  #manager

###master开启VIP [root@vdbsrv4 ~]# ssh vdbsrv1 "/sbin/ifconfig eth0:0 192.168.1.13 netmask 255.255.255.0 up"

###检测MHA监控是否运行，如果已启动，可以使用masterha_check_stop先停止 [root@vdbsrv4 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING).

###模拟从库滞后 [root@vdbsrv4 ~]# ssh vdbsrv2 "mysql -e 'stop slave io_thread'" [root@vdbsrv4 ~]# ssh vdbsrv2 "mysql -e 'show slave status\G' | egrep 'Slave_IO|Slave_SQL'"                Slave_IO_State:              Slave_IO_Running: No             Slave_SQL_Running: Yes       Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it

###主库上插入新记录 mysql> insert into tb(val,dt) values('john',current_date()); Query OK, 1 row affected (0.01 sec)

###模拟master异常宕机 [root@vdbsrv4 ~]# ssh vdbsrv1 "killall -r mysqld"

###开始手工故障转移 [root@vdbsrv4 ~]# masterha_master_switch --master_state=dead --conf=/etc/masterha/app1.cnf --dead_master_host=vdbsrv1 \ >  --dead_master_port=3306 --new_master_host=vdbsrv3 --new_master_port=3306 --ignore_last_failover --dead_master_ip=<dead_master_ip> is not set. Using 192.168.1.6. Wed Apr 22 11:48:39 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Wed Apr 22 11:48:39 2015 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Wed Apr 22 11:48:39 2015 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Wed Apr 22 11:48:39 2015 - [info] MHA::MasterFailover version 0.56. Wed Apr 22 11:48:39 2015 - [info] Starting master failover. Wed Apr 22 11:48:39 2015 - [info] Wed Apr 22 11:48:39 2015 - [info] * Phase 1: Configuration Check Phase.. Wed Apr 22 11:48:39 2015 - [info] Wed Apr 22 11:48:39 2015 - [info] GTID failover mode = 0 Wed Apr 22 11:48:39 2015 - [info] Dead Servers: Wed Apr 22 11:48:39 2015 - [info]   vdbsrv1(192.168.1.6:3306) Wed Apr 22 11:48:39 2015 - [info] Checking master reachability via MySQL(double check)... Wed Apr 22 11:48:39 2015 - [info]  ok. Wed Apr 22 11:48:39 2015 - [info] Alive Servers: Wed Apr 22 11:48:39 2015 - [info]   vdbsrv2(192.168.1.7:3306) Wed Apr 22 11:48:39 2015 - [info]   vdbsrv3(192.168.1.8:3306) Wed Apr 22 11:48:39 2015 - [info] Alive Slaves: Wed Apr 22 11:48:39 2015 - [info]   vdbsrv2(192.168.1.7:3306)  Version=5.6.22-log (oldest major version between slaves) log-bin:enabled Wed Apr 22 11:48:39 2015 - [info]     Replicating from 192.168.1.6(192.168.1.6:3306) Wed Apr 22 11:48:39 2015 - [info]   vdbsrv3(192.168.1.8:3306)  Version=5.6.22-log (oldest major version between slaves) log-bin:enabled Wed Apr 22 11:48:39 2015 - [info]     Replicating from 192.168.1.6(192.168.1.6:3306) Master vdbsrv1(192.168.1.6:3306) is dead. Proceed? (yes/NO): yes Wed Apr 22 11:48:54 2015 - [info] Starting Non-GTID based failover. Wed Apr 22 11:48:54 2015 - [info] Wed Apr 22 11:48:54 2015 - [info] ** Phase 1: Configuration Check Phase completed. Wed Apr 22 11:48:54 2015 - [info] Wed Apr 22 11:48:54 2015 - [info] * Phase 2: Dead Master Shutdown Phase.. Wed Apr 22 11:48:54 2015 - [info] Wed Apr 22 11:48:54 2015 - [info] HealthCheck: SSH to vdbsrv1 is reachable. Wed Apr 22 11:48:55 2015 - [info] Forcing shutdown so that applications never connect to the current master.. Wed Apr 22 11:48:55 2015 - [info] Executing master IP deactivation script: Wed Apr 22 11:48:55 2015 - [info]   /tmp/master_ip_failover --orig_master_host=vdbsrv1 --orig_master_ip=192.168.1.6   --orig_master_port=3306 --command=stopssh --ssh_user=root 

IN SCRIPT TEST====/sbin/ifconfig eth0:0 down==/sbin/ifconfig eth0:0 192.168.1.13/24===

Disabling the VIP on old master: vdbsrv1 Wed Apr 22 11:48:55 2015 - [info]  done. Wed Apr 22 11:48:55 2015 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. Wed Apr 22 11:48:55 2015 - [info] * Phase 2: Dead Master Shutdown Phase completed. Wed Apr 22 11:48:55 2015 - [info] Wed Apr 22 11:48:55 2015 - [info] * Phase 3: Master Recovery Phase.. Wed Apr 22 11:48:55 2015 - [info] Wed Apr 22 11:48:55 2015 - [info] * Phase 3.1: Getting Latest Slaves Phase.. Wed Apr 22 11:48:55 2015 - [info] Wed Apr 22 11:48:55 2015 - [info] The latest binary log file/position on all slaves is mysql-bin.000027:432 Wed Apr 22 11:48:55 2015 - [info] Latest slaves (Slaves that received relay log files to the latest): Wed Apr 22 11:48:55 2015 - [info]   vdbsrv3(192.168.1.8:3306)  Version=5.6.22-log (oldest major version between slaves) log-bin:enabled Wed Apr 22 11:48:55 2015 - [info]     Replicating from 192.168.1.6(192.168.1.6:3306) Wed Apr 22 11:48:55 2015 - [info] The oldest binary log file/position on all slaves is mysql-bin.000027:120 Wed Apr 22 11:48:55 2015 - [info] Oldest slaves: Wed Apr 22 11:48:55 2015 - [info]   vdbsrv2(192.168.1.7:3306)  Version=5.6.22-log (oldest major version between slaves) log-bin:enabled Wed Apr 22 11:48:55 2015 - [info]     Replicating from 192.168.1.6(192.168.1.6:3306) Wed Apr 22 11:48:55 2015 - [info] Wed Apr 22 11:48:55 2015 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase.. Wed Apr 22 11:48:55 2015 - [info] Wed Apr 22 11:48:55 2015 - [info] Fetching dead master's binary logs.. Wed Apr 22 11:48:55 2015 - [info] Executing command on the dead master vdbsrv1(192.168.1.6:3306): save_binary_logs --command=save   --start_file=mysql-bin.000027  --start_pos=432 --binlog_dir=/data/mysqldata   --output_file=/var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog --handle_raw_binlog=1 --disable_log_bin=0   --manager_version=0.56   Creating /var/tmp if not exists..    ok.  Concat binary/relay logs from mysql-bin.000027 pos 432 to mysql-bin.000027 EOF into    /var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog ..  Binlog Checksum enabled   Dumping binlog format description event, from position 0 to 120.. ok.   Dumping effective binlog data from /data/mysqldata/mysql-bin.000027 position 432 to tail(455).. ok.  Binlog Checksum enabled  Concat succeeded. saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog                100%  143     0.1KB/s   00:00    Wed Apr 22 11:48:56 2015 - [info] scp from root@192.168.1.6:/var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog to   local:/var/log/masterha/app1/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog succeeded. Wed Apr 22 11:48:56 2015 - [info] HealthCheck: SSH to vdbsrv2 is reachable. Wed Apr 22 11:48:56 2015 - [info] HealthCheck: SSH to vdbsrv3 is reachable. Wed Apr 22 11:48:56 2015 - [info] Wed Apr 22 11:48:56 2015 - [info] * Phase 3.3: Determining New Master Phase.. Wed Apr 22 11:48:56 2015 - [info] Wed Apr 22 11:48:56 2015 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. Wed Apr 22 11:48:56 2015 - [info] Checking whether vdbsrv3 has relay logs from the oldest position.. Wed Apr 22 11:48:56 2015 - [info] Executing command: apply_diff_relay_logs --command=find --latest_mlf=mysql-bin.000027 --latest_rmlp=432    --target_mlf=mysql-bin.000027 --target_rmlp=120 --server_id=1002 --workdir=/var/tmp --timestamp=20150422114839 --manager_version=0.56    --relay_log_info=/data/mysqldata/relay-log.info  --relay_dir=/data/mysqldata/  :     Opening /data/mysqldata/relay-log.info ... ok.     Relay log found at /data/mysqldata, up to vdbsrv3-relay-bin.000002  Fast relay log position search succeeded.  Target relay log file/position found. start_file:vdbsrv3-relay-bin.000002, start_pos:283. Target relay log FOUND! Wed Apr 22 11:48:56 2015 - [info] OK. vdbsrv3 has all relay logs. Wed Apr 22 11:48:56 2015 - [info] vdbsrv3 can be new master. Wed Apr 22 11:48:56 2015 - [info] New master is vdbsrv3(192.168.1.8:3306) Wed Apr 22 11:48:56 2015 - [info] Starting master failover.. Wed Apr 22 11:48:56 2015 - [info] From: vdbsrv1(192.168.1.6:3306) (current master)  +--vdbsrv2(192.168.1.7:3306)  +--vdbsrv3(192.168.1.8:3306)

To: vdbsrv3(192.168.1.8:3306) (new master)  +--vdbsrv2(192.168.1.7:3306)

Starting master switch from vdbsrv1(192.168.1.6:3306) to vdbsrv3(192.168.1.8:3306)? (yes/NO): yes Wed Apr 22 11:48:58 2015 - [info] New master decided manually is vdbsrv3(192.168.1.8:3306) Wed Apr 22 11:48:58 2015 - [info] Wed Apr 22 11:48:58 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase.. Wed Apr 22 11:48:58 2015 - [info] Wed Apr 22 11:48:58 2015 - [info]  This server has all relay logs. No need to generate diff files from the latest slave. Wed Apr 22 11:48:58 2015 - [info] Sending binlog..      saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog          100%  143     0.1KB/s   00:00    Wed Apr 22 11:48:58 2015 - [info] scp from local:/var/log/masterha/app1/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog to   root@vdbsrv3:/var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog succeeded. Wed Apr 22 11:48:58 2015 - [info] Wed Apr 22 11:48:58 2015 - [info] * Phase 3.4: Master Log Apply Phase.. Wed Apr 22 11:48:58 2015 - [info] Wed Apr 22 11:48:58 2015 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed. Wed Apr 22 11:48:58 2015 - [info] Starting recovery on vdbsrv3(192.168.1.8:3306).. Wed Apr 22 11:48:58 2015 - [info]  Generating diffs succeeded. Wed Apr 22 11:48:58 2015 - [info] Waiting until all relay logs are applied. Wed Apr 22 11:48:58 2015 - [info]  done. Wed Apr 22 11:48:58 2015 - [info] Getting slave status.. Wed Apr 22 11:48:58 2015 - [info] This slave(vdbsrv3)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000027:432).   No need to recover from Exec_Master_Log_Pos. Wed Apr 22 11:48:58 2015 - [info] Connecting to the target slave host vdbsrv3, running recover script.. Wed Apr 22 11:48:58 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=vdbsrv3    --slave_ip=192.168.1.8  --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog    --workdir=/var/tmp --target_version=5.6.22-log --timestamp=20150422114839 --handle_raw_binlog=1 --disable_log_bin=0    --manager_version=0.56 --slave_pass=xxx Wed Apr 22 11:48:59 2015 - [info] Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog on vdbsrv3:3306.   This may take long time... Applying log files succeeded. Wed Apr 22 11:48:59 2015 - [info]  All relay logs were successfully applied. Wed Apr 22 11:48:59 2015 - [info] Getting new master's binlog name and position.. Wed Apr 22 11:48:59 2015 - [info]  mysql-bin.000017:259187555 Wed Apr 22 11:48:59 2015 - [info]  All other slaves should start replication from here. Statement should be:    CHANGE MASTER TO MASTER_HOST='vdbsrv3 or 192.168.1.8', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000017',    MASTER_LOG_POS=259187555, MASTER_USER='repl', MASTER_PASSWORD='xxx'; Wed Apr 22 11:48:59 2015 - [info] Executing master IP activate script: Wed Apr 22 11:48:59 2015 - [info]   /tmp/master_ip_failover --command=start --ssh_user=root --orig_master_host=vdbsrv1   --orig_master_ip=192.168.1.6 --orig_master_port=3306 --new_master_host=vdbsrv3 --new_master_ip=192.168.1.8 --new_master_port=3306   --new_master_user='mha' --new_master_password='xxx'  Unknown option: new_master_user Unknown option: new_master_password

IN SCRIPT TEST====/sbin/ifconfig eth0:0 down==/sbin/ifconfig eth0:0 192.168.1.13/24===

Enabling the VIP - 192.168.1.13/24 on the new master - vdbsrv3 Wed Apr 22 11:48:59 2015 - [info]  OK. Wed Apr 22 11:48:59 2015 - [info] ** Finished master recovery successfully. Wed Apr 22 11:48:59 2015 - [info] * Phase 3: Master Recovery Phase completed. Wed Apr 22 11:48:59 2015 - [info] Wed Apr 22 11:48:59 2015 - [info] * Phase 4: Slaves Recovery Phase.. Wed Apr 22 11:48:59 2015 - [info] Wed Apr 22 11:48:59 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. Wed Apr 22 11:48:59 2015 - [info] Wed Apr 22 11:48:59 2015 - [info] -- Slave diff file generation on host vdbsrv2(192.168.1.7:3306) started, pid: 17143.    Check tmp log /var/log/masterha/app1/vdbsrv2_3306_20150422114839.log if it takes time.. Wed Apr 22 11:49:00 2015 - [info] Wed Apr 22 11:49:00 2015 - [info] Log messages from vdbsrv2 ... Wed Apr 22 11:49:00 2015 - [info] Wed Apr 22 11:48:59 2015 - [info] Server vdbsrv2 received relay logs up to: mysql-bin.000027:120 Wed Apr 22 11:48:59 2015 - [info] Need to get diffs from the latest slave(vdbsrv3) up to: mysql-bin.000027:432 (using the latest slave's relay logs) Wed Apr 22 11:48:59 2015 - [info] Connecting to the latest slave host vdbsrv3, generating diff relay log files.. Wed Apr 22 11:48:59 2015 - [info] Executing command: apply_diff_relay_logs --command=generate_and_send --scp_user=root    --scp_host=192.168.1.7 --latest_mlf=mysql-bin.000027 --latest_rmlp=432 --target_mlf=mysql-bin.000027 --target_rmlp=120    --server_id=1002 --diff_file_readtolatest=/var/tmp/relay_from_read_to_latest_vdbsrv2_3306_20150422114839.binlog --workdir=/var/tmp    --timestamp=20150422114839 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --relay_log_info=/data/mysqldata/relay-log.info    --relay_dir=/data/mysqldata/ Wed Apr 22 11:49:00 2015 - [info]     Opening /data/mysqldata/relay-log.info ... ok.     Relay log found at /data/mysqldata, up to vdbsrv3-relay-bin.000002  Fast relay log position search succeeded.  Target relay log file/position found. start_file:vdbsrv3-relay-bin.000002, start_pos:283.  Concat binary/relay logs from vdbsrv3-relay-bin.000002 pos 283 to vdbsrv3-relay-bin.000002 EOF into    /var/tmp/relay_from_read_to_latest_vdbsrv2_3306_20150422114839.binlog ..  Binlog Checksum enabled  Binlog Checksum enabled   Dumping binlog format description event, from position 0 to 283.. ok.   Dumping effective binlog data from /data/mysqldata/vdbsrv3-relay-bin.000002 position 283 to tail(595).. ok.  Binlog Checksum enabled  Binlog Checksum enabled  Concat succeeded.  Generating diff relay log succeeded. Saved at /var/tmp/relay_from_read_to_latest_vdbsrv2_3306_20150422114839.binlog .  scp vdbsrv3:/var/tmp/relay_from_read_to_latest_vdbsrv2_3306_20150422114839.binlog to root@192.168.1.7(22) succeeded. Wed Apr 22 11:49:00 2015 - [info]  Generating diff files succeeded. Wed Apr 22 11:49:00 2015 - [info] End of log messages from vdbsrv2. Wed Apr 22 11:49:00 2015 - [info] -- Slave diff log generation on host vdbsrv2(192.168.1.7:3306) succeeded. Wed Apr 22 11:49:00 2015 - [info] Generating relay diff files from the latest slave succeeded. Wed Apr 22 11:49:00 2015 - [info] Wed Apr 22 11:49:00 2015 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase.. Wed Apr 22 11:49:00 2015 - [info] Wed Apr 22 11:49:00 2015 - [info] -- Slave recovery on host vdbsrv2(192.168.1.7:3306) started, pid: 17149. Check tmp log   /var/log/masterha/app1/vdbsrv2_3306_20150422114839.log if it takes time.. saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog               100%  143     0.1KB/s   00:00    Wed Apr 22 11:49:00 2015 - [info] Wed Apr 22 11:49:00 2015 - [info] Log messages from vdbsrv2 ... Wed Apr 22 11:49:00 2015 - [info] Wed Apr 22 11:49:00 2015 - [info] Sending binlog.. Wed Apr 22 11:49:00 2015 - [info] scp from local:/var/log/masterha/app1/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog    to root@vdbsrv2:/var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog succeeded. Wed Apr 22 11:49:00 2015 - [info] Starting recovery on vdbsrv2(192.168.1.7:3306).. Wed Apr 22 11:49:00 2015 - [info]  Generating diffs succeeded. Wed Apr 22 11:49:00 2015 - [info] Waiting until all relay logs are applied. Wed Apr 22 11:49:00 2015 - [info]  done. Wed Apr 22 11:49:00 2015 - [info] Getting slave status.. Wed Apr 22 11:49:00 2015 - [info] This slave(vdbsrv2)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000027:120).   No need to recover from Exec_Master_Log_Pos. Wed Apr 22 11:49:00 2015 - [info] Connecting to the target slave host vdbsrv2, running recover script.. Wed Apr 22 11:49:00 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=vdbsrv2   --slave_ip=192.168.1.7  --slave_port=3306 --apply_files=/var/tmp/relay_from_read_to_latest_vdbsrv2_3306_20150422114839.binlog,   /var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog --workdir=/var/tmp --target_version=5.6.22-log   --timestamp=20150422114839 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx Wed Apr 22 11:49:00 2015 - [info]  Concat all apply files to /var/tmp/total_binlog_for_vdbsrv2_3306.20150422114839.binlog ..  Copying the first binlog file /var/tmp/relay_from_read_to_latest_vdbsrv2_3306_20150422114839.binlog to    /var/tmp/total_binlog_for_vdbsrv2_3306.20150422114839.binlog.. ok.   Dumping binlog head events (rotate events), skipping format description events from     /var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog..  Binlog Checksum enabled dumped up to pos 120. ok.  /var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog has effective binlog events from pos 120.   Dumping effective binlog data from /var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog position 120 to tail(143).. ok.  Concat succeeded. All apply target binary logs are concatinated at /var/tmp/total_binlog_for_vdbsrv2_3306.20150422114839.binlog . Applying differential binary/relay log files /var/tmp/relay_from_read_to_latest_vdbsrv2_3306_20150422114839.binlog,   /var/tmp/saved_master_binlog_from_vdbsrv1_3306_20150422114839.binlog on vdbsrv2:3306. This may take long time... Applying log files succeeded. Wed Apr 22 11:49:00 2015 - [info]  All relay logs were successfully applied. Wed Apr 22 11:49:00 2015 - [info]  Resetting slave vdbsrv2(192.168.1.7:3306) and starting replication from the new master vdbsrv3(192.168.1.8:3306).. Wed Apr 22 11:49:00 2015 - [info]  Executed CHANGE MASTER. Wed Apr 22 11:49:00 2015 - [info]  Slave started. Wed Apr 22 11:49:00 2015 - [info] End of log messages from vdbsrv2. Wed Apr 22 11:49:00 2015 - [info] -- Slave recovery on host vdbsrv2(192.168.1.7:3306) succeeded. Wed Apr 22 11:49:00 2015 - [info] All new slave servers recovered successfully. Wed Apr 22 11:49:00 2015 - [info] Wed Apr 22 11:49:00 2015 - [info] * Phase 5: New master cleanup phase.. Wed Apr 22 11:49:00 2015 - [info] Wed Apr 22 11:49:00 2015 - [info] Resetting slave info on the new master.. Wed Apr 22 11:49:00 2015 - [info]  vdbsrv3: Resetting slave info succeeded. Wed Apr 22 11:49:00 2015 - [info] Master failover to vdbsrv3(192.168.1.8:3306) completed successfully. Wed Apr 22 11:49:00 2015 - [info]

----- Failover Report -----

app1: MySQL Master failover vdbsrv1(192.168.1.6:3306) to vdbsrv3(192.168.1.8:3306) succeeded

Master vdbsrv1(192.168.1.6:3306) is down!

Check MHA Manager logs at vdbsrv4 for details.

Started manual(interactive) failover. Invalidated master IP address on vdbsrv1(192.168.1.6:3306) The latest slave vdbsrv3(192.168.1.8:3306) has all relay logs for recovery. Selected vdbsrv3(192.168.1.8:3306) as a new master. vdbsrv3(192.168.1.8:3306): OK: Applying all logs succeeded. vdbsrv3(192.168.1.8:3306): OK: Activated master IP address. vdbsrv2(192.168.1.7:3306): Generating differential relay logs up to vdbsrv3(192.168.1.8:3306)succeeded. Generating relay diff files from the latest slave succeeded. vdbsrv2(192.168.1.7:3306): OK: Applying all logs succeeded. Slave started, replicating from vdbsrv3(192.168.1.8:3306) vdbsrv3(192.168.1.8:3306): Resetting slave info succeeded. Master failover to vdbsrv3(192.168.1.8:3306) completed successfully.