; 105-MHA | Linux运维部落

105-MHA

一.MHA简介

1.1.简介

        MHA(Master HA ) 是一款开源的 MySQL高可用程序,它为MYSQL 主从复制架构提供了automating master failover

主节点自动迁移功能.MHA在监控到master节点故障时候,会自动提升拥有的数据最近进于主节点的其他从节点为主节点,

并且在此期间,会通过其他节点获取额外的信息来避免数据不一致性的问题.MHA也提供master节点的在线切换功能,

即按需切换master/slave节点.

1.2.MHA服务角色

        MHA Manager (管理节点): 通用单独部署一台机器专门用于管理一个或者多个master/slaver集群,每个master/slave
集群成为一个application;
        MHA  node(数据节点) : 运行在每台mysql 服务器上(master或slave或manager),它通过监控具备解析和清理logs

功能的脚本来实现以及加速故障转移

        MHA结构示意图:
                                

                              
 

1.3.MHA故障转移步骤

         当Mysql 集群中的master 节点发生故障时,MHA会自动将其他slave节点按一下步骤提升为master,并自动转移和补齐缺失数据.
          master切换/故障转移示意图:

             1.当master故障时候,会查找其他slave节点中,数据最接近于master的latest-slave节点,提升为master
             2.查找latest-slave中缺失的数据部分,从其他slave中获取,以还原为dead-master原拥有的完整数据. 

1.4 MHA组件

 1.4.1 Manager节点工具程序:

            – masterha_check_ssh: MHA依赖的SSH环境检测工具(检测节点之间的SSH通信状况)
            – masterha_check_repl: Mysql 复制环境检测工具;
            – masterha_manager: MHA服务主程序;
            – masterha_check_status: MHA运行状态探测工具;
            – masterha_master_monitor: Mysql master 节点可用性检测工具;
            – masterha _master_switch: master节点切换工具;
            – masterha_conf_host: 添加或删除配置的节点;
            – masterha_stop; 关闭MHA服务的工具;

1.4.2 Node节点工具程序:

            – save_binary_logs: 用于保存和复制master的二进制日志专用工具;
            – apply_diff_relay_logs: 用于识别差异的中继日志时间,并应用于其他slave节点;
            – filter_mysqlbinlog: 去除不必要的roolback事件(MHA已经不再使用此工具)
            – purge_relay_logs: 清楚中继日志(不会阻塞SQL线程);

1.4.3 其定义拓展:

            – secondary_check_script:  通过多条网络路由检测master的可用性;
            – shutdown_script: 强制关闭master节点;
            – master_ip_failover_script:  更新application 使用的masterip;
            – report_script : 用于发送报告;
            – init_conf_load_script: 用于加载初始化配置参数;
            – master_ip_online_change_script: 更新master节点Ip地址;

二.MHA环境要求

2.1 Mysql replication 环境准备

2.1.1 MHA mysql 复制环境要求:

            1.各节点需要开启二进制日志以及中继日志;
                   log-bin=master-bin
                    relay-log=relay-bin
            2.全部从节点需要开启只读模式,并关闭relay_log_purge
                    read-only=1
                    relay_log_purge=0
            3. 各节点ID在集群中必须唯一,不可以冲突
                    server_id=#
            4. MHA需要基于mysql 主从复制模式下工作,因此启用MHA之前,需要确保各 mysql 主从复制节点工作正常
            5. 确保从节点上slave IO,SQL线程工作正常;
            6. 所有mysql节点需要授权拥有管理权限的用户,以供任意节点间的互相访问
                mysql> GRANT ALL ON *.* TO ‘mhaadmin’@’IP_ADDR’ IDENTIFIED BY ‘mhappss’;
            7. 在manager-node 节点 上创建公钥私钥,并将其复制到全部节点上,以确保节点之间可以基于相同的key通信

2.1.2 MHA初始化以及配置文件

            Manager 节点需要为每个受监控的master/slave集群提供一个专用的配置文件,而所有的master/slave集群,也
            可以共享一个全局的配置.全局配置文件默认为/etc/masterha_default.cnf(此为可选项).
            若只监控一组master/slave集群,可以直接通过application 的配置来提供各服务器的默认配置信息,同时,每个
            application的配置文件路径为自定.
            

三. MHA 实例

3.1.环境说明:

                        

                 mastet-node1 : 10.1.249.184
                 slave-node2 : 10.1.252.218
                 slave-node4 :  10.1.249.70
                manager-node:  10.1.249.83
 

3.2.mysql 主从复制环境创建

3.2.1 master-node1配置:

        vim /etc/my.cnf

  1. 1
    <span class="pun">[</span><span class="pln">mysqld</span><span class="pun">]</span>
  2. 1
    <span class="pln">datadir</span><span class="pun">=/</span><span class="pln">var</span><span class="pun">/</span><span class="pln">lib</span><span class="pun">/</span><span class="pln">mysql</span>
  3. 1
    <span class="pln">socket</span><span class="pun">=/</span><span class="pln">var</span><span class="pun">/</span><span class="pln">lib</span><span class="pun">/</span><span class="pln">mysql</span><span class="pun">/</span><span class="pln">mysql</span><span class="pun">.</span><span class="pln">sock</span>
  4. 1
    <span class="com"># Disabling symbolic-links is recommended to prevent assorted security risks</span>
  5. 1
    <span class="pln">symbolic</span><span class="pun">-</span><span class="pln">links</span><span class="pun">=</span><span class="lit">0</span>
  6. 1
    <span class="com"># Settings user and group are ignored when systemd is used.</span>
  7. 1
    <span class="com"># If you need to run mysqld under a different user or group,</span>
  8. 1
    <span class="com"># customize your systemd unit file for mariadb according to the</span>
  9. 1
    <span class="com"># instructions in http://fedoraproject.org/wiki/Systemd</span>
  10. 1
    <b><span class="pln"><span>skip_name_resolve</span></span><span class="pun"><span>=</span></span><span class="pln"><span>ON</span></span></b>
  11. 1
    <b><span class="pln"><span>innodb_file_per_table</span></span><span class="pun"><span>=</span></span><span class="pln"><span>ON</span></span></b>
  12. 1
     
  13. 1
    <b><span class="pln"><span>server_id</span></span><span class="pun"><span>=</span></span><span class="lit"><span>1</span></span></b>
  14. 1
    <b><span class="pln"><span>relay</span></span><span class="pun"><span>-</span></span><span class="pln"><span>log</span></span><span class="pun"><span>=</span></span><span class="pln"><span>relay</span></span><span class="pun"><span>-</span></span><span class="pln"><span>bin</span></span></b>
  15. 1
    <b><span class="pln"><span>log</span></span><span class="pun"><span>-</span></span><span class="pln"><span>bin</span></span><span class="pun"><span>=</span></span><span class="pln"><span>master</span></span><span class="pun"><span>-</span></span><span class="pln"><span>bin</span></span></b>
             启动mariadb并进入mysql环境,
             查询binlog文件名以及binlog position
  1. 1
    <span class="typ">MariaDB</span><span class="pln"> </span><span class="pun">[(</span><span class="pln">none</span><span class="pun">)]&gt;</span><span class="pln"> GRANT ALL ON </span><span class="pun">*.*</span><span class="pln"> TO </span><span class="str">'mhaadmin'</span><span class="pun">@</span><span class="str">'10.1.%.%'</span><span class="pln"> IDENTIFIED BY </span><span class="str">'000000'</span><span class="pun">;</span>
  2. 1
    2
    <span class="pun">MariaDB [(none)]&gt; GRANT REPLICATION CLIENT,REPLICATION SLAVE ON *.* TO '<span>repluser</span>'@'10.1.%.%' IDENTIFIED BY '000000';
    </span>
  3. MariaDB [(none)]> SHOW MASTER STATUS;
  4. 1
    2
    3
    4
    5
    6
    7
    <span class="pun">+-------------------+----------+--------------+------------------+
    | File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
    +-------------------+----------+--------------+------------------+
    | <span>master-bin.000003</span> |     <span> 465</span> |              |                  |
    +-------------------+----------+--------------+------------------+
    1 row in set (0.00 sec)
    </span>
  5. 1
     

3.2.2 slave-node 配置:

        vim /etc/my.cnf

  1. 1
    <span class="pun">[</span><span class="pln">mysqld</span><span class="pun">]</span>
  2. 1
    <span class="pln">datadir</span><span class="pun">=/</span><span class="pln">var</span><span class="pun">/</span><span class="pln">lib</span><span class="pun">/</span><span class="pln">mysql</span>
  3. 1
    <span class="pln">socket</span><span class="pun">=/</span><span class="pln">var</span><span class="pun">/</span><span class="pln">lib</span><span class="pun">/</span><span class="pln">mysql</span><span class="pun">/</span><span class="pln">mysql</span><span class="pun">.</span><span class="pln">sock</span>
  4. 1
    <span class="com"># Disabling symbolic-links is recommended to prevent assorted security risks</span>
  5. 1
    <span class="pln">symbolic</span><span class="pun">-</span><span class="pln">links</span><span class="pun">=</span><span class="lit">0</span>
  6. 1
    <span class="com"># Settings user and group are ignored when systemd is used.</span>
  7. 1
    <span class="com"># If you need to run mysqld under a different user or group,</span>
  8. 1
    <span class="com"># customize your systemd unit file for mariadb according to the</span>
  9. 1
    <span class="com"># instructions in http://fedoraproject.org/wiki/Systemd</span>
  10. 1
     
  11. 1
    <span class="pln"><span>skip_name_resolve</span></span><span class="pun"><span>=</span></span><span class="pln"><span>ON</span></span>
  12. 1
    <span class="pln"><span>innodb_file_per_table</span></span><span class="pun"><span>=</span></span><span class="pln"><span>ON</span></span>
  13. 1
     
  14. 1
    <span class="pln"><span>server_id</span></span><span class="pun"><span>=</span></span><span class="lit"><span>2</span></span>
  15. 1
    <span class="pln"><span>relay</span></span><span class="pun"><span>-</span></span><span class="pln"><span>log</span></span><span class="pun"><span>=</span></span><span class="pln"><span>relay</span></span><span class="pun"><span>-</span></span><span class="pln"><span>bin</span></span>
  16. 1
    <span class="pln"><span>log</span></span><span class="pun"><span>-</span></span><span class="pln"><span>bin</span></span><span class="pun"><span>=</span></span><span class="pln"><span>master</span></span><span class="pun"><span>-</span></span><span class="pln"><span>bin</span></span>
  17. 1
    <span class="pln"><span>relay_log_purge</span></span><span class="pun"><span>=</span></span><span class="lit"><span>0</span></span>
  18. 1
    <span class="pln"><span>read_only</span></span><span class="pun"><span>=</span></span><span class="lit"><span>1</span></span>
            启动mariadb并进入mysql环境:
  1. 1
    <span class="typ">MariaDB</span><span class="pln"> </span><span class="pun">[(</span><span class="pln">none</span><span class="pun">)]&gt;</span><span class="pln"> CHANGE MASTER TO MASTER_HOST</span><span class="pun">=</span><span class="str">'10.1.249.184'</span><span class="pun">,</span><span class="pln">MASTER_USER</span><span class="pun">=</span><span class="str">'repluser'</span><span class="pun">,</span><span class="pln">MASTER_PASSWORD</span><span class="pun">=</span><span class="str">'000000'</span><span class="pun">,</span><span class="pln">MASTER_LOG_FILE</span><span class="pun">=</span><span class="str">'master-bin.000003'</span><span class="pun">,</span><span class="pln">MASTER_LOG_POS</span><span class="pun">=</span><span class="lit">465</span><span class="pun">;</span>
  2. 1
    2
    <span class="pun"><span class="typ"><span>MariaDB</span></span><span class="pln"><span> </span></span><span class="pun"><span>[(</span></span><span class="pln"><span>none</span></span><span class="pun"><span>)]&gt; </span></span><span>GRANT ALL ON *.* TO 'mhaadmin'@'10.1.%.%' IDENTIFIED BY '000000';</span><font color="#bebec5">
    </font></span>
  3. MariaDB [(none)]> START SLAVE;
  4. MariaDB [(none)]> SHOW SLAVE STATUS\G;
    *************************** 1. row ***************************
    Slave_IO_State: Waiting for master to send event
    Master_Host: 10.1.249.184
    Master_User: mhaadmin
    Master_Port: 3306
    Connect_Retry: 60
    Master_Log_File: master-bin.000003
    Read_Master_Log_Pos: 633
    Relay_Log_File: relay-bin.000002
    Relay_Log_Pos: 698
    Relay_Master_Log_File: master-bin.000003
    Slave_IO_Running: Yes
    Slave_SQL_Running: Yes
    Replicate_Do_DB:
    Replicate_Ignore_DB:
    Replicate_Do_Table:
    Replicate_Ignore_Table:
    Replicate_Wild_Do_Table:
    Replicate_Wild_Ignore_Table:
    Last_Errno: 0
    Last_Error:
    Skip_Counter: 0
    Exec_Master_Log_Pos: 633
    Relay_Log_Space: 986
    Until_Condition: None
    Until_Log_File:
    Until_Log_Pos: 0
    Master_SSL_Allowed: No
    Master_SSL_CA_File:
    Master_SSL_CA_Path:
    Master_SSL_Cert:
    Master_SSL_Cipher:
    Master_SSL_Key:
    Seconds_Behind_Master: 0
    Master_SSL_Verify_Server_Cert: No
    Last_IO_Errno: 0
    Last_IO_Error:
    Last_SQL_Errno: 0
    Last_SQL_Error:
    Replicate_Ignore_Server_Ids:
    Master_Server_Id: 1
    1 row in set (0.00 sec)


3.2.3 主从复制测试:

            在master-node1创建任意数据库,测试同步状态;
  1. 1
    <span class="typ">MariaDB</span><span class="pln"> </span><span class="pun">[(</span><span class="pln">none</span><span class="pun">)]&gt;</span><span class="pln"> SHOW DATABASES</span><span class="pun">;</span>
  2. 1
    <span class="pun">+--------------------+</span>
  3. 1
    <span class="pun">|</span><span class="pln"> </span><span class="typ">Database</span><span class="pln">           </span><span class="pun">|</span>
  4. 1
    <span class="pun">+--------------------+</span>
  5. 1
    <span class="pun">|</span><span class="pln"> information_schema </span><span class="pun">|</span>
  6. 1
    <span class="pun">|</span><span class="pln"> mysql              </span><span class="pun">|</span>
  7. 1
    <span class="pun">|</span><span class="pln"> performance_schema </span><span class="pun">|</span>
  8. 1
    <span class="pun">|</span><span class="pln"><span> <span>test</span> </span>              </span><span class="pun">|</span>
  9. 1
    <span class="pun">|</span><span class="pln"> <span>test2</span>              </span><span class="pun">|</span>
  10. 1
    <span class="pun">+--------------------+</span>
  11. 1
    <span class="lit">5</span><span class="pln"> rows </span><span class="kwd">in</span><span class="pln"> </span><span class="kwd">set</span><span class="pln"> </span><span class="pun">(</span><span class="lit">0.00</span><span class="pln"> sec</span><span class="pun">)</span>
  12. 1
     
  13. 1
    <span class="typ">MariaDB</span><span class="pln"> </span><span class="pun">[(</span><span class="pln">none</span><span class="pun">)]&gt;</span><span class="pln"> </span>
            在任意从节点查看同步情况,确保同步成功;

3.2.4 manager-node 配置:

            1. 创建ssh通信公钥:
  1. 1
    <span class="pun"><span>#创建私钥</span></span>
  2. [root@node5 .ssh]# ssh-keygen -t rsa -P ''
  3. Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 7f:d0:79:0b:ad:30:fa:0e:61:0d:4c:13:dc:60:1f:20 root@node5 The key's randomart image is: +--[ RSA 2048]----+ | E.B=. | | =.o.. | | o . | | o . o | | S = + o | | . + + + . | | o . o . | | o . | | .o | +-----------------+
  4. #添加公钥到认证信息存放文件中
  5. [root@node5 .ssh]# cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys 

  6. 1
    <span class="str">#修改认证文件权限</span>
  7. 1
    <span class="str"><span>[root@node5 .ssh]# chmod go= /root/.ssh/authorized_keys</span></span>
  8. 1
    <span class="str"><span><br></span></span>
  9. 1
    2
    <span class="str"><span>#复制认证文件等到其他各节点 </span><font color="#bebec5">
    </font></span>
  10. [root@node5 ~]# scp /root/.ssh/authorized_keys id_rsa 10.1.249.70:/root/.ssh/
  11. [root@node5 ~]# scp /root/.ssh/authorized_keys id_rsa 10.1.252.218:/root/.ssh/
  12. [root@node5 ~]# scp /root/.ssh/authorized_keys id_rsa 10.1.249.184:/root/.ssh/
  13. #复制完以后需要逐个登录,测试manager-node可否使用ssh登录其他全部节点
  14. 注意:最好先清除全部节点/root/.ssh/下的文件再复制认证文件


                2. 安装MHA

                    MHA官方有提供rpm格式安装包,可以自行前往下载;CentOS7 中可以直接使用el6的程序包,
                    此外,MHA manager 和 MHA node 程序包版本不强制要求保持一致;

                    #官方源中无rpm,需要自行下载rpm包安装

                    

                   manager-node :
                                    yum install ./mha4mysql-manager-0.56-0.el6.noarch.rpm mha4mysql-node-0.56-                                                                                         0.el6.noarch.rpm -y   

                                注意: mha4mysql-manager与mha4mysql-node两个包需要同时安装,否则会安装失败

                    其他所有节点:

                                    yum install ./mha4mysql-node-0.56-0.el6.noarch.rpm
    

3.2.5 初始化MHA

                1.创建配置文件,(此处使用application为集群提供默认配置,路径为/etc/masterha/app1.cnf)

                   #需要手动创建/etc/masterh/app1目录以及/data/masterha/app1/
  1. 1
    <span class="pun"></span>
  2. 1
    <span class="pun"><span>vim /etc/masterha/app1.cnf</span></span>
  3. 1
    <span class="pun">[</span><span class="pln">server default</span><span class="pun">]</span>
  4. 1
    <span class="pln">user</span><span class="pun">=</span><span class="pln">mhaadmin</span>
  5. 1
    <span class="pln">password</span><span class="pun">=</span><span class="lit">000000</span>
  6. 1
    <span class="pln">manager_workdir</span><span class="pun">=/</span><span class="pln">data</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span>
  7. 1
    <span class="pln">manager_log</span><span class="pun">=/</span><span class="pln">data</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span><span class="pun">/</span><span class="pln">manager</span><span class="pun">.</span><span class="pln">log</span>
  8. 1
    <span class="pln">remote_workdir</span><span class="pun">=/</span><span class="pln">data</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span>
  9. 1
    <span class="pln">ssh_user</span><span class="pun">=</span><span class="pln">root</span>
  10. 1
    <span class="pln">repl_user</span><span class="pun">=</span><span class="pln">repluser</span>
  11. 1
    <span class="pln">repl_password</span><span class="pun">=</span><span class="pln">replpass</span>
  12. 1
    <span class="pln">ping_interval</span><span class="pun">=</span><span class="lit">1</span>
  13. 1
     
  14. 1
    <span class="pun">[</span><span class="pln">server1</span><span class="pun">]</span>
  15. 1
    <span class="pln">hostname</span><span class="pun">=</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.184</span>
  16. 1
    <span class="pln">candidate_master</span><span class="pun">=</span><span class="lit">1</span>
  17. 1
    <span class="com">#ssh_port=22022</span>
  18. 1
     
  19. 1
    <span class="pun">[</span><span class="pln">server2</span><span class="pun">]</span>
  20. 1
    <span class="pln">hostname</span><span class="pun">=</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">252.218</span>
  21. 1
    <span class="pln">candidate_master</span><span class="pun">=</span><span class="lit">1</span>
  22. 1
    <span class="com">#ssh_port=22022</span>
  23. 1
     
  24. 1
    <span class="pun">[</span><span class="pln">server3</span><span class="pun">]</span>
  25. 1
    <span class="pln">hostname</span><span class="pun">=</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.70</span>
  26. 1
    <span class="com">#ssh_port=22022</span>
  27. 1
    <span class="com">#no_master=1</span>
  28. 1
     

               2. 修改全部节点的/etc/hosts文件:

  1. 1
    <span class="lit">127.0</span><span class="pun">.</span><span class="lit">0.1</span><span class="pln">   localhost localhost</span><span class="pun">.</span><span class="pln">localdomain localhost4 localhost4</span><span class="pun">.</span><span class="pln">localdomain4</span>
  2. 1
    <span class="pun">::</span><span class="lit">1</span><span class="pln">         localhost localhost</span><span class="pun">.</span><span class="pln">localdomain localhost6 localhost6</span><span class="pun">.</span><span class="pln">localdomain6</span>
  3. 1
    <span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.83</span><span class="pln">  node5</span><span class="pun">.</span><span class="pln">com node5</span>
  4. 1
    <span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.184</span><span class="pln"> node1</span><span class="pun">.</span><span class="pln">com node1</span>
  5. 1
    <span class="lit">10.1</span><span class="pun">.</span><span class="lit">252.218</span><span class="pln"> node2</span><span class="pun">.</span><span class="pln">com node2</span>
  6. 1
    <span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.70</span><span class="pln">  node4</span><span class="pun">.</span><span class="pln">com node4</span>
             

               3.测试各节点间ssh通信配置:

  1. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> masterha_check_ssh </span><span class="pun">--</span><span class="pln">conf</span><span class="pun">=/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span><span class="pun">.</span><span class="pln">cnf</span>
  2. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">20</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">warning</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Global</span><span class="pln"> configuration file </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">masterha_default</span><span class="pun">.</span><span class="pln">cnf not found</span><span class="pun">.</span><span class="pln"> </span><span class="typ">Skipping</span><span class="pun">.</span>
  3. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">20</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Reading</span><span class="pln"> application default configuration from </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span><span class="pun">.</span><span class="pln">cnf</span><span class="pun">..</span>
  4. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">20</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Reading</span><span class="pln"> server configuration from </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span><span class="pun">.</span><span class="pln">cnf</span><span class="pun">..</span>
  5. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">20</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Starting</span><span class="pln"> SSH connection tests</span><span class="pun">..</span>
  6. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">21</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln"> </span>
  7. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">20</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">  </span><span class="typ">Connecting</span><span class="pln"> via SSH from root@10</span><span class="pun">.</span><span class="lit">1.249</span><span class="pun">.</span><span class="lit">184</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.184</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)</span><span class="pln"> to root@10</span><span class="pun">.</span><span class="lit">1.252</span><span class="pun">.</span><span class="lit">218</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">252.218</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)..</span>
  8. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">21</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">   ok</span><span class="pun">.</span>
  9. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">21</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">  </span><span class="typ">Connecting</span><span class="pln"> via SSH from root@10</span><span class="pun">.</span><span class="lit">1.249</span><span class="pun">.</span><span class="lit">184</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.184</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)</span><span class="pln"> to root@10</span><span class="pun">.</span><span class="lit">1.249</span><span class="pun">.</span><span class="lit">70</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.70</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)..</span>
  10. 1
    <span class="typ">Warning</span><span class="pun">:</span><span class="pln"> </span><span class="typ">Permanently</span><span class="pln"> added </span><span class="str">'10.1.249.70'</span><span class="pln"> </span><span class="pun">(</span><span class="pln">ECDSA</span><span class="pun">)</span><span class="pln"> to the list of known hosts</span><span class="pun">.</span>
  11. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">21</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">   ok</span><span class="pun">.</span>
  12. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">22</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln"> </span>
  13. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">21</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">  </span><span class="typ">Connecting</span><span class="pln"> via SSH from root@10</span><span class="pun">.</span><span class="lit">1.252</span><span class="pun">.</span><span class="lit">218</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">252.218</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)</span><span class="pln"> to root@10</span><span class="pun">.</span><span class="lit">1.249</span><span class="pun">.</span><span class="lit">184</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.184</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)..</span>
  14. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">21</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">   ok</span><span class="pun">.</span>
  15. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">21</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">  </span><span class="typ">Connecting</span><span class="pln"> via SSH from root@10</span><span class="pun">.</span><span class="lit">1.252</span><span class="pun">.</span><span class="lit">218</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">252.218</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)</span><span class="pln"> to root@10</span><span class="pun">.</span><span class="lit">1.249</span><span class="pun">.</span><span class="lit">70</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.70</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)..</span>
  16. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">22</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">   ok</span><span class="pun">.</span>
  17. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">22</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln"> </span>
  18. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">21</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">  </span><span class="typ">Connecting</span><span class="pln"> via SSH from root@10</span><span class="pun">.</span><span class="lit">1.249</span><span class="pun">.</span><span class="lit">70</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.70</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)</span><span class="pln"> to root@10</span><span class="pun">.</span><span class="lit">1.249</span><span class="pun">.</span><span class="lit">184</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.184</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)..</span>
  19. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">22</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">   ok</span><span class="pun">.</span>
  20. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">22</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">  </span><span class="typ">Connecting</span><span class="pln"> via SSH from root@10</span><span class="pun">.</span><span class="lit">1.249</span><span class="pun">.</span><span class="lit">70</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.70</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)</span><span class="pln"> to root@10</span><span class="pun">.</span><span class="lit">1.252</span><span class="pun">.</span><span class="lit">218</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">252.218</span><span class="pun">:</span><span class="lit">22</span><span class="pun">)..</span>
  21. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">22</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">debug</span><span class="pun">]</span><span class="pln">   ok</span><span class="pun">.</span>
  22. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">17</span><span class="pun">:</span><span class="lit">24</span><span class="pun">:</span><span class="lit">22</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"><span> </span></span><span class="typ"><span>All</span></span><span class="pln"><span> SSH connection tests passed successfully</span></span><span class="pun"><span>.</span></span>

               4. 检查MHA管理的mysql集群主从复制以及连接配置等是否正常:

  1. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">~]#</span><span class="pln"> masterha_check_repl </span><span class="pun">--</span><span class="pln">conf</span><span class="pun">=/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span><span class="pun">.</span><span class="pln">cnf</span>
  2. 1
    <span class="pln">........</span>
  3. 1
    <span class="pln">......</span>
  4. 1
    <span class="pln">......</span>
  5. 1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    <span class="pln">Sun Nov 27 19:46:05 2016 - [info] Checking replication health on 10.1.252.218..
    Sun Nov 27 19:46:05 2016 - [info]  ok.
    Sun Nov 27 19:46:05 2016 - [info] Checking replication health on 10.1.249.70..
    Sun Nov 27 19:46:06 2016 - [info]  ok.
    Sun Nov 27 19:46:06 2016 - [warning] master_ip_failover_script is not defined.
    Sun Nov 27 19:46:06 2016 - [warning] shutdown_script is not defined.
    Sun Nov 27 19:46:06 2016 - [info] Got exit code 0 (Not master dead).

    <span>MySQL Replication Health is OK.</span>
    </span>

                5. 启动MHA,并测试主节点工作状况

  1. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> nohup masterha_manager </span><span class="pun">--</span><span class="pln">conf</span><span class="pun">=/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span><span class="pun">.</span><span class="pln">cnf </span><span class="pun">&gt;</span><span class="pln"> </span><span class="pun">/</span><span class="pln">data</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span><span class="pun">/</span><span class="pln">manager</span><span class="pun">.</span><span class="pln">log </span><span class="lit">2</span><span class="pun">&gt;&amp;</span><span class="lit">1</span><span class="pun">&amp;</span>
  2. 1
    <span class="pun">[</span><span class="lit">1</span><span class="pun">]</span><span class="pln"> </span><span class="lit">7852</span>
  3. 1
    <span class="lit"><span>#nohup:后台运行masterha_manager,否则当终端关闭时候也会将</span></span>
    masterha_manager关闭
  4. #后续管道作用: 将启动信息等导入指定的日志文件中

  5. 1
    <span class="pun"><span>#检查主节点状态:</span></span>
  6. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> masterha_check_status </span><span class="pun">--</span><span class="pln">conf</span><span class="pun">=/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">masterha</span><span class="pun">/</span><span class="pln">app1</span><span class="pun">.</span><span class="pln">cnf</span>
  7. 1
    <span class="pln">app1 </span><span class="pun">(</span><span class="pln">pid</span><span class="pun">:</span><span class="lit">7852</span><span class="pun">)</span><span class="pln"> is<span> running</span></span><span class="pun"><span>(</span></span><span class="lit"><span>0</span></span><span class="pun"><span>:</span></span><span class="pln"><span>PING_OK</span></span><span class="pun"><span>)</span>,</span><span class="pln"> master</span><span class="pun">:</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.184</span>
  8. 1
    <span class="lit"><span>#若主节点或集群工作不正常,则会显示"app1 is stopped,......"等类似信息</span></span>
                  6. 若要停止MHA ,可使用masterha_stop 命令
  1. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> masterha_stop --conf=/etc/masterha/app1.cnf</span>
  2. 1
    <span class="pln"><span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]# Stopped app1 successfully.</span><br></span>

3.3 故障转移测试

3.3.1 主节点故障转移

           1. 在master节点关闭mariadb服务

  1. 1
    <span class="pun">[</span><span class="pln">root@node1 </span><span class="pun">~]#</span><span class="pln"> ps aux </span><span class="pun">|</span><span class="pln"> grep mysql</span>
  2. 1
    <span class="pln">mysql     </span><span class="lit">23655</span><span class="pln">  </span><span class="lit">0.0</span><span class="pln">  </span><span class="lit">0.0</span><span class="pln"> </span><span class="lit">113252</span><span class="pln">   </span><span class="lit">456</span><span class="pln"> </span><span class="pun">?</span><span class="pln">        </span><span class="typ">Ss</span><span class="pln">   </span><span class="lit">12</span><span class="pun">:</span><span class="lit">53</span><span class="pln">   </span><span class="lit">0</span><span class="pun">:</span><span class="lit">00</span><span class="pln"> </span><span class="pun">/</span><span class="pln">bin</span><span class="pun">/</span><span class="pln">sh </span><span class="pun">/</span><span class="pln">usr</span><span class="pun">/</span><span class="pln">bin</span><span class="pun">/</span><span class="pln">mysqld_safe </span><span class="pun">--</span><span class="pln">basedir</span><span class="pun">=/</span><span class="pln">usr</span>
  3. 1
    <span class="pln">mysql     </span><span class="lit">23872</span><span class="pln">  </span><span class="lit">0.0</span><span class="pln"> </span><span class="lit">12.4</span><span class="pln"> </span><span class="lit">1102560</span><span class="pln"> </span><span class="lit">80756</span><span class="pln"> </span><span class="pun">?</span><span class="pln">       </span><span class="typ">Sl</span><span class="pln">   </span><span class="lit">12</span><span class="pun">:</span><span class="lit">53</span><span class="pln">   </span><span class="lit">0</span><span class="pun">:</span><span class="lit">11</span><span class="pln"> </span><span class="pun">/</span><span class="pln">usr</span><span class="pun">/</span><span class="pln">libexec</span><span class="pun">/</span><span class="pln">mysqld </span><span class="pun">--</span><span class="pln">basedir</span><span class="pun">=/</span><span class="pln">usr </span><span class="pun">--</span><span class="pln">datadir</span><span class="pun">=/</span><span class="pln">var</span><span class="pun">/</span><span class="pln">lib</span><span class="pun">/</span><span class="pln">mysql </span><span class="pun">--</span><span class="pln">plugin</span><span class="pun">-</span><span class="pln">dir</span><span class="pun">=/</span><span class="pln">usr</span><span class="pun">/</span><span class="pln">lib64</span><span class="pun">/</span><span class="pln">mysql</span><span class="pun">/</span><span class="pln">plugin </span><span class="pun">--</span><span class="pln">log</span><span class="pun">-</span><span class="pln">error</span><span class="pun">=/</span><span class="pln">var</span><span class="pun">/</span><span class="pln">log</span><span class="pun">/</span><span class="pln">mariadb</span><span class="pun">/</span><span class="pln">mariadb</span><span class="pun">.</span><span class="pln">log </span><span class="pun">--</span><span class="pln">pid</span><span class="pun">-</span><span class="pln">file</span><span class="pun">=/</span><span class="pln">var</span><span class="pun">/</span><span class="pln">run</span><span class="pun">/</span><span class="pln">mariadb</span><span class="pun">/</span><span class="pln">mariadb</span><span class="pun">.</span><span class="pln">pid </span><span class="pun">--</span><span class="pln">socket</span><span class="pun">=/</span><span class="pln">var</span><span class="pun">/</span><span class="pln">lib</span><span class="pun">/</span><span class="pln">mysql</span><span class="pun">/</span><span class="pln">mysql</span><span class="pun">.</span><span class="pln">sock</span>
  4. 1
    <span class="pln">root      </span><span class="lit">30281</span><span class="pln">  </span><span class="lit">0.0</span><span class="pln">  </span><span class="lit">0.1</span><span class="pln"> </span><span class="lit">112644</span><span class="pln">   </span><span class="lit">952</span><span class="pln"> pts</span><span class="pun">/</span><span class="lit">0</span><span class="pln">    R</span><span class="pun">+</span><span class="pln">   </span><span class="lit">20</span><span class="pun">:</span><span class="lit">16</span><span class="pln">   </span><span class="lit">0</span><span class="pun">:</span><span class="lit">00</span><span class="pln"> grep </span><span class="pun">--</span><span class="pln">color</span><span class="pun">=</span><span class="pln">auto mysql</span>
  5. 1
    <span class="pun">[</span><span class="pln">root@node1 </span><span class="pun">~]#</span><span class="pln"> </span>
  6. 1
    <span class="pun">[</span><span class="pln">root@node1 </span><span class="pun">~]#</span><span class="pln"> </span>
  7. 1
    <span class="pun">[</span><span class="pln">root@node1 </span><span class="pun">~]#</span><span class="pln"> <b>killall mysqld mysqld_safe</b></span>
            2. 在manager-node 上查看/data/masterha/app1/manager.log
  1. 1
    <span class="typ">vim /data/masterha/app1/manager.log</span>
  2. 1
    <span class="typ">......................</span>
  3. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"><span> </span></span><span class="typ"><span>Master</span></span><span class="pln"><span> is down</span></span><span class="pun"><span>!</span></span>
  4. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Terminating</span><span class="pln"> monitoring script</span><span class="pun">.</span>
  5. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Got</span><span class="pln"> exit code </span><span class="lit">20</span><span class="pln"> </span><span class="pun">(</span><span class="typ">Master</span><span class="pln"> dead</span><span class="pun">).</span>
  6. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> MHA</span><span class="pun">::</span><span class="typ">MasterFailover</span><span class="pln"> version </span><span class="lit">0.56</span><span class="pun">.</span>
  7. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Starting</span><span class="pln"> master failover</span><span class="pun">.</span>
  8. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span>
  9. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="pun">*</span><span class="pln"> </span><span class="typ">Phase</span><span class="pln"> </span><span class="lit">1</span><span class="pun">:</span><span class="pln"> </span><span class="typ">Configuration</span><span class="pln"> </span><span class="typ">Check</span><span class="pln"> </span><span class="typ">Phase</span><span class="pun">..</span>
  10. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span>
  11. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> GTID failover mode </span><span class="pun">=</span><span class="pln"> </span><span class="lit">0</span>
  12. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Dead</span><span class="pln"> </span><span class="typ">Servers</span><span class="pun">:</span>
  13. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln">   </span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.184</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.184</span><span class="pun">:</span><span class="lit">3306</span><span class="pun">)</span>
  14. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Checking</span><span class="pln"> master reachability via </span><span class="typ">MySQL</span><span class="pun">(</span><span class="pln">double check</span><span class="pun">)...</span>
  15. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln">  ok</span><span class="pun">.</span>
  16. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Alive</span><span class="pln"> </span><span class="typ">Servers</span><span class="pun">:</span>
  17. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln">   </span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">252.218</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">252.218</span><span class="pun">:</span><span class="lit">3306</span><span class="pun">)</span>
  18. 1
    <span class="typ">Sun</span><span class="pln"> </span><span class="typ">Nov</span><span class="pln"> </span><span class="lit">27</span><span class="pln"> </span><span class="lit">20</span><span class="pun">:</span><span class="lit">19</span><span class="pun">:</span><span class="lit">06</span><span class="pln"> </span><span class="lit">2016</span><span class="pln"> </span><span class="pun">-</span><span class="pln"> </span><span class="pun">[</span><span class="pln">info</span><span class="pun">]</span><span class="pln">   </span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.70</span><span class="pun">(</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">249.70</span><span class="pun">:</span><span class="lit">3306</span><span class="pun">)</span>
  19. 1
    2
    <span class="pun"><span>Sun Nov 27 20:19:06 2016 - [info] The latest binary log file/position on all slaves is master-bin.000003:1353</span><font color="#bebec5">
    </font></span>
  20. Sun Nov 27 20:19:08 2016 - [info] * Phase 3.3: Determining New Master Phase..
    Sun Nov 27 20:19:08 2016 - [info]
    Sun Nov 27 20:19:08 2016 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
    Sun Nov 27 20:19:08 2016 - [info] All slaves received relay logs to the same position. No need to resync each other.
    Sun Nov 27 20:19:08 2016 - [info] Searching new master from slaves..
    Sun Nov 27 20:19:08 2016 - [info] Candidate masters from the configuration file:
    Sun Nov 27 20:19:08 2016 - [info] 10.1.252.218(10.1.252.218:3306) Version=5.5.44-MariaDB-log (oldest major version between slaves) log-bin:enabled
    Sun Nov 27 20:19:08 2016 - [info] Replicating from 10.1.249.184(10.1.249.184:3306)
    Sun Nov 27 20:19:08 2016 - [info] Primary candidate for the new Master (candidate_master is set)
    Sun Nov 27 20:19:08 2016 - [info] Non-candidate masters:
    Sun Nov 27 20:19:08 2016 - [info] Searching from candidate_master slaves which have received the latest relay log events..
  21. Sun Nov 27 20:19:09 2016 - [info] Setting read_only=0 on 10.1.252.218(10.1.252.218:3306).. Sun Nov 27 20:19:08 2016 - [info] New master is 10.1.252.218(10.1.252.218:3306)
    Sun Nov 27 20:19:08 2016 - [info] Starting master failover..
  22. ..........................
  23. Applying log files succeeded. Sun Nov 27 20:19:09 2016 - [info] All relay logs were successfully applied. Sun Nov 27 20:19:09 2016 - [info] Getting new master's binlog name and position.. Sun Nov 27 20:19:09 2016 - [info] master-bin.000003:629 Sun Nov 27 20:19:09 2016 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.1.252.218', MASTER_PORT=3306, MASTER_LOG_FILE='master-bin.000003', MASTER_LOG_POS=629, MASTER_USER='repluser', MASTER_PASSWORD='xxx'; Sun Nov 27 20:19:09 2016 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address. Sun Nov 27 20:19:09 2016 - [info] Setting read_only=0 on 10.1.252.218(10.1.252.218:3306).. Sun Nov 27 20:19:09 2016 - [info] ok. Sun Nov 27 20:19:09 2016 - [info] ** Finished master recovery successfully. Sun Nov 27 20:19:09 2016 - [info] * Phase 3: Master Recovery Phase completed. Sun Nov 27 20:19:09 2016 - [info] Sun Nov 27 20:19:09 2016 - [info] * Phase 4: Slaves Recovery Phase.. Sun Nov 27 20:19:09 2016 - [info] Sun Nov 27 20:19:09 2016 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
  24. .....................

                        #由以上日志内容可以看到故障转移成功
                注意:
                        每次故障成功转移以后,MHA manager 会自动停止, 此时再使用maserha__check_status检测状态信息
                        会出错:
                         

                        出于数据完整性与可用性考虑,MHA每次故障转移后会停止工作,此后需要手动检查
                        数据是否有出错等,确保无误后,再重启MHA manager服务

                        

3.4 故障修复后重上线

        原master 节点故障修复后,数据恢复为原master节点上的数据,并作为一个新的集群从节点回集群中(配置需要
       重新修改) , 其IP地址必须配置为原master的ip地址,否则MHA将无法识别;
        再次启动MHA manager后,需要再检测一次各节点的工作状态,即执行masterha_check_status还有
       masterha_check_repl命令;
       
            

  1. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> <span>nohup masterha_manager </span></span><span class="pun"><span>--</span></span><span class="pln"><span>conf</span></span><span class="pun"><span>=/</span></span><span class="pln"><span>etc</span></span><span class="pun"><span>/</span></span><span class="pln"><span>masterha</span></span><span class="pun"><span>/</span></span><span class="pln"><span>app1</span></span><span class="pun"><span>.</span></span><span class="pln"><span>cnf </span></span><span class="pun"><span>&gt;</span></span><span class="pln"><span> </span></span><span class="pun"><span>/</span></span><span class="pln"><span>data</span></span><span class="pun"><span>/</span></span><span class="pln"><span>masterha</span></span><span class="pun"><span>/</span></span><span class="pln"><span>app1</span></span><span class="pun"><span>/</span></span><span class="pln"><span>manager</span></span><span class="pun"><span>.</span></span><span class="pln"><span>log </span></span><span class="lit"><span>2</span></span><span class="pun"><span>&gt;&amp;</span></span><span class="lit"><span>1</span></span><span class="pun"><span>&amp;</span></span>
  2. 1
    <span class="pun">[</span><span class="lit">1</span><span class="pun">]</span><span class="pln"> </span><span class="lit">10188</span>
  3. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> </span>
  4. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> </span>
  5. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> </span>
  6. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> <span>masterha_check_status </span></span><span class="pun"><span>--</span></span><span class="pln"><span>conf</span></span><span class="pun"><span>=/</span></span><span class="pln"><span>etc</span></span><span class="pun"><span>/</span></span><span class="pln"><span>masterha</span></span><span class="pun"><span>/</span></span><span class="pln"><span>app1</span></span><span class="pun"><span>.</span></span><span class="pln"><span>cnf</span></span>
  7. 1
    <span class="pln">app1 </span><span class="pun">(</span><span class="pln">pid</span><span class="pun">:</span><span class="lit">10188</span><span class="pun">)</span><span class="pln"> is running</span><span class="pun">(</span><span class="lit">0</span><span class="pun">:</span><span class="pln">PING_OK</span><span class="pun">),</span><span class="pln"> master</span><span class="pun">:</span><span class="lit">10.1</span><span class="pun">.</span><span class="lit">252.218</span>
  8. 1
    <span class="pun">[</span><span class="pln">root@node5 </span><span class="pun">.</span><span class="pln">ssh</span><span class="pun">]#</span><span class="pln"> </span>

  1. 1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    <span class="pln">[root@node5 .ssh]# <span>masterha_check_status --conf=/etc/masterha/app1.cnf</span>
    app1 (pid:10188) is running(0:PING_OK), master:10.1.252.218
    [root@node5 .ssh]# masterha_check_repl --conf=/etc/masterha/app1.cnf
    Sun Nov 27 21:03:04 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
    Sun Nov 27 21:03:04 2016 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
    Sun Nov 27 21:03:04 2016 - [info] Reading server configuration from /etc/masterha/app1.cnf..
    Sun Nov 27 21:03:04 2016 - [info] MHA::MasterMonitor version 0.56.
    Sun Nov 27 21:03:04 2016 - [warning] SQL Thread is stopped(no error) on 10.1.252.218(10.1.252.218:3306)
    Sun Nov 27 21:03:04 2016 - [info] Multi-master configuration is detected. Current primary(writable) master is 10.1.252.218(10.1.252.218:3306)
    Sun Nov 27 21:03:04 2016 - [info] Master configurations are as below:
    Master 10.1.252.218(10.1.252.218:3306), replicating from 10.1.249.184(10.1.249.184:3306)
    Master 10.1.249.184(10.1.249.184:3306), replicating from 10.1.252.218(10.1.252.218:3306), read-only

    Sun Nov 27 21:03:04 2016 - [info] GTID failover mode = 0
    Sun Nov 27 21:03:04 2016 - [info] Dead Servers:
    Sun Nov 27 21:03:04 2016 - [info] Alive Servers:
    Sun Nov 27 21:03:04 2016 - [info]   10.1.249.184(10.1.249.184:3306)
    Sun Nov 27 21:03:04 2016 - [info]   10.1.252.218(10.1.252.218:3306)
    Sun Nov 27 21:03:04 2016 - [info]   10.1.249.70(10.1.249.70:3306)
    Sun Nov 27 21:03:04 2016 - [info] Alive Slaves:
    Sun Nov 27 21:03:04 2016 - [info]   10.1.249.184(10.1.249.184:3306)  Version=5.5.44-MariaDB-log (oldest major version between slaves) log-bin:enabled
    Sun Nov 27 21:03:04 2016 - [info]     Replicating from 10.1.252.218(10.1.252.218:3306)
    Sun Nov 27 21:03:04 2016 - [info]     Primary candidate for the new Master (candidate_master is set)
    Sun Nov 27 21:03:04 2016 - [info]   10.1.249.70(10.1.249.70:3306)  Version=5.5.44-MariaDB-log (oldest major version between slaves) log-bin:enabled
    Sun Nov 27 21:03:04 2016 - [info]     Replicating from 10.1.252.218(10.1.252.218:3306)
    </span>
  2. 1
    <span class="pln">.....................</span>
  3. 1
    <span class="pln"><br></span>
  4. Sun Nov 27 21:03:08 2016 - [info] Checking replication health on 10.1.249.184.. Sun Nov 27 21:03:08 2016 - [info] ok. Sun Nov 27 21:03:08 2016 - [info] Checking replication health on 10.1.249.70.. Sun Nov 27 21:03:08 2016 - [info] ok. Sun Nov 27 21:03:08 2016 - [warning] master_ip_failover_script is not defined. Sun Nov 27 21:03:08 2016 - [warning] shutdown_script is not defined. Sun Nov 27 21:03:08 2016 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK.
 


 

null

原创文章,作者:ldt195175108,如若转载,请注明出处:/61094

联系我们

400-080-6560

在线咨询:点击这里给我发消息

邮件:1660809109@qq.com

工作时间:周一至周五,9:30-18:30,节假日同时也值班

友情链接:测试  万达登录  万达娱乐主管QQ  万达娱乐注册  万达注册  万达直属  万达主管  万达娱乐  万达娱乐招商  华宇招商