Rootop 服务器运维与web架构

上一篇文章中说到了两台机器之间的热备问题，并且已经实现主机down机后，备份机自动接管VIP。现在需要在此基础上不仅要提供主备之间自动切换，而且实现real server健康检查，通过keepalived就可以实现，因为2.6版本以后的内核都默认支持ipvs，不需要再单独安装lvs。通过keepalived参数定义即可实现lvs功能。

服务器环境IP信息：
real server 1        :192.168.1.10
real server 2        :192.168.1.11
direct server master :192.168.1.12
direct server slaves :192.168.1.13
VIP                  :192.168.1.14

查看是否支持：

[root@localhost ~]# modprobe -l | grep ipvs
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko

本文章跟之前有关keepalived包括lvs文章中出现的ip信息不一样，这是因为一部分是在公司部署环境并记录的，一部分是在家中，IP信息不一样，大家注意一下。

安装keepalived前面已经说过，不再提，编辑主配置文件：

! Configuration File for keepalived

global_defs {
   notification_email {
   root@networkquestions.org
   }
   notification_email_from venus@networkquestions.org
   smtp_server localhost
   smtp_connect_timeout 30
   router_id LVS_Node_master
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.14                //提供服务的VIP
    }
}

virtual_server 192.168.1.14 80 {    //定义虚拟服务器组
    delay_loop 6                    //运行情况检查
    lb_algo rr                      //调度算法
    lb_kind DR                      //负载机制
    persistence_timeout 50          //会话保持时间
    protocol TCP                    //定义转发协议类型

    real_server 192.168.1.10 80 {   //节点服务器
        weight 1                    //权重
        TCP_CHECK {                 //tcp 检查
        connect_timeout 3           //连接超时/秒
        nb_get_retry 3              //重试次数
        delay_before_retry 3        //重试间隔
                  }
           }
     real_server 192.168.1.11 80 {
        weight 1
        TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
                    }
            }
    }

}

保存，退出，在direct server BACKUP中直接复制粘贴以上部分。只需修改从机为BACKUP状态和priority 99权值，小于主机即可。重启keepalived。

这样direct server和real server的健康检查配置完成。在real server中只需参考：https://www.rootop.org/pages/2078.html  运行real server的执行脚本即可。

direct server 主备切换测试：

停掉主direct server的keepalived服务，或者直接拔掉网线。

[root@localhost ~]# service keepalived stop
停止 keepalived： [确定]
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:5b:dd:da brd ff:ff:ff:ff:ff:ff
inet 192.168.1.12/24 brd 192.168.1.255 scope global eth0
inet6 fe80::20c:29ff:fe5b:ddda/64 scope link
valid_lft forever preferred_lft forever
3: sit0: <NOARP> mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0

主服务器释放192.168.1.14 IP，查看备用服务器：

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:3b:7e:f3 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.13/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.14/32 scope global eth0
inet6 fe80::20c:29ff:fe3b:7ef3/64 scope link
valid_lft forever preferred_lft forever
3: sit0: <NOARP> mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0

备机接管vip。

通过浏览器访问VIP，返回内容正常。启动主服务器keepalived服务，重新接管VIP。

real server 健康检查：

停掉real server 1的web服务或者拔掉网线，查看direct server的 /var/log/messages

[root@localhost ~]# tail -f /var/log/messages
Feb 22 09:42:10 localhost Keepalived_healthcheckers[3353]: TCP connection to [192.168.1.10]:80 failed !!!
Feb 22 09:42:11 localhost Keepalived_healthcheckers[3353]: Removing service [192.168.1.10]:80 from VS [192.168.1.14]:80
Feb 22 09:42:11 localhost Keepalived_healthcheckers[3353]: Remote SMTP server [0.0.0.0]:25 connected.
Feb 22 09:42:12 localhost Keepalived_healthcheckers[3353]: SMTP alert successfully sent.
web服务停止后自动从vs中移出节点。

启动real server 1 的web服务，查看direct server 日志：
Feb 22 09:43:35 localhost Keepalived_healthcheckers[3353]: TCP connection to [192.168.1.10]:80 success.
Feb 22 09:43:35 localhost Keepalived_healthcheckers[3353]: Adding service [192.168.1.10]:80 to VS [192.168.1.14]:80
Feb 22 09:43:35 localhost Keepalived_healthcheckers[3353]: Remote SMTP server [0.0.0.0]:25 connected.
Feb 22 09:43:36 localhost Keepalived_healthcheckers[3353]: SMTP alert successfully sent.

web服务恢复后节点自动加到vs中。此过程中，运维只需要修复web服务，无需对keepalived做任何配置。

之前写了基本的安装，现在简单配置测试一下，主要看切换效果。

keepalived官方手册：http://www.keepalived.org/pdf/UserGuide.pdf

服务器信息：
master : 192.168.1.51
backup : 192.168.1.50
VIP       : 192.168.1.55

主服务器配置 /etc/keepalived/keepalived.conf ：

global_defs {
notification_email {
root@networkquestions.org     //定义报警邮件
}
notification_email_from warn@networkquestions.org            //定义发件人
smtp_server localhost               //指定smtp服务器，这里直接改为localhost
smtp_connect_timeout 30       //smtp连接超时时间
router_id node1                         //节点名称
}

vrrp_instance VI_1 {
state MASTER                //设置为主服务器
interface eth0                 //定义虚拟ip绑定接口
virtual_router_id 51     //VRRP组名，两个节点必须一样，指明各个节点属于同一VRRP组 mcast_src_ip 192.168.1.51    //发送多播包的地址，如果不设置默认使用绑定的网卡
priority 100                    //优先级，必须高于从服务器
advert_int 1                     //组播信息发送间隔，两个节点设置必须一样
authentication {               //认证，默认即可。主从需一致。
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.55                  //绑定的虚拟ip
}
}

我这里只保留了以上部分，其它参数全部删除。

从服务器配置 /etc/keepalived/keepalived.conf ：

global_defs {
notification_email {
acassen@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server localhost
smtp_connect_timeout 30
router_id node2
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.1.50      //从服务器ip
priority 99                                   //小于主服务器
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.55
}
}

保存退出，重启keepalived。

主服务器执行：
[root@rhel ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:d4:de:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.51/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.55/32 scope global eth0
inet6 fe80::20c:29ff:fed4:de01/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
link/ether 00:0c:29:d4:de:0b brd ff:ff:ff:ff:ff:ff
4: sit0: <NOARP> mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
看到192.168.1.55 已经绑定到eth0中。
tail -f /var/log/messages 会有相关信息输出。

现在拔掉主服务器网线，去从服务器查看：
[root@rhel ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0c:29:7f:5b:93 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.50/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.55/32 scope global eth0
inet6 fe80::20c:29ff:fe7f:5b93/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
link/ether 00:0c:29:7f:5b:9d brd ff:ff:ff:ff:ff:ff
4: sit0: <NOARP> mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
接管了192.168.1.55 绑定到eth0

插上主服务器的网线，重启keepalived，再次执行ip  a 发现重新接管VIP。
测试完成。

PS:

keepalived 互为主备，当同一时间只有一台节点接管vip时，另一台处于备份状态，利用率不高，所以配置两个VIP，互为主备。

节点1：

global_defs {
   notification_email {
     xxx@qq.com
   }
   notification_email_from root@localhost
   smtp_server localhost
   smtp_connect_timeout 30
   router_id node1
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.200.16
    }
}

vrrp_instance VI_2 {
    state BACKUP
    interface eth0
    virtual_router_id 52
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.200.26
    }
}

节点2：

global_defs {
   notification_email {
     xxx@qq.com
   }
   notification_email_from root@localhost
   smtp_server localhost
   smtp_connect_timeout 30
   router_id node2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.200.16
    }
}

vrrp_instance VI_2 {
    state MASTER
    interface eth0
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.200.26
    }
}

官网地址：http://www.keepalived.org/
官网文档：http://www.keepalived.org/documentation.html

Keepalived的作用是检测web服务器的状态，如果有一台web服务器死机或工作出现故障
Keepalived将检测到，并将有故障的web服务器从系统中剔除，当web服务器工作正常后
Keepalived自动将web服务器加入到服务器群中，这些工作全部自动完成，不需要人工干涉
需要人工做的只是修复故障的web服务器。

Layer3,4,5工作在IP/TCP协议栈的IP层，TCP层，及应用层,原理分别如下：

Layer3：Keepalived使用Layer3的方式工作式时，Keepalived会定期向服务器群中的服务器
发送一个ICMP的数据包（既我们平时用的Ping程序）,如果发现某台服务的IP地址没有激活
Keepalived便报告这台服务器失效，并将它从服务器群中剔除，这种情况的典型例子是某台
服务器被非法关机。

Layer3的方式是以服务器的IP地址是否有效作为服务器工作正常与否的标准。

Layer4:如果您理解了Layer3的方式，Layer4就容易了。Layer4主要以TCP端口的状态来
定服务器工作正常与否。如web server的服务端口一般是80，如果Keepalived检测到80端
口没有启动，则Keepalived将把这台服务器从服务器群中剔除。

Layer5：Layer5就是工作在具体的应用层了，比Layer3,Layer4要复杂一点，在网络上占用
的带宽也要大一些。Keepalived将根据用户的设定检查服务器程序的运行是否正常，如果与
用户的设定不相符，则Keepalived将把服务器从服务器群中剔除。

系统环境：
[root@rhel ~]# uname -a
Linux rhel 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

目前keepalived最新版本下载：
[root@rhel ~]#wget -c http://www.keepalived.org/software/keepalived-1.2.12.tar.gz
[root@rhel ~]# tar zxvf keepalived-1.2.12.tar.gz
[root@rhel ~]# cd keepalived-1.2.12
安装编译环境：
[root@rhel ~]# yum install -y gcc gcc-c++ openssl openssl-devel

[root@rhel keepalived-1.2.12]# ./configure --prefix=/usr/local/keepalived //指定安装路径

[root@rhel keepalived-1.2.12]# make && make install //安装
将keepalived命令软连接到/usr/bin下
[root@rhel keepalived-1.2.12]# ln -s /usr/local/keepalived/sbin/keepalived /usr/bin/keepalived

添加启动脚本且方便用service keepalived start/stop/restart管理
[root@rhel keepalived-1.2.12]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived
[root@rhel keepalived-1.2.12]# chmod 755 /etc/init.d/keepalived //添加执行权限
[root@rhel keepalived-1.2.12]# chkconfig keepalived on //开机启动

修改/etc/init.d/keepalived中的程序路径
# Source configuration file (we set KEEPALIVED_OPTIONS there)
. /etc/sysconfig/keepalived

改为：
# Source configuration file (we set KEEPALIVED_OPTIONS there)
. /usr/local/keepalived/etc/sysconfig/keepalived

默认情况下，keepalived 会读取 /etc/keepalived 下keepalived.conf 文件
如果没有建立这个文件，keepalived也不会报错，但是会发现，所创建的关于keepalived的相关参数根本就没有生效。
[root@rhel keepalived-1.2.12]# mkdir /etc/keepalived
[root@rhel keepalived-1.2.12]# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

启动测试：
[root@rhel keepalived-1.2.12]# service keepalived restart
停止 keepalived： [确定]
启动 keepalived： [确定]

安装完成