• Keepalived/ipvs을 이용한 리눅스 L4 – DR(Direct Routing) 구축

    Keepalived/ipvs을 이용한 리눅스 L4 – DR(Direct Routing) 구축

    작성자

    글 쓴 이: 차상환(netggio@netggio.pe.kr)

    홈페이지: http://netggio.pe.kr

    작 성 일: 2012년 06월 21일

    IPVS+keepalived 구성도


    리눅스 ipvs기반으로 리눅스L4를 구성 하고 failover를 구현을 목적으로 한다.

    실제 서비스에서 트래픽 2G 까지 무난히 버티는게 검증 됐고, 장비 장애시 failover가 정상적으로 되는게 확인 됐다.

    저비용 단순 다운로드 용도 펌 구성시 추천할만 하다.

    H/W

     

    Svr Name

    OS

    Memory

    Hdd

    IP

    Remarks

    1

    Active

    CentOS 5.xx 64bit

    4G

    73G x 1EA

    192.168.100.98(eth0)

    172.16.1.98(eth1)

    Load Balance

    2

    Backup

    CentOS 5.xx 64bit

    4G

    73G x 1EA

    192.168.100.99(eth0)

    172.16.1.99(eth1)

    Load Balance

    3

    Real01

    CentOS 4.xx 32bit

    2G

    73G x 1EA

    192.168.100.101

    Real Server

    4

    Real02

    CentOS 4.xx 32bit

    2G

    73G x 1EA

    192.168.100.102

    Real Server

    5

    Real03

    CentOS 4.xx 32bit

    2G

    73G x 1EA

    192.168.100.103

    Real Server

    6

    Real04

    CentOS 4.xx 32bit

    2G

    73G x 1EA

    192.168.100.104

    Real Server

    7

    Real05

    CentOS 4.xx 32bit

    2G

    73G x 1EA

    192.168.100.105

    Real Server

    • VIP로 사용할 ip 192.168.100.100
    • failover를 위해 Active <-> Backup 두 장비는 크로스케이블로 연결 되어 있어야 한다.

      APP

     

    Svr Name

    APP

    Remarks

    1

    Active

    Backup

    ipvsadm

    kernel-devel

    sendmail

    keepalived

    openssl

    openssl-devel

    ipvsadm,kernel-devel,sendmail,openssl,openssl-devel은

    OS에 포함되어 있는 rpm으로 사용

    2

    Real01

    Real02

    Real03

    Real04

    Real05

    httpd

    arptables_jf

    arptables은 OS에 포함되어 있는 rpm으로 사용

    keepalived 는 Version 1.2.1버전 사용했음

     http://www.keepalived.org/

     http://www.keepalived.org/software/keepalived-1.2.1.tar.gz

    keepalived은 ipvs control, failover, fail notice, vrrp 제공해주는 툴이라고 생각하면 된다.

    Active와 backup 장비는 인증절차 없이 양방향 ssh접속이 가능 하게 되어야 한다. (ssh-keygen -t rsa로 구글링 해보면 tip많을것임)

    keepalived 설치

    tar xzvf keepalived-1.2.1.tar.gz
    cd keepalived-1.2.1
    ./configure –enable-debug –with-kernel-dir=/lib/modules/uname-r/build

    configure 결과가 아래와 같이 리스팅됨. 아래와 같이 YES로 되어 있는지 확인

    ———————————
    Keepalived version
     : 1.2.1
    Compiler
     : gcc
    Compiler flags
     : -g -O2 -DETHERTYPE_IPV6=0x86dd
    Extra Lib
     : -lpopt -lssl -lcrypto
    Use IPVS Framework
     : Yes
    IPVS sync daemon support
     : Yes
    Use VRRP Framework
     : Yes
    Use Debug flags
     : Yes

    conffigure 끝나면 rpm으로 빌드 해서 rpm으로 설치 하면 됨

    cp keepalived-1.2.1/keepalived.spec /usr/src/redhat/SPEC/
    cp keepalived-1.2.1.tar.gz /usr/sr/redhat/SOURCE/
    cd /usr/src/redhat/SPEC
    rpmbuild -bb keepalived.spec
    rpm -iVh /usr/src/redhat/RPMS/i386/keepalived-*

    위 설치 했던 방식으로 동일하게 Backup에서 설치해준다.

    Keepalived Active/Backup 설정

    Keepalived Active conf

    config -> /etc/keepalived/keepalived.conf

    global_defs {
    notification_email {
    se@netggio.pe.kr
    }
    notification_email_from Ative@netggio.pe.kr
    smtp_server localhost
    smtp_connect_timeout 30
    }
    vrrp_instance VI_1 {
    interface eth0
    state MASTER
    virtual_router_id 9
    priority 200
    garp_master_delay 1
    advert_int 1
    lvs_sync_daemon_interface eth1
    track_interface {
    eth0
    }
    authentication {
    auth_type PASS
    auth_pass p@ssw0rd
    }
    virtual_ipaddress {
    192.168.100.100/32 brd 192.168.100.255 dev eth0 scope global
    }
    notify_master “/etc/keepalived/script/M.sh”
    notify_backup “/etc/keepalived/script/B.sh”
    }
    virtual_server 192.168.100.100 80 {
    delay_loop 2
    lb_algo rr
    lb_kind DR
    protocol TCP
    real_server 192.168.100.101 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 2
    }
    }
    real_server 192.168.100.102 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 2
    }
    }
    real_server 192.168.100.103 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 2
    }
    }
    real_server 192.168.100.104 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 2
    }
    }
    real_server 192.168.100.105 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 2
    }
    }
    }

    Keepalived Active Script

    1, Script Base Directory Create

    mkdir -p /etc/keepalived/script/run
    ln -s /sbin/keepalived /etc/keepalived/script/run/keepalived

    2, M.sh

    keepalived가 Active로 전환될때 자동으로 실행되는 script

    script 위치 -> /etc/keepalived/script

    /bin/sed -i ‘s/state BACKUP/state MASTER/g’ /etc/keepalived/keepalived.conf
    /bin/sed -i ‘s/priority 100/priority 200/g’ /etc/keepalived/keepalived.conf
    echo date +%Y/%m/%d" "%H:%M:%S | /bin/mail -s “hostname(date +%Y/%m/%d" "%H:%M:%S)
     :: 192.168.100.98 change Master-LVS” se@netggio.pe.kr

    3, B.sh

    keepalived가 backup로 전환될때 자동으로 실행되는 script

    script 위치 -> /etc/keepalived/script

    /bin/sed -i ‘s/state MASTER/state BACKUP/g’ /etc/keepalived/keepalived.conf
    /bin/sed -i ‘s/priority 200/priority 100/g’ /etc/keepalived/keepalived.conf
    echo date +%Y/%m/%d" "%H:%M:%S | /bin/mail -s “hostname(date +%Y/%m/%d" "%H:%M:%S)
     :: 192.168.100.98 change Backup-LVS” se@netggio.pe.kr

    4,START.sh

    Keepalived 구동 -> /etc/keepalived/script/START.sh

    script 위치 -> /etc/keepalived/script

    #!/bin/bash

    LOCAL=172.16.1.98
    REMOTE=172.16.1.99

    NET_CHK=nc -z -w 2 172.16.1.99 22 > /dev/null

    if [[ $? == “0” ]]

    then
    STATUS_REMOTE=ssh $REMOTE ipvsadm -L --daemon | awk '{print $1}'
    STATUS_LOCAL=ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'

    if [[ $STATUS_REMOTE == “master” ]]

    then

    if [[ $STATUS_LOCAL == “” ]]

    then
    echo “1”
    echo “/etc/keepalived/script/B.sh”
    /etc/keepalived/script/B.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi
    if [[ $STATUS_LOCAL == “master” ]]

    then
    echo “2”
    echo “hostname is Already running master”
    fi

    if [[ $STATUS_LOCAL == “backup” ]]

    then
    echo “3”
    echo “hostname is Already running Backup”
    fi

    fi

    if [[ $STATUS_REMOTE == “backup” ]]

    then

    if [[ $STATUS_LOCAL == “” ]]

    then
    echo “4”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi
    if [[ $STATUS_LOCAL == “master” ]]

    then
    echo “5”
    echo “hostname is Already running master”
    fi

    fi

    if [[ $STATUS_REMOTE == “” ]]

    then

    if [[ $STATUS_LOCAL == “” ]]

    then
    echo “6”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi
    if [[ $STATUS_LOCAL == “master” ]]

    then
    echo “7”
    echo “hostname is Already running Master”
    fi

    if [[ $STATUS_LOCAL == “backup” ]]

    then
    echo “8”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi

    fi

    else

    echo “$REMOTE NETWROK Error”

    STATUS_LOCAL=ipvsadm -L --daemon | awk '{print $1}'

    if [[ $STATUS_LOCAL == “” ]]

    then
    echo “9”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi

    if [[ $STATUS_LOCAL == “backup” ]]

    then
    echo “10”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start

    fi

    if [[ $STATUS_LOCAL == “master” ]]

    then
    echo “11”
    echo “hostname is Already running Master”

    fi

    fi

    5,STOP.sh

    Keepalived 구동 -> /etc/keepalived/script/STOP.sh

    script 위치 -> /etc/keepalived/script

    /etc/keepalived/script/run/keepalived stop

    6,net_stat.sh

    밸런스 확인

    script 위치 -> /etc/keepalived/

    #!/bin/bash
    while true
    do
    sleep 1.5
    echo “###############################################”
    ipvsadm -L -n –sort
    echo “###############################################”
    done


    7, status.sh

    현재 어느 장비가 active/backup인지 확인 하는 script

    script 위치 -> /etc/keepalived/


    #!/bin/bash

    LOCAL=192.168.100.98
    REMOTE=192.168.100.99

    nc -z -w 2 $REMOTE 22 > /dev/null

    if [[ $? == “0” ]]
    then
    echo “$LOCAL –” ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'
    echo “$REMOTE –” ssh $REMOTE ipvsadm -L --daemon | awk '{print $1}'

    else

    echo “$LOCAL –” ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'
    echo “$REMOTE — Network error”

    fi

    Keepalived Backup conf

    config -> /etc/keepalived/keepalived.conf

    global_defs {
    notification_email {
    se@netggio.pe.kr
    }
    notification_email_from backup@netggio.pe.kr
    smtp_server localhost
    smtp_connect_timeout 30
    }
    vrrp_instance VI_1 {
    interface eth0
    state BACKUP
    virtual_router_id 9
    priority 100
    garp_master_delay 1
    advert_int 1
    lvs_sync_daemon_interface eth1
    track_interface {
    eth0
    }
    authentication {
    auth_type PASS
    auth_pass p@ssw0rd
    }
    virtual_ipaddress {
    192.168.100.100/32 brd 192.168.100.255 dev eth0 scope global
    }
    notify_master “/etc/keepalived/script/M.sh”
    notify_backup “/etc/keepalived/script/B.sh”
    }
    virtual_server 192.168.100.100 80 {
    delay_loop 2
    lb_algo rr
    lb_kind DR
    protocol TCP
    real_server 192.168.100.101 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 3
    }
    }
    real_server 192.168.100.102 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 2
    }
    }
    real_server 192.168.100.103 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 2
    }
    }
    real_server 192.168.100.104 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 2
    }
    }
    real_server 192.168.100.105 80 {
    TCP_CHECK {
    connect_timeout 2
    nb_get_retry 2
    delay_before_retry 2
    }
    }
    }

    Keepalived Backup Script

    1, Script Base Directory Create

    mkdir -p /etc/keepalived/script/run
    ln -s /sbin/keepalived /etc/keepalived/script/run/keepalived

    2, M.sh

    keepalived가 Active로 전환될때 자동으로 실행되는 script

    script 위치 -> /etc/keepalived/script

    /bin/sed -i ‘s/state BACKUP/state MASTER/g’ /etc/keepalived/keepalived.conf
    /bin/sed -i ‘s/priority 100/priority 200/g’ /etc/keepalived/keepalived.conf
    echo date +%Y/%m/%d" "%H:%M:%S | /bin/mail -s “hostname(date +%Y/%m/%d" "%H:%M:%S)
     :: 192.168.100.99 change Master-LVS” se@netggio.pe.kr

    3, B.sh

    keepalived가 backup로 전환될때 자동으로 실행되는 script

    script 위치 -> /etc/keepalived/script

    /bin/sed -i ‘s/state MASTER/state BACKUP/g’ /etc/keepalived/keepalived.conf
    /bin/sed -i ‘s/priority 200/priority 100/g’ /etc/keepalived/keepalived.conf
    echo date +%Y/%m/%d" "%H:%M:%S | /bin/mail -s “hostname(date +%Y/%m/%d" "%H:%M:%S)
     :: 192.168.100.99 change Backup-LVS” se@netggio.pe.kr

    4,START.sh

    Keepalived 구동 -> /etc/keepalived/script/START.sh

    script 위치 -> /etc/keepalived/script

    #!/bin/bash

    LOCAL=172.16.1.99
    REMOTE=172.16.1.98

    NET_CHK=nc -z -w 2 172.16.1.99 22 > /dev/null

    if [[ $? == “0” ]]

    then
    STATUS_REMOTE=ssh $REMOTE ipvsadm -L --daemon | awk '{print $1}'
    STATUS_LOCAL=ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'

    if [[ $STATUS_REMOTE == “master” ]]

    then

    if [[ $STATUS_LOCAL == “” ]]

    then
    echo “1”
    echo “/etc/keepalived/script/B.sh”
    /etc/keepalived/script/B.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi
    if [[ $STATUS_LOCAL == “master” ]]

    then
    echo “2”
    echo “hostname is Already running master”
    fi

    if [[ $STATUS_LOCAL == “backup” ]]

    then
    echo “3”
    echo “hostname is Already running Backup”
    fi

    fi

    if [[ $STATUS_REMOTE == “backup” ]]

    then

    if [[ $STATUS_LOCAL == “” ]]

    then
    echo “4”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi
    if [[ $STATUS_LOCAL == “master” ]]

    then
    echo “5”
    echo “hostname is Already running master”
    fi

    fi

    if [[ $STATUS_REMOTE == “” ]]

    then

    if [[ $STATUS_LOCAL == “” ]]

    then
    echo “6”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi
    if [[ $STATUS_LOCAL == “master” ]]

    then
    echo “7”
    echo “hostname is Already running Master”
    fi

    if [[ $STATUS_LOCAL == “backup” ]]

    then
    echo “8”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi

    fi

    else

    echo “$REMOTE NETWROK Error”

    STATUS_LOCAL=ipvsadm -L --daemon | awk '{print $1}'

    if [[ $STATUS_LOCAL == “” ]]

    then
    echo “9”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start
    fi

    if [[ $STATUS_LOCAL == “backup” ]]

    then
    echo “10”
    echo “/etc/keepalived/script/M.sh”
    /etc/keepalived/script/M.sh
    echo “/etc/keepalived/script/run/keepalived start”
    /etc/keepalived/script/run/keepalived start

    fi

    if [[ $STATUS_LOCAL == “master” ]]

    then
    echo “11”
    echo “hostname is Already running Master”

    fi

    fi

    5,STOP.sh

    Keepalived 구동 -> /etc/keepalived/script/STOP.sh

    script 위치 -> /etc/keepalived/script

    /etc/keepalived/script/run/keepalived stop

    6,net_stat.sh

    밸런스 확인

    script 위치 -> /etc/keepalived/

    #!/bin/bash
    while true
    do
    sleep 1.5
    echo “###############################################”
    ipvsadm -L -n –sort
    echo “###############################################”
    done


    7, status.sh

    현재 어느 장비가 active/backup인지 확인 하는 script

    script 위치 -> /etc/keepalived/


    #!/bin/bash

    LOCAL=192.168.100.99
    REMOTE=192.168.100.98

    nc -z -w 2 $REMOTE 22 > /dev/null

    if [[ $? == “0” ]]
    then
    echo “$LOCAL –” ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'
    echo “$REMOTE –” ssh $REMOTE ipvsadm -L --daemon | awk '{print $1}'

    else

    echo “$LOCAL –” ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'
    echo “$REMOTE — Network error”

    fi

    그 밖에 확인 및 설정들..

    1, cat /etc/rc.local

    /etc/keepalived/script/START.sh
    /sbin/ipvsadm –set 8 5 8

    2, cat /etc/sysctl.conf

    net.ipv4.ip_forward = 1

    3, cat /etc/selinux/config ( selinux설정 변경시 리붓해야 적용됨. 리붓 못하는 상태라면 setenforce 0 으로 바로 적용 가능)

    SELINUX=disabled

    4, Sendmail 구동 여부

    Real Server 구성

    1, Arptables 설치되어 있어야 한다. (OS 디플트 패키지로 설치 하면됨)

    2, chkconfig –levels 35 arptables_jf on

    3, arptables 룰(cat /etc/sysconfig/arptables), arptables는 iptables과 사용방법이 유사함

    아래의 룰은 rip 가 192.168.100.101 이고 vip가 192.168.100.100인경우 예시임

    *filter
    :IN ACCEPT [12:336]
    :OUT ACCEPT [1:28]
    :FORWARD ACCEPT [0:0]
    [0:0] -A IN -d 192.168.100.100 -j DROP
    [0:0] -A OUT -d 192.168.100.100 -j mangle –mangle-ip-s 192.168.100.101
    COMMIT

    arptables -L -n (arptables 룰 상태 확인)

    Chain IN (policy ACCEPT)
    target source-ip destination-ip source-hw destination-hw hlen op hrd pro
    DROP 0.0.0.0/0 192.168.100.100 00/00 00/00 any 0000/0000 0000/0000 0000/0000
    Chain OUT (policy ACCEPT)
    target source-ip destination-ip source-hw destination-hw hlen op hrd pro
    mangle 0.0.0.0/0 192.168.100.100 00/00 00/00 any 0000/0000 0000/0000 0000/0000 –mangle-ip-s 192.168.100.101
    Chain FORWARD (policy ACCEPT)
    target source-ip destination-ip source-hw destination-hw hlen op hrd pro

    4, loopback 설정 cat /etc/sysconfig/network-scripts/ifcfg-lo:0

    DEVICE=lo:0
    IPADDR=192.168.100.100
    NETMASK=255.255.255.255
    BROADCAST=192.168.100.100
    ONBOOT=yes
    NAME=loopback

    5, 커널 파라메타값 수정

    net.ipv4.tcp_syncookies = 4
    net.ipv4.tcp_fin_timeout = 4
    net.ipv4.tcp_keepalive_time = 6
    net.ipv4.conf.lo.arp_ignore = 1
    net.ipv4.conf.lo.arp_announce = 2

    6, iptables로 인한 성능 저하 문제로 룰을 넣지 않았다.

    7, 아파치 인스톨

    출처: <http://netggio.pe.kr/wiki/index.php/Keepalived/ipvs%EC%9D%84_%EC%9D%B4%EC%9A%A9%ED%95%9C_%EB%A6%AC%EB%88%85%EC%8A%A4_L4_-_DR(Direct_Routing)_%EA%B5%AC%EC%B6%95>

    Comments

    comments

    Post Tagged with , ,

답글 남기기

이메일은 공개되지 않습니다. 필수 입력창은 * 로 표시되어 있습니다.