GSLB 구성 http://www.joinc.co.kr/modules/moniwiki/wiki.php/Site/cloud/ClusterLBWithOpenSource

Haproxy 옵션, 구동 방식 설명 등 http://helloworld.naver.com/helloworld/284659  

위 내용 토대로 정리할 예정.

 

로드밸런서를 꼭 로드밸런싱을 위해서만 써야 하는 것인가? 그렇지 않은 경우도 존재한다.

Active => Slave 형태로 haproxy를 사용 하고자 할때의 설정 방법은 아래와 같다.

 

/etc/haproxy/haproxy.cfg 파일 중 node 영역

<pre>

</pre>

구성하는 옵션은 Active => downinter 500 / Slave => backup 옵션 입니다. Active의 서비스 다운 발생시 Downtime interval 이후 Slave로 넘어가며, Active의 서비스 원상복구시 Slave => Active 로 변경 됩니다.

Keepalived/ipvs을 이용한 리눅스 L4 – DR(Direct Routing) 구축

작성자

글 쓴 이: 차상환(netggio@netggio.pe.kr)

홈페이지: http://netggio.pe.kr

작 성 일: 2012년 06월 21일

IPVS+keepalived 구성도


리눅스 ipvs기반으로 리눅스L4를 구성 하고 failover를 구현을 목적으로 한다.

실제 서비스에서 트래픽 2G 까지 무난히 버티는게 검증 됐고, 장비 장애시 failover가 정상적으로 되는게 확인 됐다.

저비용 단순 다운로드 용도 펌 구성시 추천할만 하다.

H/W

 

Svr Name

OS

Memory

Hdd

IP

Remarks

1

Active

CentOS 5.xx 64bit

4G

73G x 1EA

192.168.100.98(eth0)

172.16.1.98(eth1)

Load Balance

2

Backup

CentOS 5.xx 64bit

4G

73G x 1EA

192.168.100.99(eth0)

172.16.1.99(eth1)

Load Balance

3

Real01

CentOS 4.xx 32bit

2G

73G x 1EA

192.168.100.101

Real Server

4

Real02

CentOS 4.xx 32bit

2G

73G x 1EA

192.168.100.102

Real Server

5

Real03

CentOS 4.xx 32bit

2G

73G x 1EA

192.168.100.103

Real Server

6

Real04

CentOS 4.xx 32bit

2G

73G x 1EA

192.168.100.104

Real Server

7

Real05

CentOS 4.xx 32bit

2G

73G x 1EA

192.168.100.105

Real Server

  • VIP로 사용할 ip 192.168.100.100
  • failover를 위해 Active <-> Backup 두 장비는 크로스케이블로 연결 되어 있어야 한다.

    APP

 

Svr Name

APP

Remarks

1

Active

Backup

ipvsadm

kernel-devel

sendmail

keepalived

openssl

openssl-devel

ipvsadm,kernel-devel,sendmail,openssl,openssl-devel은

OS에 포함되어 있는 rpm으로 사용

2

Real01

Real02

Real03

Real04

Real05

httpd

arptables_jf

arptables은 OS에 포함되어 있는 rpm으로 사용

keepalived 는 Version 1.2.1버전 사용했음

 http://www.keepalived.org/

 http://www.keepalived.org/software/keepalived-1.2.1.tar.gz

keepalived은 ipvs control, failover, fail notice, vrrp 제공해주는 툴이라고 생각하면 된다.

Active와 backup 장비는 인증절차 없이 양방향 ssh접속이 가능 하게 되어야 한다. (ssh-keygen -t rsa로 구글링 해보면 tip많을것임)

keepalived 설치

tar xzvf keepalived-1.2.1.tar.gz
cd keepalived-1.2.1
./configure –enable-debug –with-kernel-dir=/lib/modules/uname-r/build

configure 결과가 아래와 같이 리스팅됨. 아래와 같이 YES로 되어 있는지 확인

———————————
Keepalived version
 : 1.2.1
Compiler
 : gcc
Compiler flags
 : -g -O2 -DETHERTYPE_IPV6=0x86dd
Extra Lib
 : -lpopt -lssl -lcrypto
Use IPVS Framework
 : Yes
IPVS sync daemon support
 : Yes
Use VRRP Framework
 : Yes
Use Debug flags
 : Yes

conffigure 끝나면 rpm으로 빌드 해서 rpm으로 설치 하면 됨

cp keepalived-1.2.1/keepalived.spec /usr/src/redhat/SPEC/
cp keepalived-1.2.1.tar.gz /usr/sr/redhat/SOURCE/
cd /usr/src/redhat/SPEC
rpmbuild -bb keepalived.spec
rpm -iVh /usr/src/redhat/RPMS/i386/keepalived-*

위 설치 했던 방식으로 동일하게 Backup에서 설치해준다.

Keepalived Active/Backup 설정

Keepalived Active conf

config -> /etc/keepalived/keepalived.conf

global_defs {
notification_email {
se@netggio.pe.kr
}
notification_email_from Ative@netggio.pe.kr
smtp_server localhost
smtp_connect_timeout 30
}
vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 9
priority 200
garp_master_delay 1
advert_int 1
lvs_sync_daemon_interface eth1
track_interface {
eth0
}
authentication {
auth_type PASS
auth_pass p@ssw0rd
}
virtual_ipaddress {
192.168.100.100/32 brd 192.168.100.255 dev eth0 scope global
}
notify_master “/etc/keepalived/script/M.sh”
notify_backup “/etc/keepalived/script/B.sh”
}
virtual_server 192.168.100.100 80 {
delay_loop 2
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.100.101 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 2
}
}
real_server 192.168.100.102 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 2
}
}
real_server 192.168.100.103 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 2
}
}
real_server 192.168.100.104 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 2
}
}
real_server 192.168.100.105 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 2
}
}
}

Keepalived Active Script

1, Script Base Directory Create

mkdir -p /etc/keepalived/script/run
ln -s /sbin/keepalived /etc/keepalived/script/run/keepalived

2, M.sh

keepalived가 Active로 전환될때 자동으로 실행되는 script

script 위치 -> /etc/keepalived/script

/bin/sed -i ‘s/state BACKUP/state MASTER/g’ /etc/keepalived/keepalived.conf
/bin/sed -i ‘s/priority 100/priority 200/g’ /etc/keepalived/keepalived.conf
echo date +%Y/%m/%d" "%H:%M:%S | /bin/mail -s “hostname(date +%Y/%m/%d" "%H:%M:%S)
 :: 192.168.100.98 change Master-LVS” se@netggio.pe.kr

3, B.sh

keepalived가 backup로 전환될때 자동으로 실행되는 script

script 위치 -> /etc/keepalived/script

/bin/sed -i ‘s/state MASTER/state BACKUP/g’ /etc/keepalived/keepalived.conf
/bin/sed -i ‘s/priority 200/priority 100/g’ /etc/keepalived/keepalived.conf
echo date +%Y/%m/%d" "%H:%M:%S | /bin/mail -s “hostname(date +%Y/%m/%d" "%H:%M:%S)
 :: 192.168.100.98 change Backup-LVS” se@netggio.pe.kr

4,START.sh

Keepalived 구동 -> /etc/keepalived/script/START.sh

script 위치 -> /etc/keepalived/script

#!/bin/bash

LOCAL=172.16.1.98
REMOTE=172.16.1.99

NET_CHK=nc -z -w 2 172.16.1.99 22 > /dev/null

if [[ $? == “0” ]]

then
STATUS_REMOTE=ssh $REMOTE ipvsadm -L --daemon | awk '{print $1}'
STATUS_LOCAL=ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'

if [[ $STATUS_REMOTE == “master” ]]

then

if [[ $STATUS_LOCAL == “” ]]

then
echo “1”
echo “/etc/keepalived/script/B.sh”
/etc/keepalived/script/B.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi
if [[ $STATUS_LOCAL == “master” ]]

then
echo “2”
echo “hostname is Already running master”
fi

if [[ $STATUS_LOCAL == “backup” ]]

then
echo “3”
echo “hostname is Already running Backup”
fi

fi

if [[ $STATUS_REMOTE == “backup” ]]

then

if [[ $STATUS_LOCAL == “” ]]

then
echo “4”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi
if [[ $STATUS_LOCAL == “master” ]]

then
echo “5”
echo “hostname is Already running master”
fi

fi

if [[ $STATUS_REMOTE == “” ]]

then

if [[ $STATUS_LOCAL == “” ]]

then
echo “6”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi
if [[ $STATUS_LOCAL == “master” ]]

then
echo “7”
echo “hostname is Already running Master”
fi

if [[ $STATUS_LOCAL == “backup” ]]

then
echo “8”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi

fi

else

echo “$REMOTE NETWROK Error”

STATUS_LOCAL=ipvsadm -L --daemon | awk '{print $1}'

if [[ $STATUS_LOCAL == “” ]]

then
echo “9”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi

if [[ $STATUS_LOCAL == “backup” ]]

then
echo “10”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start

fi

if [[ $STATUS_LOCAL == “master” ]]

then
echo “11”
echo “hostname is Already running Master”

fi

fi

5,STOP.sh

Keepalived 구동 -> /etc/keepalived/script/STOP.sh

script 위치 -> /etc/keepalived/script

/etc/keepalived/script/run/keepalived stop

6,net_stat.sh

밸런스 확인

script 위치 -> /etc/keepalived/

#!/bin/bash
while true
do
sleep 1.5
echo “###############################################”
ipvsadm -L -n –sort
echo “###############################################”
done


7, status.sh

현재 어느 장비가 active/backup인지 확인 하는 script

script 위치 -> /etc/keepalived/


#!/bin/bash

LOCAL=192.168.100.98
REMOTE=192.168.100.99

nc -z -w 2 $REMOTE 22 > /dev/null

if [[ $? == “0” ]]
then
echo “$LOCAL –” ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'
echo “$REMOTE –” ssh $REMOTE ipvsadm -L --daemon | awk '{print $1}'

else

echo “$LOCAL –” ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'
echo “$REMOTE — Network error”

fi

Keepalived Backup conf

config -> /etc/keepalived/keepalived.conf

global_defs {
notification_email {
se@netggio.pe.kr
}
notification_email_from backup@netggio.pe.kr
smtp_server localhost
smtp_connect_timeout 30
}
vrrp_instance VI_1 {
interface eth0
state BACKUP
virtual_router_id 9
priority 100
garp_master_delay 1
advert_int 1
lvs_sync_daemon_interface eth1
track_interface {
eth0
}
authentication {
auth_type PASS
auth_pass p@ssw0rd
}
virtual_ipaddress {
192.168.100.100/32 brd 192.168.100.255 dev eth0 scope global
}
notify_master “/etc/keepalived/script/M.sh”
notify_backup “/etc/keepalived/script/B.sh”
}
virtual_server 192.168.100.100 80 {
delay_loop 2
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.100.101 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 3
}
}
real_server 192.168.100.102 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 2
}
}
real_server 192.168.100.103 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 2
}
}
real_server 192.168.100.104 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 2
}
}
real_server 192.168.100.105 80 {
TCP_CHECK {
connect_timeout 2
nb_get_retry 2
delay_before_retry 2
}
}
}

Keepalived Backup Script

1, Script Base Directory Create

mkdir -p /etc/keepalived/script/run
ln -s /sbin/keepalived /etc/keepalived/script/run/keepalived

2, M.sh

keepalived가 Active로 전환될때 자동으로 실행되는 script

script 위치 -> /etc/keepalived/script

/bin/sed -i ‘s/state BACKUP/state MASTER/g’ /etc/keepalived/keepalived.conf
/bin/sed -i ‘s/priority 100/priority 200/g’ /etc/keepalived/keepalived.conf
echo date +%Y/%m/%d" "%H:%M:%S | /bin/mail -s “hostname(date +%Y/%m/%d" "%H:%M:%S)
 :: 192.168.100.99 change Master-LVS” se@netggio.pe.kr

3, B.sh

keepalived가 backup로 전환될때 자동으로 실행되는 script

script 위치 -> /etc/keepalived/script

/bin/sed -i ‘s/state MASTER/state BACKUP/g’ /etc/keepalived/keepalived.conf
/bin/sed -i ‘s/priority 200/priority 100/g’ /etc/keepalived/keepalived.conf
echo date +%Y/%m/%d" "%H:%M:%S | /bin/mail -s “hostname(date +%Y/%m/%d" "%H:%M:%S)
 :: 192.168.100.99 change Backup-LVS” se@netggio.pe.kr

4,START.sh

Keepalived 구동 -> /etc/keepalived/script/START.sh

script 위치 -> /etc/keepalived/script

#!/bin/bash

LOCAL=172.16.1.99
REMOTE=172.16.1.98

NET_CHK=nc -z -w 2 172.16.1.99 22 > /dev/null

if [[ $? == “0” ]]

then
STATUS_REMOTE=ssh $REMOTE ipvsadm -L --daemon | awk '{print $1}'
STATUS_LOCAL=ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'

if [[ $STATUS_REMOTE == “master” ]]

then

if [[ $STATUS_LOCAL == “” ]]

then
echo “1”
echo “/etc/keepalived/script/B.sh”
/etc/keepalived/script/B.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi
if [[ $STATUS_LOCAL == “master” ]]

then
echo “2”
echo “hostname is Already running master”
fi

if [[ $STATUS_LOCAL == “backup” ]]

then
echo “3”
echo “hostname is Already running Backup”
fi

fi

if [[ $STATUS_REMOTE == “backup” ]]

then

if [[ $STATUS_LOCAL == “” ]]

then
echo “4”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi
if [[ $STATUS_LOCAL == “master” ]]

then
echo “5”
echo “hostname is Already running master”
fi

fi

if [[ $STATUS_REMOTE == “” ]]

then

if [[ $STATUS_LOCAL == “” ]]

then
echo “6”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi
if [[ $STATUS_LOCAL == “master” ]]

then
echo “7”
echo “hostname is Already running Master”
fi

if [[ $STATUS_LOCAL == “backup” ]]

then
echo “8”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi

fi

else

echo “$REMOTE NETWROK Error”

STATUS_LOCAL=ipvsadm -L --daemon | awk '{print $1}'

if [[ $STATUS_LOCAL == “” ]]

then
echo “9”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start
fi

if [[ $STATUS_LOCAL == “backup” ]]

then
echo “10”
echo “/etc/keepalived/script/M.sh”
/etc/keepalived/script/M.sh
echo “/etc/keepalived/script/run/keepalived start”
/etc/keepalived/script/run/keepalived start

fi

if [[ $STATUS_LOCAL == “master” ]]

then
echo “11”
echo “hostname is Already running Master”

fi

fi

5,STOP.sh

Keepalived 구동 -> /etc/keepalived/script/STOP.sh

script 위치 -> /etc/keepalived/script

/etc/keepalived/script/run/keepalived stop

6,net_stat.sh

밸런스 확인

script 위치 -> /etc/keepalived/

#!/bin/bash
while true
do
sleep 1.5
echo “###############################################”
ipvsadm -L -n –sort
echo “###############################################”
done


7, status.sh

현재 어느 장비가 active/backup인지 확인 하는 script

script 위치 -> /etc/keepalived/


#!/bin/bash

LOCAL=192.168.100.99
REMOTE=192.168.100.98

nc -z -w 2 $REMOTE 22 > /dev/null

if [[ $? == “0” ]]
then
echo “$LOCAL –” ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'
echo “$REMOTE –” ssh $REMOTE ipvsadm -L --daemon | awk '{print $1}'

else

echo “$LOCAL –” ssh $LOCAL ipvsadm -L --daemon | awk '{print $1}'
echo “$REMOTE — Network error”

fi

그 밖에 확인 및 설정들..

1, cat /etc/rc.local

/etc/keepalived/script/START.sh
/sbin/ipvsadm –set 8 5 8

2, cat /etc/sysctl.conf

net.ipv4.ip_forward = 1

3, cat /etc/selinux/config ( selinux설정 변경시 리붓해야 적용됨. 리붓 못하는 상태라면 setenforce 0 으로 바로 적용 가능)

SELINUX=disabled

4, Sendmail 구동 여부

Real Server 구성

1, Arptables 설치되어 있어야 한다. (OS 디플트 패키지로 설치 하면됨)

2, chkconfig –levels 35 arptables_jf on

3, arptables 룰(cat /etc/sysconfig/arptables), arptables는 iptables과 사용방법이 유사함

아래의 룰은 rip 가 192.168.100.101 이고 vip가 192.168.100.100인경우 예시임

*filter
:IN ACCEPT [12:336]
:OUT ACCEPT [1:28]
:FORWARD ACCEPT [0:0]
[0:0] -A IN -d 192.168.100.100 -j DROP
[0:0] -A OUT -d 192.168.100.100 -j mangle –mangle-ip-s 192.168.100.101
COMMIT

arptables -L -n (arptables 룰 상태 확인)

Chain IN (policy ACCEPT)
target source-ip destination-ip source-hw destination-hw hlen op hrd pro
DROP 0.0.0.0/0 192.168.100.100 00/00 00/00 any 0000/0000 0000/0000 0000/0000
Chain OUT (policy ACCEPT)
target source-ip destination-ip source-hw destination-hw hlen op hrd pro
mangle 0.0.0.0/0 192.168.100.100 00/00 00/00 any 0000/0000 0000/0000 0000/0000 –mangle-ip-s 192.168.100.101
Chain FORWARD (policy ACCEPT)
target source-ip destination-ip source-hw destination-hw hlen op hrd pro

4, loopback 설정 cat /etc/sysconfig/network-scripts/ifcfg-lo:0

DEVICE=lo:0
IPADDR=192.168.100.100
NETMASK=255.255.255.255
BROADCAST=192.168.100.100
ONBOOT=yes
NAME=loopback

5, 커널 파라메타값 수정

net.ipv4.tcp_syncookies = 4
net.ipv4.tcp_fin_timeout = 4
net.ipv4.tcp_keepalive_time = 6
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2

6, iptables로 인한 성능 저하 문제로 룰을 넣지 않았다.

7, 아파치 인스톨

출처: <http://netggio.pe.kr/wiki/index.php/Keepalived/ipvs%EC%9D%84_%EC%9D%B4%EC%9A%A9%ED%95%9C_%EB%A6%AC%EB%88%85%EC%8A%A4_L4_-_DR(Direct_Routing)_%EA%B5%AC%EC%B6%95>