HA概述
HA(High Available)即由两台计算机组成并对外提供一组相同的服务，也叫做一主一备模式。正常情况下由主服务器提供服务，备服务器处于待机备用，备机可以随时接替主服务器的工作。也就是当主服务器宕机或所提供的服务不可用时，备用服务器会主动激活并且替换主服务器继续提供服务，这时主服务器上的服务资源包括网络(IP)、存储、服务(Web/数据库)就会转移到备机接管，从而提供不间断的服务。基于此，便可以将服务器的宕机时间减少到最低，对业务实现不中断或短暂中断。
由一组计算机(多台)组成一个整体并向用户提供相同的网络资源或服务，这种模式叫做HA集群(High Available Cluster)。
Oracle HA
Oracle HA高可用，又叫做双机热备，一般用于关键性业务。
Oracle双机热备模式
常用的有Data Guard、RAC(Real Application Clusters)、基于HA软件实现的双机热备。
作用和区别
l Data Guard采用重做日志复制技术，对主业务数据进行实时的异步同步备份，有一主一备、一主多备模式，一般常用于容灾(异地备份、灾难恢复)；存储独立、数据完整备份；备机可以做数据分析、报表统计等；主备自由切换，能实现自动快速故障转移；备用角色强制转换为主用角色会破环Data Guard架构；设计复杂、维护难度大
l RAC多个实例同时运行，无主备概念，有集群负载功能，其中某一台down机，不影响整体服务，不存在故障切换时间，可以提供高性能服务；存储共享，由ASM管理存储；硬件成本低；设计简单、维护难度大
l HA双机热备可以提供高可用性，保证业务的持续稳定运行，可以实现自动快速故障转移，存在短暂的切换时间(10-30s)。一般用于关键性业务；存储共享；有商业的HA方案，也可以使用开源的高可用软件keepalved、heartbeat搭建实现HA方案；设计复杂、维护难度小
自由组合
l HA双机和Data Guard
l Data Guard和RAC
Oracle HA+Keepalived架构
l Oracle HA双机热备+Keepalived
l Oracle HA双机+DataGuard+Keepalived集群
搭建Oracle HA主备服务器
软件版本：
Orace 11g R2
Keepalived 1.3.2
主服务器: DB1
备服务器: DB2
环境介绍
Hostname |
IP |
OS |
Role |
hmdg-db1 |
172.16.10.25 |
CentOS6.9 |
MASTER |
hmdg-db2 |
172.16.10.26 |
CentOS6.9 |
BACKUP |
VIP |
172.16.10.130、172.16.10.131 |
||
Share Disk |
/dev/sdb1 mount on /oradata |
||
ORACLE_BASE |
/u01/app/oracle |
||
ORACLE_HOME |
/u01/app/oracle/product/11.2.0/db_1 |
||
ORACLE_SID |
HMODB |
||
Datadir |
/oradata/HMODB |
||
Controlfiles |
/oradata/HMODB/control01.ctl,/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl, /home/oracle/rman/HMODB/control03.ctl |
||
Oracle软件安装步骤
1、 同时在主备上安装Oracle数据库软件，数据库的安装目录和环境变量保持一致
2、 仅在主服务器上建立数据库实例(主服务器上挂载共享存储并使用DBCA新建库)
3、 主服务创建数据库实例后，将主库上的控制文件、参数文件(SPFILE)、以及密码文件传输到备用服务器上相应的目录
l 备用服务器DB2
##创建对应的数据库文件目录 [root@hmdg-db2~]#su-oracle [oracle@hmdg-db2~]$mkdir/u01/app/oracle/admin/HMODB [oracle@hmdg-db2~]$mkdir/u01/app/oracle/admin/HMODB/{adump,dpdump,pfile} [oracle@hmdg-db2~]$mkdir/u01/app/oracle/flash_recovery_area/HMODB ##密码文件 [oracle@hmdg-db2~]$scporacle@hmdg-db1:/u01/app/oracle/product/11.2.0/db_1/dbs/orapwHMODB/u01/app/oracle/product/11.2.0/db_1/dbs/orapwHMODB ##SPFILE参数文件 [oracle@hmdg-db2~]$scporacle@hmdg-db1:/u01/app/oracle/product/11.2.0/db_1/dbs/spfileHMODB.ora/u01/app/oracle/product/11.2.0/db_1/dbs/spfileHMODB.ora ##控制文件，仅需要将位于非共享存储的数据目录下的控制文件传输到其他磁盘或目录位置下 [oracle@hmdg-db2~]$scporacle@hmdg-db1:/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl [oracle@hmdg-db2~]$cp/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl/home/oracle/rman/HMODB/control03.ctl
在备用服务器上启动数据库
2、 主服务器上卸载共享存储
3、 备服务器上挂载共享存储
4、 备服务器上启动监听
5、 备服务器上启动数据库实例
注意：不要同时挂载共享存储启动数据库实例，否则会导致数据不一致
l 主服务器DB1
##关闭数据库实例 [oracle@hmdg-db1~]$sqlplus/assysdba sql>shutdownimmediate ##关闭数据库监听 [oracle@hmdg-db1~]$lsnrctlstop ##卸载共享存储 [oracle@hmdg-db1~]$umount/oradata
l 备服务器DB2
##挂载/dev/sdb1共享存储到/oradta下 [root@hmdg-db2~]#mount/dev/sdb1/oradata/ ##启动数据库监听 [oracle@hmdg-db2~]$lsnrctlstart [oracle@hmdg-db2~]$exportORACLE_SID=HMODB ##启动数据库实例 [oracle@hmdg-db2~]$sqlplus/assysdba sql>startup ORACLEinstancestarted. TotalSystemGlobalArea1603411968bytes FixedSize2213776bytes VariableSize1056966768bytes DatabaseBuffers536870912bytes RedoBuffers7360512bytes Databasemounted. DatabaSEOpened. sql> ##检验数据库 sql>SHOWPARAMETERDB_NAME NAMETYPEVALUE ----------------------------------------------------------------------------- db_namestringHMODB sql>SELECTOPEN_MODE,NAME,DATABASE_ROLEFROMV$DATABASE; OPEN_MODENAMEDATABASE_ROLE --------------------------------------------- READWRITEHMODBPRIMARY
再次手动切换步骤
由于安装数据库实例时，创建了多个控制文件，并且这些控制文件都保存在不同的磁盘上，在主备进行切换的时，位于非共享存储的控制文件需要手动进行同步。这是由于在关闭数据库实例时Oracle同时将当前数据库状态信息记录到控制文件中，在没有启动数据库实例的备机就无法同步自身的其他的控制文件。这时就需要使用共享存储里的控制文件进行恢复。
##DB2: [oracle@hmdg-db2~]$sqlplus/assysdba sql>shutdownimmediate [root@hmdg-db2~]#umount/oradata/
2、在DB1上进行恢复启动数据库实例
##DB1： [root@hmdg-db1~]#mount/dev/sdb1/oradata/ [oracle@hmdg-db1~]$exportORACLE_SID=HMODB [oracle@hmdg-db1~]$sqlpus/assysdba sql>startupnomount sql>alterdatabasemount; sql>alterdatabaSEOpen;
如果控制文件在不同的磁盘目录，那么就需要执行这一步恢复控制文件
[oracle@hmdg-db1~]$rmantarget/ RMAN>restorecontrolfilefrom'/oradata/HMODB/control01.ctl';
启动数据库监听
[oracle@hmdg-db1~]$lsnrctlstart
或者直接只有操作系统的cp、scp等命令进行拷贝恢复
[oracle@hmdg-db1~]$cp/oradata/HMODB/control01.ctl/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl sql>SELECTNAME,OPEN_MODE,DATABASE_ROLEFROMV$DATABASE; NAMEOPEN_MODENAMEDATABASE_ROLE ------------------------------------------------------ HMODBREADWRITEHMODBPRIMARY
主备服务SSH密钥配置
配置oracle用户的ssh密钥，用于主备通过oracle身份自动检测对方的数据共享存储状态。在keepalived脚本管理中自动检查对方的挂载状态，并在主备切换的过程中，待接管的服务器等待对方umount共享存储之后，正常启动数据库实例。（如果主服务异常关机或网络中断，不需要等待对方umount存储，主服务器异常关机时keepalived会即时检测到并自动切换到备机上）
[oracle@hmdg-db1~]$ssh-keygen-trsa-b2048 [oracle@hmdg-db1~]$ssh-copy-id-i~/.ssh/id_rsa.puboracle@hmdg-db2 [oracle@hmdg-db2~]$ssh-keygen-trsa-b2048 [oracle@hmdg-db2~]$ssh-copy-id-i~/.ssh/id_rsa.puboracle@hmdg-db1
添加主备防火墙互通规则
#iptables-IINPUT-ptcp-s172.16.10.0/24-jACCEPT #iptables-IINPUT-pudp-s172.16.10.0/24-jACCEPT
Keepalived配置和管理脚本
keepalived配置
主服务器的keepalived配置，备用服务器需要做相应的修改
!ConfigurationFileforkeepalived
global_defs{
notification_email{
mail@huangming.org
}
notification_email_fromAlexandre.Cassen@firewall.loc
smtp_server127.0.0.1
smtp_connect_timeout30
router_idORACLE_HA_MASTER#backup为ORACLE_HA_BACKUP
!vrrp_skip_check_adv_addr
}
vrrp_sync_groupORACLE_HA_GROUP{
group{
Oracle_HA_1#定义一个vrrp实例组
}
}
vrrp_scriptmonitor_oracle_status{#oracle数据库服务器状态检查脚本
script"/usr/local/keepalived/scripts/monitor.sh"
interval10#脚本执行时间间隔
fall3#脚本执行的最大失败次数
rise1#脚本执行成功一次则返回正常状态
weight20#脚本执行成功时的附加权值
}
vrrp_scriptchange_monitor_status{#服务器状态监控并修改主备检查脚本
script"/usr/local/keepalived/scripts/change_monitor_status.sh"
interval2
}
vrrp_instanceOracle_HA_1{
stateBACKUP#主备都设置为BACKUP
interfaceeth1#绑定vip的网络接口
virtual_router_id200#主备保存一致
priority100#备为90
advert_int2#主备状态同步时间间隔
nopreempt#主设置为不抢占模式
authentication{
auth_typePASS
auth_pass11112222
}
virtual_ipaddress{
172.16.10.130/24deveth1#需要绑定的VIP
172.16.10.131/24deveth1
}
track_script{
monitor_oracle_status
change_monitor_status
}
track_interface{
eth0#同时监控检测其他的网接口
}
notify_master"/usr/local/keepalived/scripts/keepalived_notify.shmaster"
notify_backup"/usr/local/keepalived/scripts/keepalived_notify.shbackup"
notify_fault"/usr/local/keepalived/scripts/keepalived_notify.shfault"
notify_stop"/usr/local/keepalived/scripts/keepalived_notify.shstop"
!notify管理脚本的定义
!分别对应keepalived转换为master、backup、fault、stop状态时执行的脚本
!在oracle的双机管理中主要是依赖这些脚本实现资源的接管
}
keepalived监控脚本
[root@hmdg-db1scripts]#ls-l/usr/local/keepalived/scripts/ -rwxr-xr-x1rootroot818Apr1420:13change_monitor_status.sh -rw-r--r--1rootroot57Apr1420:13Controlfile.sql -rw-r--r--1rootroot1308Apr1420:13keepalived -rwxr-xr-x1rootroot10549Apr1423:18keepalived_notify.sh -rwxr-xr-x1rootroot926Apr1423:12monitor_primary_oracle.sh -rwxr-xr-x1rootroot895Apr1516:46monitor.sh -rwxr-xr-x1rootroot895Apr1420:13monitor_standby_oracle.sh -rw-r--r--1rootroot1657Apr1515:13oracle_init.sh -rwxr-xr-x1rootroot8441Apr1420:13notify.sh_20180413
关于这些keepalived脚本，由于篇幅有限我没有全部贴到本文中，大家可以访问我的Github仓库找到：https://github.com/hmlinux/oracle-keepalived
l keepalived_notify.sh #keepalived的notify脚本，用于实现主备的资源自动切换
l monitor.sh #keepalved的监控脚本，监控oracle数据库运行状态、共享存储状态
l change_monitor_status.sh #异常或正常检测脚本，当备服务器处于正常运行状态，并且处于keepalived的BACKUP状态时，将自身的监控脚本修改为在备机就绪时执行都返回正常结果，主要作用时让正常的备机可以随时接管主服务器的资源
l monitor_primary_oracle.sh #处于MASTER状态时的服务器监控脚本
l monitor_standby_oracle.sh #处于BACKUP状态时的服务器监控脚本
l oracle_init.sh #notify脚本调用oracle启动、停止、状态检测命令
#!/bin/bash
#author:hm@huangming.org
#keepalivednotifyscript
#bindvipinterface
interface="eth1"
#keepalivedvirtual_ipaddress
virtual_ipaddress="172.16.10.130","172.16.10.131"
MASTER_HOSTNAME="hmdg-db1"#DB1
BACKUP_HOSTNAME="hmdg-db2"#DB2
LOGDIR="/usr/local/keepalived/log"
LOGFILE="$LOGDIR/keepalived_haswitch.log"
TMPLOG=/tmp/notify_.log
sharedisk="/dev/sdb1"
sharedisk_mount_point="/oradata"
#oracle_datadir="/oradata/path/HMODB"
oracle_datadir="/oradata"
#Sourceoracledatabaseinstancestartup/shutdownscript
./usr/local/keepalived/scripts/oracle_init.sh
control_files="/$oracle_datadir/$ORACLE_SID/control01.ctl","/u01/app/oracle/flash_recovery_area/$ORACLE_SID/control02.ctl","/home/oracle/rman/$ORACLE_SID/control03.ctl"
RETVAL=0
OLD_IFS="$IFS"
IFS=","
[-d$LOGDIR]||mkdir$LOGDIR
controlfile_backpath="/backup/oracle/control"
controlfile_back=$controlfile_backpath/control_$(date'+%Y%d%m%H%M%S')
[-d$controlfile_backpath]||mkdir-p$controlfile_backpath&&chown-Roracle:oinstall$controlfile_backpath
info_log(){
printf"$(date'+%b%d%T%a')$HOSTNAME[keepalived_notify]:$1"
}
control01_ctl=`printf${control_files[0]}`
backup_controlfile(){
su-oracle<<EOF
exportORACLE_SID=$ORACLE_SID
$ORACLE_HOME/bin/sqlplus-S"/assysdba"
alterdatabasebackupcontrolfileto'$controlfile_back';
exit
EOF
}
#runuser-loracle-c"exportORACLE_SID=$ORACLE_SID;rmantarget/cmdfile=/usr/scripts/ControlfileRestore.sql"
restore_controlfile(){
su-oracle<<EOF
exportORACLE_SID=$ORACLE_SID
$ORACLE_HOME/bin/rmantarget/nocatalog
RESTORECONTROLFILEFROM'$control01_ctl';
exit
EOF
}
ssh_p=$(netstat-ntp|awk'/sshd/{print$4}'|awk-F':''{print$2}'|head-1)
chk_remote_node_sharedisk(){
if[$(hostname)==$MASTER_HOSTNAME];then
ifping-c1-w2$BACKUP_HOSTNAME&>/dev/null;then
runuser-loracle-c"ssh-p$ssh_p$BACKUP_HOSTNAME2>/dev/nulldf|grep$sharedisk|wc-l"
fi
fi
if[$(hostname)==$BACKUP_HOSTNAME];then
ifping-c1-w2$MASTER_HOSTNAME&>/dev/null;then
runuser-loracle-c"ssh-p$ssh_p$MASTER_HOSTNAME2>/dev/nulldf|grep$sharedisk|wc-l"
fi
fi
}
master(){
info_log"DatabaseSwitchoverToMASTER\n"
info_log"Checkremotenodesharediskmounted.\n"
i=1
while(($i<=30))
do
chk_status=$(chk_remote_node_sharedisk)
if[$chk_status-ge1];then
info_log"$sharediskisalreadymountedonremotenodeorbusy.checking[$i]...\n"
else
info_log"$sharediskcheckpassed.\n"
break
fi
if[$i-eq20];then
info_log"Diskstatusabnormal.\n"
exit1
fi
sleep1
i=$(($i+1))
done
ismount=$(df-h|grep$sharedisk|grep$sharedisk_mount_point|wc-l)
if[$ismount-eq0];then
info_log"mount$sharediskon$sharedisk_mount_point\n"
mount$sharedisk$sharedisk_mount_point
RETVAL=$?
if[$RETVAL-eq0];then
#shutdown_instance
info_log"restorecontrolfile1\n"
startup_nomount
restore_controlfile
else
info_log"Error:$sharediskcannotmountor$sharedisk_mount_pointbusy\n"
exit$RETVAL
fi
else
disk=$(df-h|grep$sharedisk_mount_point|awk'{print$1}')
if[$disk==$sharedisk];then
info_log"mount:$sharediskisalreadymountedon$sharedisk_mount_point\n"
else
info_log"Warning:$sharediskalreadymountedon$disk\n"
fi
fi
status=$(check_instance_status|grep-Eio-e"\bOPEN\b"-e"\bMOUNTED\b"-e"\bSTARTED\b")
if["$status"=="OPEN"];then
info_log"adatabasealreadyopenbytheinstance.\n"
elif["$status"=="MOUNTED"];then
info_log"re-opendatabaseinstance\n"
open_instance|tee$TMPLOG
opened=$(cat$TMPLOG|grep-Eio"\bDatabasealtered\b")
if["$opened"!="Databasealtered"];then
info_log"Error:databaseinstanceopenfail!\n"
exit2
fi
elif["$status"=="STARTED"];then
info_log"alterdatabasetomount\n"
mount_instance|tee$TMPLOG
mounted=$(cat$TMPLOG|grep-Eio"\bDatabasealtered\b")
if["$mounted"=="Databasealtered"];then
info_log"alterdatabasetoopen\n"
open_instance|tee$TMPLOG
opened=$(cat$TMPLOG|grep-Eio"\bDatabasealtered\b")
if["$opened"=="Databasealtered"];then
info_log"DatabaSEOpened.\n"
else
info_log"DatabaSEOpenFailed\n"
exit4
fi
else
info_log"DatabasemountFailed\n"
exit3
fi
else
info_log"Startupdatabaseandopeninstance\n"
shutdown_instance&>/dev/null
startup_instance|tee$TMPLOG
started=$(cat$TMPLOG|grep-Eio"\bDatabaSEOpened\b")
if["$started"!="DatabaSEOpened"];then
info_log"Databaseinstanceopenfail.\n"
info_log"restorecontrolfile2\n"
shutdown_instance|tee$TMPLOG
startup_nomount|tee$TMPLOG
restore_controlfile
shutdown_instance|tee$TMPLOG
startup_instance|tee$TMPLOG
started=$(cat$TMPLOG|grep-Eio"\bDatabaSEOpened\b")
if["$started"!="DatabaSEOpened"];then
info_log"Databaserestorefail!\n"
exit5
else
info_log"DatabaSEOpened.\n"
fi
else
info_log"DatabaSEOpened.\n"
fi
fi
info_log"Startuplistener\n"
runuser-loracle-c"lsnrctlstatus&>/dev/null"
if[$?-eq0];then
info_log"listeneralreadystarted.\n"
else
info_log"startinglistener...\n"
runuser-loracle-c"lsnrctlstart&>/dev/null"
if[$?-eq0];then
info_log"Thelistenerstartupsuccessfully\n"
else
info_log"Listenerstartfailure!\n"
fi
fi
echo
}
backup(){
info_log"DatabaseSwitchoverToBACKUP\n"
ismount=$(df-h|grep$sharedisk|grep$sharedisk_mount_point|wc-l)
if[$ismount-ge1];then
disk=$(df-h|grep$sharedisk_mount_point|awk'{print$1}')
if[$disk==$sharedisk];then
status=$(check_instance_status|grep-Eio-e"\bOPEN\b"-e"\bMOUNTED\b"-e"\bSTARTED\b")
if["$status"=="OPEN"-o"$status"=="MOUNTED"];then
info_log"Databaseinstancestateismounted\n"
info_log"Backupcurrentcontrolfile.\n"
echo-e"\nsql>alterdatabasebackupcontrolfileto'$controlfile_back';\n"
backup_controlfile
info_log"Shutdowndatabaseinstance,pleasewait...\n"
shutdown_instance|tee$TMPLOG
shuted=$(cat$TMPLOG|grep-Eio"\binstanceshutdown\b")
if["$shuted"=="instanceshutdown"];then
info_log"Databaseinstanceshutdownsuccessfully.\n"
else
info_log"DatabaseinstanceshutdownFailed.\n"
info_log"shutdownabort.\n"
shutdown_abort
fi
elif["$status"=="STARTED"];then
info_log"DatabaseinstancestateisSTARTED\n"
info_log"Shutdowndatabaseinstance,pleasewait...\n"
shutdown_instance|tee$TMPLOG
shuted=$(cat$TMPLOG|grep-Eio"\binstanceshutdown\b")
if["$shuted"=="instanceshutdown"];then
info_log"Databaseinstanceshutdownsuccessfully.\n"
else
info_log"DatabaseinstanceshutdownFailed.\n"
info_log"shutdownabort.\n"
shutdown_abort
fi
else
shutdown_instance|tee$TMPLOG
info_log"Databaseinstancenotavailable.\n"
fi
echo
info_log"umountsharedisk\n"
echo
umount$sharedisk_mount_point&&RETVAL=$?
if[$RETVAL-eq0];then
info_log"umount$sharedisk_mount_pointsuccess.\n"
else
info_log"umount$sharedisk_mount_pointfail!\n"
fi
else
info_log"$sharediskisnotmounton$sharedisk_mount_pointorbusy.\n"
fi
else
info_log"$sharedisk_mount_pointisnomount\n"
fi
info_log"stoppinglistener...\n"
runuser-loracle-c"lsnrctlstatus"&>/dev/null
RETVAL=$?
if[$RETVAL-eq0];then
runuser-loracle-c"lsnrctlstop"&>/dev/null
RETVAL=$?
if[$RETVAL-eq0];then
info_log"Thelistenerstopsuccessfully\n"
else
info_log"Listenerstopfailure!\n"
fi
else
info_log"listenerisnotstarted.\n"
fi
echo
}
notify_master(){
echo-e"\n-------------------------------------------------------------------------------"
echo"`date'+%b%d%T%a'`$(hostname)[keepalived_notify]:Transitionto$1STATE";
echo"`date'+%b%d%T%a'`$(hostname)[keepalived_notify]:SetuptheVIPon$interface$virtual_ipaddress";
}
notify_backup(){
echo-e"\n-------------------------------------------------------------------------------"
echo"`date'+%b%d%T%a'`$HOSTNAME[keepalived_notify]:Transitionto$1STATE";
echo"`date'+%b%d%T%a'`$HOSTNAME[keepalived_notify]:removingtheVIPon$interfacefor$virtual_ipaddress";
}
case$1in
master)
notify_masterMASTER|tee-a$LOGFILE
master|tee-a$LOGFILE
;;
backup)
notify_backupBACKUP|tee-a$LOGFILE
backup|tee-a$LOGFILE
;;
fault)
notify_backupFAULT|tee-a$LOGFILE
backup|tee-a$LOGFILE
;;
stop)
notify_backupSTOP|tee-a$LOGFILE
/etc/init.d/keepalivedstart
#sleep6&&backup|tee-a$LOGFILE
;;
*)
echo"Usage:`basename$0`{master|backup|fault|stop}"
RETVAL=1
;;
esac
exit$RETVAL
手动执行notify切换脚本
l 将主服务器切换为BACKUP状态
执行./keepalived_notify.sh backup
[root@hmdg-db1scripts]#./keepalived_notify.shbackup ------------------------------------------------------------------------------- Apr1517:23:05Sunhmdg-db1[keepalived_notify]:TransitiontoBACKUPSTATE Apr1517:23:05Sunhmdg-db1[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131 Apr1517:23:05Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToBACKUP Apr1517:23:05Sunhmdg-db1[keepalived_notify]:/oradataisnomount Apr1517:23:05Sunhmdg-db1[keepalived_notify]:stoppinglistener... Apr1517:23:05Sunhmdg-db1[keepalived_notify]:listenerisnotstarted.
观察执行脚本的输出结果，这是由于在本机上并没有启动oracle实例，所以脚本最终提示本机并没有启动oracle数据库实例
l 将主服务器切换为MASTER状态
执行./keepalived_notify.sh master
[root@hmdg-db1scripts]#./keepalived_notify.shmaster ------------------------------------------------------------------------------- Apr1517:23:43Sunhmdg-db1[keepalived_notify]:TransitiontoMASTERSTATE Apr1517:23:43Sunhmdg-db1[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131 Apr1517:23:43Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToMASTER Apr1517:23:43Sunhmdg-db1[keepalived_notify]:Checkremotenodesharediskmounted. Apr1517:23:43Sunhmdg-db1[keepalived_notify]:/dev/sdb1checkpassed. Apr1517:23:43Sunhmdg-db1[keepalived_notify]:mount/dev/sdb1on/oradata Apr1517:23:43Sunhmdg-db1[keepalived_notify]:restorecontrolfile1 ORACLEinstancestarted. TotalSystemGlobalArea1603411968bytes FixedSize2213776bytes VariableSize1056966768bytes DatabaseBuffers536870912bytes RedoBuffers7360512bytes RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1517:23:442018 Copyright(c)1982,2009,Oracleand/oritsaffiliates.Allrightsreserved. connectedtotargetdatabase:HMODB(notmounted) usingtargetdatabasecontrolfileinsteadofrecoverycatalog RMAN> Startingrestoreat15-APR-18 allocatedchannel:ORA_DISK_1 channelORA_DISK_1:SID=572devicetype=DISK channelORA_DISK_1:copiedcontrolfilecopy outputfilename=/oradata/HMODB/control01.ctl outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl outputfilename=/home/oracle/rman/HMODB/control03.ctl Finishedrestoreat15-APR-18 RMAN> RecoveryManagercomplete. Apr1517:23:46Sunhmdg-db1[keepalived_notify]:alterdatabasetomount Databasealtered. Apr1517:23:50Sunhmdg-db1[keepalived_notify]:alterdatabasetoopen Databasealtered. Apr1517:23:57Sunhmdg-db1[keepalived_notify]:DatabaSEOpened. Apr1517:23:57Sunhmdg-db1[keepalived_notify]:Startuplistener Apr1517:23:57Sunhmdg-db1[keepalived_notify]:startinglistener... Apr1517:23:57Sunhmdg-db1[keepalived_notify]:Thelistenerstartupsuccessfully
观察执行脚本的输出结果
1、脚本接收转换到MASTER信息
2、首先提示绑定的VIP
3、然后检测远程备用节点的磁盘是否挂载，或本机是否挂载了磁盘，检测成功则把磁盘挂载
5、启动数据库实例到open状态
6、启动数据库监听，切换完成
注意：如果使用系统命令手动切换，应该要保证正在运行oracle服务的主节点先正常关闭，并且卸载共享存储之后，再启动备用节点的oracle服务(即不能同时挂载共享存储)
l 再次将主服务器切换到BACKUP状态
执行./keepalived_notify.sh backup
[root@hmdg-db1scripts]#./keepalived_notify.shbackup ------------------------------------------------------------------------------- Apr1517:34:54Sunhmdg-db1[keepalived_notify]:TransitiontoBACKUPSTATE Apr1517:34:54Sunhmdg-db1[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131 Apr1517:34:54Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToBACKUP Apr1517:34:55Sunhmdg-db1[keepalived_notify]:Databaseinstancestateismounted Apr1517:34:55Sunhmdg-db1[keepalived_notify]:Backupcurrentcontrolfile. sql>alterdatabasebackupcontrolfileto'/backup/oracle/control/control_20181504173454'; Databasealtered. Apr1517:34:55Sunhmdg-db1[keepalived_notify]:Shutdowndatabaseinstance,pleasewait... Databaseclosed. Databasedismounted. ORACLEinstanceshutdown. Apr1517:35:04Sunhmdg-db1[keepalived_notify]:Databaseinstanceshutdownsuccessfully. Apr1517:35:04Sunhmdg-db1[keepalived_notify]:umountsharedisk Apr1517:35:05Sunhmdg-db1[keepalived_notify]:umount/oradatasuccess. Apr1517:35:05Sunhmdg-db1[keepalived_notify]:stoppinglistener... Apr1517:35:08Sunhmdg-db1[keepalived_notify]:Thelistenerstopsuccessfully
观察执行脚本的输出结果
1、 脚本接收转换到BACKUP信息
2、 首先移除VIP(由keepalived绑定到主机上浮动IP地址，同时也是对外提供服务的IP地址)，实际上是由keepalived检测到异常或本身的服务器不可用时自动剔除VIP
3、 备份控制文件到/backup/oracle/control/目录下
5、 卸载共享存储
l notify脚本切换演示
如果备机主动切换到MASTER状态时，脚本首先或检测对方磁盘是否umount，如果没有则最多等待20秒，提示异常并正常推出切换到MASTER的请求
[root@hmdg-db1scripts]#./keepalived_notify.shmaster ------------------------------------------------------------------------------- Apr1517:40:08Sunhmdg-db1[keepalived_notify]:TransitiontoMASTERSTATE Apr1517:40:08Sunhmdg-db1[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131 Apr1517:40:08Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToMASTER Apr1517:40:08Sunhmdg-db1[keepalived_notify]:Checkremotenodesharediskmounted. Apr1517:40:08Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]... Apr1517:40:09Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]... Apr1517:40:10Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]... Apr1517:40:12Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[4]... Apr1517:40:13Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[5]... Apr1517:40:14Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[6]... Apr1517:40:15Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[7]... Apr1517:40:16Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[8]... Apr1517:40:17Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[9]... Apr1517:40:18Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[10]... Apr1517:40:20Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[11]... Apr1517:40:21Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[12]... Apr1517:40:22Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[13]... Apr1517:40:23Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[14]... Apr1517:40:24Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[15]... Apr1517:40:25Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[16]... Apr1517:40:26Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[17]... Apr1517:40:27Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[18]... Apr1517:40:29Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[19]... Apr1517:40:30Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[20]... Apr1517:40:30Sunhmdg-db1[keepalived_notify]:Diskstatusabnormal.
这时如果需要手动切换，则需要先将远程备用节点的服务关闭并umount存储之后，才能正常切换
[root@hmdg-db1scripts]#./keepalived_notify.shmaster ------------------------------------------------------------------------------- Apr1517:45:29Sunhmdg-db1[keepalived_notify]:TransitiontoMASTERSTATE Apr1517:45:29Sunhmdg-db1[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131 Apr1517:45:29Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToMASTER Apr1517:45:29Sunhmdg-db1[keepalived_notify]:Checkremotenodesharediskmounted. Apr1517:45:29Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]... Apr1517:45:30Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]... Apr1517:45:31Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]... Apr1517:45:32Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[4]... Apr1517:45:33Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[5]... Apr1517:45:35Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[6]... Apr1517:45:36Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[7]... Apr1517:45:37Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[8]... Apr1517:45:38Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[9]... Apr1517:45:39Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[10]... Apr1517:45:40Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[11]... Apr1517:45:41Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[12]... Apr1517:45:42Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[13]... Apr1517:45:44Sunhmdg-db1[keepalived_notify]:/dev/sdb1checkpassed. Apr1517:45:44Sunhmdg-db1[keepalived_notify]:mount/dev/sdb1on/oradata Apr1517:45:44Sunhmdg-db1[keepalived_notify]:restorecontrolfile1 ORACLEinstancestarted. TotalSystemGlobalArea1603411968bytes FixedSize2213776bytes VariableSize1056966768bytes DatabaseBuffers536870912bytes RedoBuffers7360512bytes RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1517:45:452018 Copyright(c)1982,Oracleand/oritsaffiliates.Allrightsreserved. connectedtotargetdatabase:HMODB(notmounted) usingtargetdatabasecontrolfileinsteadofrecoverycatalog RMAN> Startingrestoreat15-APR-18 allocatedchannel:ORA_DISK_1 channelORA_DISK_1:SID=572devicetype=DISK channelORA_DISK_1:copiedcontrolfilecopy outputfilename=/oradata/HMODB/control01.ctl outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl outputfilename=/home/oracle/rman/HMODB/control03.ctl Finishedrestoreat15-APR-18 RMAN> RecoveryManagercomplete. Apr1517:45:46Sunhmdg-db1[keepalived_notify]:alterdatabasetomount Databasealtered. Apr1517:45:51Sunhmdg-db1[keepalived_notify]:alterdatabasetoopen Databasealtered. Apr1517:45:58Sunhmdg-db1[keepalived_notify]:DatabaSEOpened. Apr1517:45:58Sunhmdg-db1[keepalived_notify]:Startuplistener Apr1517:45:58Sunhmdg-db1[keepalived_notify]:startinglistener... Apr1517:45:58Sunhmdg-db1[keepalived_notify]:Thelistenerstartupsuccessfully
启动keepalived服务接管Oracle服务
启动keepalived服务时，keepalived首先进行资源检测，并根据权值设定MASTER角色
keepalived接管的资源：Oracle实例、存储、VIP
1、启动主服务器DB1的keepalived
##启动keepalived [root@hmdg-db1scripts]#/etc/init.d/keepalivedstart ##设置开机启动 [root@hmdg-db1scripts]#chkconfigkeepalivedon
2、查看主服务器的keepalived系统日志
[root@hmdg-db1scripts]#tail-f/var/log/messages Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:RegisteringKernelnetlinkreflector Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:RegisteringKernelnetlinkcommandchannel Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:RegisteringgratuitousARPsharedchannel Apr1517:51:25hmdg-db1Keepalived_healthcheckers[82071]:UsingLinkWatchkernelnetlinkreflector... Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'. Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs. Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:SECURITYVIOLATION-scriptsarebeingexecutedbutscript_securitynotenabled. Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:SyncgroupORACLE_HA_GROUPhasonly1virtualrouter(s)-removing Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:UsingLinkWatchkernelnetlinkreflector... Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRPsockpool:[ifindex(3),proto(112),unicast(0),fd(10,11)] Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRP_Script(change_monitor_status)succeeded Apr1517:51:25hmdg-db1Keepalived_vrrp[82072]:VRRP_Script(monitor_oracle_status)succeeded Apr1517:51:27hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom100to120 Apr1517:51:31hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)TransitiontoMASTERSTATE Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)EnteringMASTERSTATE Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)settingprotocolVIPs. Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.130 Apr1517:51:33hmdg-db1Keepalived_healthcheckers[82071]:NetlinkreflectorreportsIP172.16.10.130added Apr1517:51:33hmdg-db1Keepalived_healthcheckers[82071]:NetlinkreflectorreportsIP172.16.10.131added Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130 Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.131 Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131 Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.130 Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.131 Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.130 Apr1517:51:33hmdg-db1Keepalived_vrrp[82072]:SendinggratuitousARPoneth1for172.16.10.131
3、启动备服务器DB2的keepalived
##启动keepalived [root@hmdg-db2scripts]#/etc/init.d/keepalivedstart ##设置开机启动 [root@hmdg-db2scripts]#chkconfigkeepalivedon
4、查看备服务器的keepalived系统日志
Apr1517:51:41hmdg-db2Keepalived[71709]:StartingKeepalivedv1.3.2(04/14,2018) Apr1517:51:41hmdg-db2Keepalived[71709]:WARNING-defaultuser'keepalived_script'forscriptexecutiondoesnotexist-pleasecreate. Apr1517:51:41hmdg-db2Keepalived[71709]:openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'. Apr1517:51:41hmdg-db2Keepalived[71710]:StartingHealthcheckchildprocess,pid=71711 Apr1517:51:41hmdg-db2Keepalived[71710]:StartingVRRPchildprocess,pid=71712 Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIP192.168.6.26added Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIP172.16.10.26added Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d175added Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIP192.168.6.26added Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d17fadded Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d189added Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIP172.16.10.26added Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d193added Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:RegisteringKernelnetlinkreflector Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d175added Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d17fadded Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d189added Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:NetlinkreflectorreportsIPfe80::20c:29ff:fedd:d193added Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:RegisteringKernelnetlinkreflector Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:RegisteringKernelnetlinkcommandchannel Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:RegisteringgratuitousARPsharedchannel Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'. Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs. Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:SECURITYVIOLATION-scriptsarebeingexecutedbutscript_securitynotenabled. Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:SyncgroupORACLE_HA_GROUPhasonly1virtualrouter(s)-removing Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:RegisteringKernelnetlinkcommandchannel Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'. Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:UsingLinkWatchkernelnetlinkreflector... Apr1517:51:41hmdg-db2Keepalived_healthcheckers[71711]:UsingLinkWatchkernelnetlinkreflector... Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:VRRPsockpool:[ifindex(3),11)] Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:pid71721exitedwithstatus1 Apr1517:51:41hmdg-db2Keepalived_vrrp[71712]:VRRP_Script(change_monitor_status)succeeded Apr1517:52:01hmdg-db2Keepalived_vrrp[71712]:VRRP_Script(monitor_oracle_status)succeeded Apr1517:52:01hmdg-db2Keepalived_vrrp[71712]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom90to110
5、主备Oracle、VIP、磁盘服务状态检查
##DB1 [root@hmdg-db1scripts]#ps-ef|grepora_ oracle824391017:51?00:00:00ora_pmon_HMODB oracle824411017:51?00:00:00ora_vktm_HMODB oracle824451017:51?00:00:00ora_gen0_HMODB oracle824471017:51?00:00:00ora_diag_HMODB oracle824491017:51?00:00:00ora_dbrm_HMODB oracle824511017:51?00:00:00ora_psp0_HMODB oracle824531017:51?00:00:00ora_dia0_HMODB oracle824551017:51?00:00:00ora_mman_HMODB oracle824571017:51?00:00:00ora_dbw0_HMODB oracle824591017:51?00:00:00ora_lgwr_HMODB oracle824611017:51?00:00:00ora_ckpt_HMODB oracle824631017:51?00:00:00ora_smon_HMODB oracle824651017:51?00:00:00ora_reco_HMODB oracle824671017:51?00:00:00ora_mmon_HMODB oracle824691017:51?00:00:00ora_mmnl_HMODB oracle824711017:51?00:00:00ora_d000_HMODB oracle824731017:51?00:00:00ora_s000_HMODB oracle826381017:51?00:00:00ora_arc0_HMODB oracle826531017:51?00:00:00ora_arc1_HMODB oracle826551017:51?00:00:00ora_arc2_HMODB oracle826571017:51?00:00:00ora_arc3_HMODB oracle827551017:51?00:00:00ora_qmnc_HMODB oracle828071017:51?00:00:00ora_cjq0_HMODB oracle828351017:51?00:00:00ora_vkrm_HMODB oracle828631017:51?00:00:00ora_q000_HMODB oracle828651017:51?00:00:00ora_q001_HMODB oracle840991117:54?00:00:00ora_j000_HMODB oracle841011017:54?00:00:00ora_j001_HMODB root842683823017:54pts/200:00:00grepora_ [root@hmdg-db1scripts]#df-Th FilesystemTypeSizeUsedAvailUse%Mountedon /dev/sda3ext450G19G29G40%/ tmpfstmpfs3.9G909M3.0G24%/dev/shm /dev/sda1ext4190M40M141M22%/boot /dev/sdb1ext430G6.0G22G22%/oradata [root@hmdg-db1scripts]#ipaddrshowdeveth1 3:eth1:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000 link/ether00:0c:29:59:d5:91brdff:ff:ff:ff:ff:ff inet172.16.10.25/24brd172.16.10.255scopeglobaleth1 inet172.16.10.130/24scopeglobalsecondaryeth1 inet172.16.10.131/24scopeglobalsecondaryeth1 inet6fe80::20c:29ff:fe59:d591/64scopelink valid_lftforeverpreferred_lftforever ##DB2 [root@hmdg-db2scripts]#ps-ef|grepora_ root7325131390017:55pts/200:00:00grepora_ [root@hmdg-db2scripts]#df-Th FilesystemTypeSizeUsedAvailUse%Mountedon /dev/sda3ext450G18G30G37%/ tmpfstmpfs3.9G03.9G0%/dev/shm /dev/sda1ext4190M40M141M22%/boot [root@hmdg-db2scripts]#ipaddrshowdeveth1 3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000 link/ether00:0c:29:dd:d1:7fbrdff:ff:ff:ff:ff:ff inet172.16.10.26/24brd172.16.10.255scopeglobaleth1 inet6fe80::20c:29ff:fedd:d17f/64scopelink valid_lftforeverpreferred_lftforever
测试Keepalived故障切换
@H_677_404@当主服务故障时的切换状态
停止或重启主服务器的keepalibed服务
主服务器DB1
##系统日志 Apr1518:08:02hmdg-db1Keepalived[89311]:Stopping Apr1518:08:02hmdg-db1Keepalived_healthcheckers[89313]:Stopped Apr1518:08:02hmdg-db1Keepalived_vrrp[89314]:VRRP_Instance(Oracle_HA_1)sent0priority Apr1518:08:02hmdg-db1Keepalived_vrrp[89314]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs. Apr1518:08:03hmdg-db1Keepalived_vrrp[89314]:Stopped Apr1518:08:03hmdg-db1Keepalived[89311]:StoppedKeepalivedv1.3.2(04/14,2018) ##nitify脚本的输出日志 ------------------------------------------------------------------------------- Apr1518:08:03Sunhmdg-db1[keepalived_notify]:TransitiontoSTOPSTATE Apr1518:08:03Sunhmdg-db1[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131 Apr1518:08:09Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToBACKUP Apr1518:08:09Sunhmdg-db1[keepalived_notify]:Databaseinstancestateismounted Apr1518:08:09Sunhmdg-db1[keepalived_notify]:Backupcurrentcontrolfile. sql>alterdatabasebackupcontrolfileto'/backup/oracle/control/control_20181504180803'; Databasealtered. Apr1518:08:09Sunhmdg-db1[keepalived_notify]:Shutdowndatabaseinstance,pleasewait... Databaseclosed. Databasedismounted. ORACLEinstanceshutdown. Apr1518:08:18Sunhmdg-db1[keepalived_notify]:Databaseinstanceshutdownsuccessfully. Apr1518:08:18Sunhmdg-db1[keepalived_notify]:umountsharedisk Apr1518:08:18Sunhmdg-db1[keepalived_notify]:umount/oradatasuccess. Apr1518:08:18Sunhmdg-db1[keepalived_notify]:stoppinglistener... Apr1518:08:19Sunhmdg-db1[keepalived_notify]:Thelistenerstopsuccessfully ##服务资源检查 [root@hmdg-db1scripts]#ps-ef|grepora_ root907582451018:09pts/000:00:00grepora_ [root@hmdg-db1scripts]#df-Th FilesystemTypeSizeUsedAvailUse%Mountedon /dev/sda3ext450G19G29G40%/ tmpfstmpfs3.9G03.9G0%/dev/shm /dev/sda1ext4190M40M141M22%/boot [root@hmdg-db1scripts]#ipaddrshowdeveth1 3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000 link/ether00:0c:29:59:d5:91brdff:ff:ff:ff:ff:ff inet172.16.10.25/24brd172.16.10.255scopeglobaleth1 inet6fe80::20c:29ff:fe59:d591/64scopelink valid_lftforeverpreferred_lftforever
由于主服务的keepalived异常，keepalived实例组将会重新分配MASTER角色，并将资源转移到BACKUP上(即重新选举后新的MASTER)
备服务器DB2
##系统日志 Apr1518:08:03hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)TransitiontoMASTERSTATE Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)EnteringMASTERSTATE Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)settingprotocolVIPs. Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130 Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131 Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:05hmdg-db2Keepalived_healthcheckers[77810]:NetlinkreflectorreportsIP172.16.10.130added Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:08:05hmdg-db2Keepalived_healthcheckers[77810]:NetlinkreflectorreportsIP172.16.10.131added Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:05hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:08:10hmdg-db2Keepalived_vrrp[77811]:SendinggratuitousARPoneth1for172.16.10.131 ##notify脚本日志 ------------------------------------------------------------------------------- Apr1518:08:05Sunhmdg-db2[keepalived_notify]:TransitiontoMASTERSTATE Apr1518:08:05Sunhmdg-db2[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131 Apr1518:08:05Sunhmdg-db2[keepalived_notify]:DatabaseSwitchoverToMASTER Apr1518:08:05Sunhmdg-db2[keepalived_notify]:Checkremotenodesharediskmounted. Apr1518:08:05Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]... Apr1518:08:06Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]... Apr1518:08:08Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]... Apr1518:08:09Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[4]... Apr1518:08:10Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[5]... Apr1518:08:11Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[6]... Apr1518:08:12Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[7]... Apr1518:08:13Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[8]... Apr1518:08:14Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[9]... Apr1518:08:15Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[10]... Apr1518:08:17Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[11]... Apr1518:08:18Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[12]... Apr1518:08:19Sunhmdg-db2[keepalived_notify]:/dev/sdb1checkpassed. Apr1518:08:19Sunhmdg-db2[keepalived_notify]:mount/dev/sdb1on/oradata Apr1518:08:19Sunhmdg-db2[keepalived_notify]:restorecontrolfile1 ORACLEinstancestarted. TotalSystemGlobalArea1603411968bytes FixedSize2213776bytes VariableSize1056966768bytes DatabaseBuffers536870912bytes RedoBuffers7360512bytes RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1518:08:202018 Copyright(c)1982,Oracleand/oritsaffiliates.Allrightsreserved. connectedtotargetdatabase:HMODB(notmounted) usingtargetdatabasecontrolfileinsteadofrecoverycatalog RMAN> Startingrestoreat15-APR-18 allocatedchannel:ORA_DISK_1 channelORA_DISK_1:SID=572devicetype=DISK channelORA_DISK_1:copiedcontrolfilecopy outputfilename=/oradata/HMODB/control01.ctl outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl outputfilename=/home/oracle/rman/HMODB/control03.ctl Finishedrestoreat15-APR-18 RMAN> RecoveryManagercomplete. Apr1518:08:22Sunhmdg-db2[keepalived_notify]:alterdatabasetomount Databasealtered. Apr1518:08:26Sunhmdg-db2[keepalived_notify]:alterdatabasetoopen Databasealtered. Apr1518:08:33Sunhmdg-db2[keepalived_notify]:DatabaSEOpened. Apr1518:08:33Sunhmdg-db2[keepalived_notify]:Startuplistener Apr1518:08:33Sunhmdg-db2[keepalived_notify]:startinglistener... Apr1518:08:33Sunhmdg-db2[keepalived_notify]:Thelistenerstartupsuccessfully ##服务检查 [root@hmdg-db2scripts]#ps-ef|grepora_ oracle786321018:08?00:00:00ora_pmon_HMODB oracle786341018:08?00:00:00ora_vktm_HMODB oracle786381018:08?00:00:00ora_gen0_HMODB oracle786401018:08?00:00:00ora_diag_HMODB oracle786421018:08?00:00:00ora_dbrm_HMODB oracle786441018:08?00:00:00ora_psp0_HMODB oracle786461018:08?00:00:00ora_dia0_HMODB oracle786481018:08?00:00:00ora_mman_HMODB oracle786501018:08?00:00:00ora_dbw0_HMODB oracle786521018:08?00:00:00ora_lgwr_HMODB oracle786541018:08?00:00:00ora_ckpt_HMODB oracle786561018:08?00:00:00ora_smon_HMODB oracle786581018:08?00:00:00ora_reco_HMODB oracle786601018:08?00:00:00ora_mmon_HMODB oracle786621018:08?00:00:00ora_mmnl_HMODB oracle786641018:08?00:00:00ora_d000_HMODB oracle786661018:08?00:00:00ora_s000_HMODB oracle788441018:08?00:00:00ora_arc0_HMODB oracle788471018:08?00:00:00ora_arc1_HMODB oracle788491018:08?00:00:00ora_arc2_HMODB oracle788511018:08?00:00:00ora_arc3_HMODB oracle788791018:08?00:00:00ora_qmnc_HMODB oracle789681018:08?00:00:00ora_cjq0_HMODB oracle790311018:08?00:00:00ora_vkrm_HMODB oracle790591018:08?00:00:00ora_q000_HMODB oracle790611018:08?00:00:00ora_q001_HMODB oracle816521018:13?00:00:00ora_smco_HMODB oracle816671018:13?00:00:00ora_w000_HMODB root8180831390018:13pts/200:00:00grepora_ [root@hmdg-db2scripts]#df-Th FilesystemTypeSizeUsedAvailUse%Mountedon /dev/sda3ext450G18G30G37%/ tmpfstmpfs3.9G909M3.0G24%/dev/shm /dev/sda1ext4190M40M141M22%/boot /dev/sdb1ext430G6.0G22G22%/oradata [root@hmdg-db2scripts]#ipaddrshowdeveth1 3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000 link/ether00:0c:29:dd:d1:7fbrdff:ff:ff:ff:ff:ff inet172.16.10.26/24brd172.16.10.255scopeglobaleth1 inet172.16.10.130/24scopeglobalsecondaryeth1 inet172.16.10.131/24scopeglobalsecondaryeth1 inet6fe80::20c:29ff:fedd:d17f/64scopelink valid_lftforeverpreferred_lftforever
测试当主服务恢复后备服务器故障的切换状态
首先将DB1的Keepalived服务启动
Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:openingfile'/usr/local/keepalived/etc/keepalived/keepalived.conf'. Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs. Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:SECURITYVIOLATION-scriptsarebeingexecutedbutscript_securitynotenabled. Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:SyncgroupORACLE_HA_GROUPhasonly1virtualrouter(s)-removing Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:UsingLinkWatchkernelnetlinkreflector... Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRPsockpool:[ifindex(3),11)] Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:pid90794exitedwithstatus1 Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(change_monitor_status)succeeded Apr1518:15:34hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(monitor_oracle_status)succeeded Apr1518:15:34hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom100to120@H_677_404@
停止备服务器的keepalived服务
##系统日志 Apr1518:16:24hmdg-db2Keepalived_vrrp[83304]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE Apr1518:16:24hmdg-db2Keepalived_vrrp[83304]:VRRPsockpool:[ifindex(3),11)] Apr1518:16:24hmdg-db2Keepalived_vrrp[83304]:VRRP_Script(change_monitor_status)succeeded Apr1518:16:25hmdg-db2Keepalived_vrrp[83304]:VRRP_Script(monitor_oracle_status)succeeded Apr1518:16:26hmdg-db2Keepalived_vrrp[83304]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom90to110 ##notify日志 ------------------------------------------------------------------------------- Apr1518:16:24Sunhmdg-db2[keepalived_notify]:TransitiontoSTOPSTATE Apr1518:16:24Sunhmdg-db2[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131 ------------------------------------------------------------------------------- Apr1518:16:24Sunhmdg-db2[keepalived_notify]:TransitiontoBACKUPSTATE Apr1518:16:24Sunhmdg-db2[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131 Apr1518:16:24Sunhmdg-db2[keepalived_notify]:DatabaseSwitchoverToBACKUP Apr1518:16:24Sunhmdg-db2[keepalived_notify]:Databaseinstancestateismounted Apr1518:16:25Sunhmdg-db2[keepalived_notify]:Backupcurrentcontrolfile. sql>alterdatabasebackupcontrolfileto'/backup/oracle/control/control_20181504181624'; Databasealtered. Apr1518:16:25Sunhmdg-db2[keepalived_notify]:Shutdowndatabaseinstance,pleasewait... Databaseclosed. Databasedismounted. ORACLEinstanceshutdown. Apr1518:16:34Sunhmdg-db2[keepalived_notify]:Databaseinstanceshutdownsuccessfully. Apr1518:16:34Sunhmdg-db2[keepalived_notify]:umountsharedisk Apr1518:16:34Sunhmdg-db2[keepalived_notify]:umount/oradatasuccess. Apr1518:16:34Sunhmdg-db2[keepalived_notify]:stoppinglistener... Apr1518:16:38Sunhmdg-db2[keepalived_notify]:Thelistenerstopsuccessfully ##服务状态 [root@hmdg-db2scripts]#ps-ef|grepora_ root8461931390018:18pts/200:00:00grepora_ [root@hmdg-db2scripts]#df-Th FilesystemTypeSizeUsedAvailUse%Mountedon /dev/sda3ext450G18G30G37%/ tmpfstmpfs3.9G03.9G0%/dev/shm /dev/sda1ext4190M40M141M22%/boot [root@hmdg-db2scripts]#ipaddrshowdeveth1 3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000 link/ether00:0c:29:dd:d1:7fbrdff:ff:ff:ff:ff:ff inet172.16.10.26/24brd172.16.10.255scopeglobaleth1 inet6fe80::20c:29ff:fedd:d17f/64scopelink valid_lftforeverpreferred_lftforever
检查DB1是否转换为MASTER并接管服务资源
##DB1的系统日志 Apr1518:15:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(change_monitor_status)succeeded Apr1518:15:34hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(monitor_oracle_status)succeeded Apr1518:15:34hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom100to120 Apr1518:16:23hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)TransitiontoMASTERSTATE Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)EnteringMASTERSTATE Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)settingprotocolVIPs. Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:16:25hmdg-db1Keepalived_healthcheckers[90790]:NetlinkreflectorreportsIP172.16.10.130added Apr1518:16:25hmdg-db1Keepalived_healthcheckers[90790]:NetlinkreflectorreportsIP172.16.10.131added Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:16:25hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130 Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131 Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:16:30hmdg-db1Keepalived_vrrp[90792]:SendinggratuitousARPoneth1for172.16.10.131 ##DB1的notify脚本日志 ------------------------------------------------------------------------------- Apr1518:16:25Sunhmdg-db1[keepalived_notify]:TransitiontoMASTERSTATE Apr1518:16:25Sunhmdg-db1[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131 Apr1518:16:25Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToMASTER Apr1518:16:25Sunhmdg-db1[keepalived_notify]:Checkremotenodesharediskmounted. Apr1518:16:26Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]... Apr1518:16:27Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]... Apr1518:16:28Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]... Apr1518:16:29Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[4]... Apr1518:16:30Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[5]... Apr1518:16:31Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[6]... Apr1518:16:32Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[7]... Apr1518:16:34Sunhmdg-db1[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[8]... Apr1518:16:35Sunhmdg-db1[keepalived_notify]:/dev/sdb1checkpassed. Apr1518:16:35Sunhmdg-db1[keepalived_notify]:mount/dev/sdb1on/oradata Apr1518:16:35Sunhmdg-db1[keepalived_notify]:restorecontrolfile1 ORACLEinstancestarted. TotalSystemGlobalArea1603411968bytes FixedSize2213776bytes VariableSize1056966768bytes DatabaseBuffers536870912bytes RedoBuffers7360512bytes RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1518:16:382018 Copyright(c)1982,Oracleand/oritsaffiliates.Allrightsreserved. connectedtotargetdatabase:HMODB(notmounted) usingtargetdatabasecontrolfileinsteadofrecoverycatalog RMAN> Startingrestoreat15-APR-18 allocatedchannel:ORA_DISK_1 channelORA_DISK_1:SID=572devicetype=DISK channelORA_DISK_1:copiedcontrolfilecopy outputfilename=/oradata/HMODB/control01.ctl outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl outputfilename=/home/oracle/rman/HMODB/control03.ctl Finishedrestoreat15-APR-18 RMAN> RecoveryManagercomplete. Apr1518:16:40Sunhmdg-db1[keepalived_notify]:alterdatabasetomount Databasealtered. Apr1518:16:44Sunhmdg-db1[keepalived_notify]:alterdatabasetoopen Databasealtered. Apr1518:16:50Sunhmdg-db1[keepalived_notify]:DatabaSEOpened. Apr1518:16:50Sunhmdg-db1[keepalived_notify]:Startuplistener Apr1518:16:50Sunhmdg-db1[keepalived_notify]:startinglistener... Apr1518:16:50Sunhmdg-db1[keepalived_notify]:Thelistenerstartupsuccessfully ##服务状态 [root@hmdg-db1scripts]#ps-ef|grepora_ oracle917651018:16?00:00:00ora_pmon_HMODB oracle917671018:16?00:00:00ora_vktm_HMODB oracle917711018:16?00:00:00ora_gen0_HMODB oracle917731018:16?00:00:00ora_diag_HMODB oracle917751018:16?00:00:00ora_dbrm_HMODB oracle917771018:16?00:00:00ora_psp0_HMODB oracle917791018:16?00:00:00ora_dia0_HMODB oracle917811018:16?00:00:00ora_mman_HMODB oracle917831018:16?00:00:00ora_dbw0_HMODB oracle917851018:16?00:00:00ora_lgwr_HMODB oracle917871018:16?00:00:00ora_ckpt_HMODB oracle917891018:16?00:00:00ora_smon_HMODB oracle917911018:16?00:00:00ora_reco_HMODB oracle917931018:16?00:00:00ora_mmon_HMODB oracle917951018:16?00:00:00ora_mmnl_HMODB oracle917971018:16?00:00:00ora_d000_HMODB oracle917991018:16?00:00:00ora_s000_HMODB oracle919651018:16?00:00:00ora_arc0_HMODB oracle919671018:16?00:00:00ora_arc1_HMODB oracle919691018:16?00:00:00ora_arc2_HMODB oracle919711018:16?00:00:00ora_arc3_HMODB oracle919991018:16?00:00:00ora_qmnc_HMODB oracle920901018:16?00:00:00ora_cjq0_HMODB oracle921051018:16?00:00:00ora_vkrm_HMODB oracle921811018:16?00:00:00ora_q000_HMODB oracle921831018:16?00:00:00ora_q001_HMODB oracle938171018:20?00:00:00ora_j000_HMODB oracle938191018:20?00:00:00ora_j001_HMODB root940302451018:20pts/000:00:00grepora_ [root@hmdg-db1scripts]#df-Th FilesystemTypeSizeUsedAvailUse%Mountedon /dev/sda3ext450G19G29G40%/ tmpfstmpfs3.9G909M3.0G24%/dev/shm /dev/sda1ext4190M40M141M22%/boot /dev/sdb1ext430G6.0G22G22%/oradata [root@hmdg-db1scripts]#ipaddrshowdeveth1 3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000 link/ether00:0c:29:59:d5:91brdff:ff:ff:ff:ff:ff inet172.16.10.25/24brd172.16.10.255scopeglobaleth1 inet172.16.10.130/24scopeglobalsecondaryeth1 inet172.16.10.131/24scopeglobalsecondaryeth1 inet6fe80::20c:29ff:fe59:d591/64scopelink valid_lftforeverpreferred_lftforever
测试当主服务器Oracle服务关闭时
[oracle@hmdg-db1~]$exportORACLE_SID=HMODB [oracle@hmdg-db1~]$sqlplus/assysdba sql*Plus:Release11.2.0.1.0ProductiononSunApr1518:21:522018 Copyright(c)1982,Oracle.Allrightsreserved. Connectedto: OracleDatabase11gEnterpriseEditionRelease11.2.0.1.0-64bitProduction WiththePartitioning,OLAP,DataMiningandRealApplicationTestingoptions sql>SELECTOPEN_MODE,DATABASE_ROLEFROMV$DATABASE; OPEN_MODENAMEDATABASE_ROLE --------------------------------------------- READWRITEHMODBPRIMARY sql>shutdownimmediate Databaseclosed. Databasedismounted. ORACLEinstanceshutdown.
2、查看keepalived日志
## Apr1518:23:34hmdg-db1Keepalived_vrrp[90792]:pid96098exitedwithstatus2 Apr1518:23:48hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Receivedadvertwithhigherpriority110,ours100 Apr1518:23:48hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)EnteringBACKUPSTATE Apr1518:23:48hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)removingprotocolVIPs. Apr1518:23:48hmdg-db1Keepalived_healthcheckers[90790]:NetlinkreflectorreportsIP172.16.10.130removed Apr1518:23:48hmdg-db1Keepalived_healthcheckers[90790]:NetlinkreflectorreportsIP172.16.10.131removed Apr1518:23:54hmdg-db1Keepalived_vrrp[90792]:pid96486exitedwithstatus2 Apr1518:24:14hmdg-db1Keepalived_vrrp[90792]:VRRP_Script(monitor_oracle_status)succeeded Apr1518:24:15hmdg-db1Keepalived_vrrp[90792]:VRRP_Instance(Oracle_HA_1)Changingeffectivepriorityfrom100to120 ------------------------------------------------------------------------------- Apr1518:23:48Sunhmdg-db1[keepalived_notify]:TransitiontoBACKUPSTATE Apr1518:23:48Sunhmdg-db1[keepalived_notify]:removingtheVIPoneth1for172.16.10.130,172.16.10.131 Apr1518:23:48Sunhmdg-db1[keepalived_notify]:DatabaseSwitchoverToBACKUP Apr1518:23:49Sunhmdg-db1[keepalived_notify]:Databaseinstancestateismounted Apr1518:23:49Sunhmdg-db1[keepalived_notify]:Backupcurrentcontrolfile. sql>alterdatabasebackupcontrolfileto'/backup/oracle/control/control_20181504182348'; Databasealtered. Apr1518:23:49Sunhmdg-db1[keepalived_notify]:Shutdowndatabaseinstance,pleasewait... Databaseclosed. Databasedismounted. ORACLEinstanceshutdown. Apr1518:23:55Sunhmdg-db1[keepalived_notify]:Databaseinstanceshutdownsuccessfully. Apr1518:23:55Sunhmdg-db1[keepalived_notify]:umountsharedisk Apr1518:23:55Sunhmdg-db1[keepalived_notify]:umount/oradatasuccess. Apr1518:23:55Sunhmdg-db1[keepalived_notify]:stoppinglistener... Apr1518:23:56Sunhmdg-db1[keepalived_notify]:Thelistenerstopsuccessfully [root@hmdg-db1scripts]#ps-ef|grepora_ root975202451018:26pts/000:00:00grepora_ [root@hmdg-db1scripts]#df-Th FilesystemTypeSizeUsedAvailUse%Mountedon /dev/sda3ext450G19G29G40%/ tmpfstmpfs3.9G03.9G0%/dev/shm /dev/sda1ext4190M40M141M22%/boot [root@hmdg-db1scripts]#ipaddrshowdeveth1 3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000 link/ether00:0c:29:59:d5:91brdff:ff:ff:ff:ff:ff inet172.16.10.25/24brd172.16.10.255scopeglobaleth1 inet6fe80::20c:29ff:fe59:d591/64scopelink valid_lftforeverpreferred_lftforever
3、查看DB2是否转换为MASTER并接管服务
##系统日志 Apr1518:23:49hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)forcinganewMASTERelection Apr1518:23:51hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)TransitiontoMASTERSTATE Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)EnteringMASTERSTATE Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)settingprotocolVIPs. Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:23:53hmdg-db2Keepalived_healthcheckers[88133]:NetlinkreflectorreportsIP172.16.10.130added Apr1518:23:53hmdg-db2Keepalived_healthcheckers[88133]:NetlinkreflectorreportsIP172.16.10.131added Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:53hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:23:57hmdg-db2kernel:EXT4-fs(sdb1):warning:maximalmountcountreached,runninge2fsckisrecommended Apr1518:23:57hmdg-db2kernel:EXT4-fs(sdb1):mountedfilesystemwithordereddatamode.Opts: Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.130 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:VRRP_Instance(Oracle_HA_1)Sending/queueinggratuitousARPsoneth1for172.16.10.131 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.130 Apr1518:23:58hmdg-db2Keepalived_vrrp[88134]:SendinggratuitousARPoneth1for172.16.10.131 ##notify脚本日志 ------------------------------------------------------------------------------- Apr1518:23:53Sunhmdg-db2[keepalived_notify]:TransitiontoMASTERSTATE Apr1518:23:53Sunhmdg-db2[keepalived_notify]:SetuptheVIPoneth1172.16.10.130,172.16.10.131 Apr1518:23:53Sunhmdg-db2[keepalived_notify]:DatabaseSwitchoverToMASTER Apr1518:23:53Sunhmdg-db2[keepalived_notify]:Checkremotenodesharediskmounted. Apr1518:23:53Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[1]... Apr1518:23:54Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[2]... Apr1518:23:56Sunhmdg-db2[keepalived_notify]:/dev/sdb1isalreadymountedonremotenodeorbusy.checking[3]... Apr1518:23:57Sunhmdg-db2[keepalived_notify]:/dev/sdb1checkpassed. Apr1518:23:57Sunhmdg-db2[keepalived_notify]:mount/dev/sdb1on/oradata Apr1518:23:57Sunhmdg-db2[keepalived_notify]:restorecontrolfile1 ORACLEinstancestarted. TotalSystemGlobalArea1603411968bytes FixedSize2213776bytes VariableSize1056966768bytes DatabaseBuffers536870912bytes RedoBuffers7360512bytes RecoveryManager:Release11.2.0.1.0-ProductiononSunApr1518:23:582018 Copyright(c)1982,Oracleand/oritsaffiliates.Allrightsreserved. connectedtotargetdatabase:HMODB(notmounted) usingtargetdatabasecontrolfileinsteadofrecoverycatalog RMAN> Startingrestoreat15-APR-18 allocatedchannel:ORA_DISK_1 channelORA_DISK_1:SID=572devicetype=DISK channelORA_DISK_1:copiedcontrolfilecopy outputfilename=/oradata/HMODB/control01.ctl outputfilename=/u01/app/oracle/flash_recovery_area/HMODB/control02.ctl outputfilename=/home/oracle/rman/HMODB/control03.ctl Finishedrestoreat15-APR-18 RMAN> RecoveryManagercomplete. Apr1518:23:59Sunhmdg-db2[keepalived_notify]:alterdatabasetomount Databasealtered. Apr1518:24:04Sunhmdg-db2[keepalived_notify]:alterdatabasetoopen Databasealtered. Apr1518:24:11Sunhmdg-db2[keepalived_notify]:DatabaSEOpened. Apr1518:24:11Sunhmdg-db2[keepalived_notify]:Startuplistener Apr1518:24:11Sunhmdg-db2[keepalived_notify]:listeneralreadystarted. ##服务状态检查(Oracle进程、磁盘、VIP状态) [root@hmdg-db2scripts]#ps-ef|grepora_ oracle886291018:23?00:00:00ora_pmon_HMODB oracle886311018:23?00:00:00ora_vktm_HMODB oracle886351018:23?00:00:00ora_gen0_HMODB oracle886371018:23?00:00:00ora_diag_HMODB oracle886391018:23?00:00:00ora_dbrm_HMODB oracle886411018:23?00:00:00ora_psp0_HMODB oracle886431018:23?00:00:00ora_dia0_HMODB oracle886451018:23?00:00:00ora_mman_HMODB oracle886471018:23?00:00:00ora_dbw0_HMODB oracle886491018:23?00:00:00ora_lgwr_HMODB oracle886511018:23?00:00:00ora_ckpt_HMODB oracle886531018:23?00:00:00ora_smon_HMODB oracle886551018:23?00:00:00ora_reco_HMODB oracle886571018:23?00:00:00ora_mmon_HMODB oracle886591018:23?00:00:00ora_mmnl_HMODB oracle886611018:23?00:00:00ora_d000_HMODB oracle886631018:23?00:00:00ora_s000_HMODB oracle888281018:24?00:00:00ora_arc0_HMODB oracle888431018:24?00:00:00ora_arc1_HMODB oracle888451018:24?00:00:00ora_arc2_HMODB oracle888471018:24?00:00:00ora_arc3_HMODB oracle889441018:24?00:00:00ora_qmnc_HMODB oracle890111018:24?00:00:00ora_cjq0_HMODB oracle890261018:24?00:00:00ora_vkrm_HMODB oracle890541018:24?00:00:00ora_q000_HMODB oracle890561018:24?00:00:00ora_q001_HMODB root9079031390018:27pts/200:00:00grepora_ [root@hmdg-db2scripts]#df-Th FilesystemTypeSizeUsedAvailUse%Mountedon /dev/sda3ext450G18G30G37%/ tmpfstmpfs3.9G909M3.0G24%/dev/shm /dev/sda1ext4190M40M141M22%/boot /dev/sdb1ext430G6.0G22G22%/oradata [root@hmdg-db2scripts]#ipaddrshowdeveth1 3:eth1:<BROADCAST,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000 link/ether00:0c:29:dd:d1:7fbrdff:ff:ff:ff:ff:ff inet172.16.10.26/24brd172.16.10.255scopeglobaleth1 inet172.16.10.130/24scopeglobalsecondaryeth1 inet172.16.10.131/24scopeglobalsecondaryeth1 inet6fe80::20c:29ff:fedd:d17f/64scopelink valid_lftforeverpreferred_lftforever
结尾
本篇中主要的重点再在于测试部分。关于如何设计数据库高可用集群方案可作为参考
