Oracle RAC 故障处理(二)(+DATA磁盘组故障)

环境：测试 @H_301_6@

@H_301_6@DB:Oracle 11.2.0.4.0

@H_301_6@OS:Oracle Linux Server release 6.3 on Oracle VM VirtualBox

@H_301_6@node:rac1,rac2

@H_301_6@instance:cjcdb1,cjcdb2

@H_301_6@问题二：DATA磁盘组损坏，导致数据库无法启动和使用

@H_301_6@问题原因：测试RAC环境进行过多次增加节点、删除节点、删除实例等操作，导致DATA磁盘损坏

@H_301_6@解决方案：

@H_301_6@本次案例为测试环境，直接删除重建DATA磁盘组,重建实例，测试数据将丢失，正式环境勿用！

@H_301_6@---1 数据库启动失败，+DATA磁盘组故障

@H_301_6@---参数文件，控制文件，数据文件等都存储在+DATA磁盘组

@H_301_6@sql> startup

@H_301_6@ORA-01078: failure in processing system parameters

@H_301_6@ORA-01565: error in identifying file ‘+DATA/cjcdb/spfilecjcdb.ora‘

@H_301_6@ORA-17503: ksfdopn:2 Failed to open file +DATA/cjcdb/spfilecjcdb.ora

@H_301_6@ORA-15056: additional error message

@H_301_6@ORA-17503: ksfdopn:2 Failed to open file +DATA/cjcdb/spfilecjcdb.ora

@H_301_6@ORA-15001: diskgroup "DATA" does not exist or is not mounted

@H_301_6@ORA-06512: at line 4

@H_301_6@[[email protected] bin]# ./srvctl start instance -d cjcdb -i cjcdb1 -o open

@H_301_6@PRCR-1013 : Failed to start resource ora.cjcdb.db

@H_301_6@PRCR-1064 : Failed to start resource ora.cjcdb.db on node rac1

@H_301_6@CRS-5017: The resource action "ora.DATA.dg start" encountered the following error:

@H_301_6@ORA-15032: not all alterations performed

@H_301_6@ORA-15040: diskgroup is incomplete

@H_301_6@ORA-15042: ASM disk "3" is missing from group number "1"

@H_301_6@. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_grid/oraagent_grid.log".

@H_301_6@CRS-2674: Start of ‘ora.DATA.dg‘ on ‘rac1‘ Failed

@H_301_6@---2 查看当前只有OCR磁盘组，没有DATA磁盘组

@H_301_6@[[email protected] rac1]# su - grid

@H_301_6@[[email protected] ~]$ asmcmd

@H_301_6@ASMCMD> ls

@H_301_6@OCR/

@H_301_6@[[email protected] bin]# su - grid

@H_301_6@[[email protected] ~]$ asmcmd

@H_301_6@ASMCMD> ls

@H_301_6@OCR/

@H_301_6@---3 挂载DATA磁盘组失败

@H_301_6@sql> shutdown abort

@H_301_6@ASM instance shutdown

@H_301_6@sql> startup

@H_301_6@ORA-00099: warning: no parameter file specified for ASM instance

@H_301_6@ASM instance started

@H_301_6@Total System Global Area 1135747072 bytes

@H_301_6@Fixed Size 2260728 bytes

@H_301_6@Variable Size 1108320520 bytes

@H_301_6@ASM Cache 25165824 bytes

@H_301_6@ASM diskgroups mounted

@H_301_6@ASM diskgroups volume enabled

@H_301_6@sql> set line 100

@H_301_6@sql> select name,group_number,state from v$asm_diskgroup;

@H_301_6@NAME GROUP_NUMBER STATE

@H_301_6@OCR 2 MOUNTED

@H_301_6@DATA 0 DISMOUNTED

@H_301_6@sql> set line 150

@H_301_6@sql> col path for a30

@H_301_6@sql> select GROUP_NUMBER,disk_number,state,name,path from v$asm_disk;

@H_301_6@GROUP_NUMBER DISK_NUMBER STATE NAME PATH

@H_301_6@0 0 NORMAL /dev/asm-diske

@H_301_6@0 1 NORMAL /dev/asm-diskg

@H_301_6@0 2 NORMAL /dev/asm-diskf

@H_301_6@1 1 NORMAL OCR_0001 /dev/asm-diskc

@H_301_6@1 0 NORMAL OCR_0000 /dev/asm-diskb

@H_301_6@1 2 NORMAL OCR_0002 /dev/asm-diskd

@H_301_6@6 rows selected.

@H_301_6@sql> alter diskgroup data mount;

@H_301_6@alter diskgroup data mount

@H_301_6@*

@H_301_6@ERROR at line 1:

@H_301_6@ORA-15032: not all alterations performed

@H_301_6@ORA-15040: diskgroup is incomplete

@H_301_6@ORA-15042: ASM disk "3" is missing from group number "1"

@H_301_6@sql> drop diskgroup DATA;
drop diskgroup DATA

@H_301_6@*

@H_301_6@ERROR at line 1:

@H_301_6@ORA-15039: diskgroup not dropped

@H_301_6@ORA-15001: diskgroup "DATA" does not exist or is not mounted

@H_301_6@---4 重新初始化DATA磁盘组内磁盘

@H_301_6@[[email protected] ~]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules

@H_301_6@KERNEL=="sd*",BUS=="scsi",PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name",RESULT=="1ATA_VBox_HARDDISK_VBde6560cc-5b43c5dc",NAME="asm-diskb",OWNER="grid",GROUP="asmadmin",MODE="0660"

@H_301_6@KERNEL=="sd*",RESULT=="1ATA_VBox_HARDDISK_VB0bc9f076-5bc9501f",NAME="asm-diskc",RESULT=="1ATA_VBox_HARDDISK_VB69c7e48f-359e5a75",NAME="asm-diskd",RESULT=="1ATA_VBox_HARDDISK_VBf71d68a1-6c9355a0",NAME="asm-diske",RESULT=="1ATA_VBox_HARDDISK_VB6bae812a-c8d1bb88",NAME="asm-diskf",RESULT=="1ATA_VBox_HARDDISK_VB01fde635-97fae482",NAME="asm-diskg",MODE="0660"

@H_301_6@---asm-diskg 1ATA_VBox_HARDDISK_VB01fde635-97fae482

@H_301_6@---asm-diske 1ATA_VBox_HARDDISK_VBf71d68a1-6c9355a0

@H_301_6@---asm-diskf 1ATA_VBox_HARDDISK_VB6bae812a-c8d1bb88

@H_301_6@[[email protected] ~]# scsi_id -g /dev/sdg

@H_301_6@1ATA VBox HARDDISK VB01fde635-97fae482

@H_301_6@[[email protected] ~]# scsi_id -g /dev/sde

@H_301_6@1ATA VBox HARDDISK VBf71d68a1-6c9355a0

@H_301_6@[[email protected] ~]# scsi_id -g /dev/sdf

@H_301_6@1ATA VBox HARDDISK VB6bae812a-c8d1bb88

@H_301_6@[[email protected] ~]# dd if=/dev/zero of=/dev/sdg bs=1024k count=10

@H_301_6@10+0 records in

@H_301_6@10+0 records out