一、环境描述
(1)11.2.0.3 RAC ON Oracle Linux 6 x86_64，只有一个ASM外部冗余磁盘组——DATA；
(2)OCR，VOTEDISK，DATAFILE，CONTROLFILE，SPFILE全部位于这个磁盘组上；
二、故障描述
(1)存储故障导致ASM磁盘丢失。
(2)CRS因为OCR和VOTEDISK的丢失，除了OHAS还联机外，CLUSTERWARE服务都已经停止。
三、备份情况
(1)RMAN备份：包括controlfile，database，spfile，archivelog，
(2)OCR备份：没有进行过人工备份，在$CRS_HOME/cdata目录下有CRS自动备份文件。
四、操作步骤
说明：准使用CRS自动备份的文件恢复OCR，使用RMAN备份来恢复数据库；准备恢复数据的同时，调整ASM磁盘组，将OCR，VOTEDISK同数据库文件分开存放。
推荐阅读：
Oracle 11g从入门到精通 PDF+光盘源代码
12.04(amd64)安装完Oracle 11gR2后各种问题解决方法
4.1 恢复OCR和VOTEDISK
(1) 在所有RAC节点上停止CRS服务
[root@rac1 ~]# crsctl stop has -f
CRS-2791: Starting shutdownofOracle High Availability Services-managed resourceson'rac1'
CRS-2673: Attemptingtostop'ora.mdnsd'on'rac1'
CRS-2673: Attemptingtostop'ora.crf'on'rac1'
CRS-2677: Stopof'ora.mdnsd'on'rac1'succeeded
CRS-2677: Stopof'ora.crf'on'rac1'succeeded
CRS-2673: Attemptingtostop'ora.gipcd'on'rac1'
CRS-2677: Stopof'ora.gipcd'on'rac1'succeeded
CRS-2673: Attemptingtostop'ora.gpnpd'on'rac1'
CRS-2677: Stopof'ora.gpnpd'on'rac1'succeeded
CRS-2793: ShutdownofOracle High Availability Services-managed resourceson'rac1'has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac2 ~]# crsctl stop has -f
CRS-2791: Starting shutdownofOracle High Availability Services-managed resourceson'rac2'
CRS-2673: Attemptingtostop'ora.mdnsd'on'rac2'
CRS-2673: Attemptingtostop'ora.crf'on'rac2'
CRS-2677: Stopof'ora.mdnsd'on'rac2'succeeded
CRS-2677: Stopof'ora.crf'on'rac2'succeeded
CRS-2673: Attemptingtostop'ora.gipcd'on'rac2'
CRS-2677: Stopof'ora.gipcd'on'rac2'succeeded
CRS-2673: Attemptingtostop'ora.gpnpd'on'rac2'
CRS-2677: Stopof'ora.gpnpd'on'rac2'succeeded
CRS-2793: ShutdownofOracle High Availability Services-managed resourceson'rac2'has completed
CRS-4133: Oracle High Availability Services has been stopped.
(2) 在一个节点上以NOCRS方式启动CRS，此操作会启动ASM实例。
[root@rac1 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attemptingtostart'ora.mdnsd'on'rac1'
CRS-2676: Startof'ora.mdnsd'on'rac1'succeeded
CRS-2672: Attemptingtostart'ora.gpnpd'on'rac1'
CRS-2676: Startof'ora.gpnpd'on'rac1'succeeded
CRS-2672: Attemptingtostart'ora.cssdmonitor'on'rac1'
CRS-2672: Attemptingtostart'ora.gipcd'on'rac1'
CRS-2676: Startof'ora.cssdmonitor'on'rac1'succeeded
CRS-2676: Startof'ora.gipcd'on'rac1'succeeded
CRS-2672: Attemptingtostart'ora.cssd'on'rac1'
CRS-2672: Attemptingtostart'ora.diskmon'on'rac1'
CRS-2676: Startof'ora.diskmon'on'rac1'succeeded
CRS-2676: Startof'ora.cssd'on'rac1'succeeded
CRS-2679: Attemptingtoclean'ora.cluster_interconnect.haip'on'rac1'
CRS-2672: Attemptingtostart'ora.ctssd'on'rac1'
CRS-2681: Cleanof'ora.cluster_interconnect.haip'on'rac1'succeeded
CRS-2672: Attemptingtostart'ora.cluster_interconnect.haip'on'rac1'
CRS-2676: Startof'ora.ctssd'on'rac1'succeeded
CRS-2676: Startof'ora.cluster_interconnect.haip'on'rac1'succeeded
CRS-2672: Attemptingtostart'ora.asm'on'rac1'
CRS-2676: Startof'ora.asm'on'rac1'succeeded
(3) 新添加了三块磁盘，已经使用UDEV进行了绑定，查看磁盘状态。
[root@rac1 ~]# su - grid
[grid@rac1 ~]$ sqlplus /assysasm
SQL*Plus: Release 11.2.0.3.0 ProductiononFri Jul 5 17:41:49
Copyright (c) 1982, , Oracle.Allrights reserved.
Connectedto:
OracleDatabase11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
WiththeRealApplication ClustersandAutomatic Storage Management options
SQL>selectgroup_numbergroup#, disk_number disk#, OS_MB, state, path, header_statusfromv$asm_diskorderby1,2;
GROUP# DISK# OS_MB STATE PATH HEADER_STATUS
---------- ---------- ---------- ---------- -------------------- ----------------------
0 0 1024 NORMAL /dev/asm-diskc CANDIDATE
0 1 5120 NORMAL /dev/asm-diskd CANDIDATE
0 2 20480 NORMAL /dev/asm-diskb CANDIDATE
(4) 创建三个磁盘组，SYSTEMDG给CRS使用，用于存放OCR，VOTEDISK和ASM实例的SPFILE。其余两个给ORACLE使用，DATADG用于存放datafile，controlfile，redolog，spfile；ARCLOGDG存放archivelog。
SQL>creatediskgroup SYSTEMDG external redundancy
2 disk'/dev/asm-diskc'
3 ATTRIBUTE'compatible.rdbms'='11.2','compatible.asm'='11.2';
Diskgroup created.
SQL>creatediskgroup DATADG external redundancy
2 disk'/dev/asm-diskb'
3 ATTRIBUTE'compatible.rdbms'='11.2','compatible.asm'='11.2';
Diskgroup created.
SQL>creatediskgroup ARCLOGDG external redundancy
2 disk'/dev/asm-diskd'
3 ATTRIBUTE'compatible.rdbms'='11.2','compatible.asm'='11.2';
Diskgroup created.
(5) 准备恢复OCR和VOTEDISK，/etc/oracle/ocr.loc中记录了OCR路径，修改ocrconfig_loc的值，以便将OCR恢复到新的磁盘组中。
[root@rac1 ~]# more /etc/oracle/ocr.loc
ocrconfig_loc=+DATA
local_only=FALSE
[root@rac1 ~]# vi /etc/oracle/ocr.loc
ocrconfig_loc=+SYSTEMDG
local_only=FALSE
(6) 恢复OCR
[root@rac1 ~]# ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrievedfromalocalcopy
rac1 /07/05 12:30:00 /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
rac1 /07/05 08:30:00 /u01/app/11.2.0/grid/cdata/rac-cluster/backup01.ocr
rac1 /07/05 04:30:00 /u01/app/11.2.0/grid/cdata/rac-cluster/backup02.ocr
rac1 /07/05 00:29:59 /u01/app/11.2.0/grid/cdata/rac-cluster/day.ocr
rac1 /07/05 00:29:59 /u01/app/11.2.0/grid/cdata/rac-cluster/week.ocr
PROT-25: Manual backupsforthe Oracle Cluster Registry arenotavailable
[root@rac1 ~]# ocrconfig -restore /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
[root@rac1 ~]#
[root@rac1 ~]# ocrcheck
StatusofOracle Cluster Registryisasfollows :
Version : 3
Totalspace(kbytes) : 262120
Usedspace(kbytes) : 2840
Availablespace(kbytes) : 259280
ID : 59415097
Device/FileName: +SYSTEMDG
Device/File integritychecksucceeded
Device/Filenotconfigured
Device/Filenotconfigured
Device/Filenotconfigured
Device/Filenotconfigured
Cluster registry integritychecksucceeded
Logical corruptionchecksucceeded
(7) 创建VOTEDISK
[root@rac1 ~]# crsctlreplacevotedisk +SYSTEMDG
CRS-4602: Failed 27toaddvoting file afb0ca0f35684f1abfd43d5ec2dc1123.
Failedtoreplacevoting diskgroupwith+SYSTEMDG.
CRS-4000: CommandReplacefailed,orcompletedwitherrors.
以上报错是因为使用UDEV绑定ASM磁盘时需要更改默认磁盘搜索路径为/dev/asm*，修改ASM磁盘搜索路径
[root@rac1 ~]# su - grid
[grid@rac1 ~]$ sqlplus /assysasm
SQL*Plus: Release 11.2.0.3.0 ProductiononFri Jul 5 19:03:25
Copyright (c) 1982, , Oracle.Allrights reserved.
Connectedto:
OracleDatabase11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
WiththeRealApplication ClustersandAutomatic Storage Management options
SQL> show parameter asm_diskstring
NAMETYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring string
SQL>
SQL>
SQL>altersystemsetasm_diskstring ='/dev/asm*';
System altered.
SQL>createspfilefrommemory;
createspfilefrommemory
*
ERRORatline 1:
ORA-00349: failure obtaining blocksizefor
'+DATA/rac-cluster/asmparameterfile/registry.253.819922365'
ORA-15001: diskgroup"DATA"doesnotexistorisnotmounted
SQL>createspfile='+SYSTEMDG'frommemory;
File created.
SQL> startupforcemount;
ORA-3: obsoleteordeprecated parameter(s) specifiedforASM instance
ASM instance started
Total SystemGlobalArea 283930624 bytes
FixedSize2227664 bytes
VariableSize256537136 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
在次创建VOTEDISK，成功。
[root@rac1 init]# crsctlreplacevotedisk +SYSTEMDG
Successful additionofvoting disk 8ebb7a63accb4fa8bfa7ab65df7a8c8a.
Successfully replaced voting diskgroupwith+SYSTEMDG.
CRS-4266: Voting file(s) successfully replaced
(8) OCR和VOTEDISK都恢复完成后，重启CRS到正常模式。
[root@rac1 ~]# crsctl stop has -f
CRS-2791: Starting shutdownofOracle High Availability Services-managed resourceson'rac1'
CRS-2673: Attemptingtostop'ora.mdnsd'on'rac1'
CRS-2673: Attemptingtostop'ora.ctssd'on'rac1'
CRS-2673: Attemptingtostop'ora.asm'on'rac1'
CRS-2677: Stopof'ora.mdnsd'on'rac1'succeeded
CRS-2677: Stopof'ora.asm'on'rac1'succeeded
CRS-2673: Attemptingtostop'ora.cluster_interconnect.haip'on'rac1'
CRS-2677: Stopof'ora.ctssd'on'rac1'succeeded
CRS-2677: Stopof'ora.cluster_interconnect.haip'on'rac1'succeeded
CRS-2673: Attemptingtostop'ora.cssd'on'rac1'
CRS-2677: Stopof'ora.cssd'on'rac1'succeeded
CRS-2673: Attemptingtostop'ora.gipcd'on'rac1'
CRS-2677: Stopof'ora.gipcd'on'rac1'succeeded
CRS-2673: Attemptingtostop'ora.gpnpd'on'rac1'
CRS-2677: Stopof'ora.gpnpd'on'rac1'succeeded
CRS-2793: ShutdownofOracle High Availability Services-managed resourceson'rac1'has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 ~]# crsctlcheckcrs
CRS-4638: Oracle High Availability Servicesisonline
CRS-4537: Cluster Ready Servicesisonline
CRS-4529: Cluster Synchronization Servicesisonline
CRS-4533: Event Managerisonline
[root@rac1 ~]#
如果觉得《oracle 磁盘组掉了 Oracle 11g RAC ASM磁盘全部丢失后的恢复（一）》对你有帮助，请点赞、收藏，并留下你的观点哦！
