ORA-15040 ora-15017ASM磁盘无法挂载故障处理

原创

PHP开发工程师

修改于 2021-05-10 18:08:55

1.2K0

修改于 2021-05-10 18:08:55

文章被收录于专栏：thinkphp+vuethinkphp+vue

测试环境突然一个节点的数据库无法启动，报错很清楚，找不到控制文件，而控制文件在ASM上，第一个节点正常，下面是处理流程。

数据库启动报错

SQL> startup

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2684354560 bytes

Fixed Size 8624312 bytes

Variable Size 989857608 bytes

Database Buffers 1677721600 bytes

Redo Buffers 8151040 bytes

ORA-00205: error in identifying control file, check alert log for more info

告警日志：

alter database mount

2021-04-22T15:06:27.293867+08:00

NOTE: ASMB mounting group 3 (USERDATA)

NOTE: ASM background process initiating disk discovery for grp 3 (reqid:0)

WARNING: group 3 (USERDATA) has missing disks

ORA-15040: diskgroup is incomplete

WARNING: group 3 is being dismounted.

WARNING: ASMB force dismounting group 3 (USERDATA) due to missing disks

SUCCESS: diskgroup USERDATA was dismounted

2021-04-22T15:06:27.330633+08:00

ORA-00210: cannot open the specified control file

ORA-00202: control file: '+USERDATA/PROD/CONTROLFILE/current.257.1055783219'

ORA-17503: ksfdopn:2 Failed to open file +USERDATA/PROD/CONTROLFILE/current.257.1055783219

ORA-15001: diskgroup "USERDATA" does not exist or is not mounted

ORA-15040: diskgroup is incomplete

ORA-205 signalled during: alter database mount...

磁盘组状态

SQL> select name,state,total_mb,type from v$asm_diskgroup

NAME STATE TOTAL_MB TYPE

-------------------- -------------------- ---------- ------------------

ACFSDATA MOUNTED 12224 EXTERN

DATA MOUNTED 51196 EXTERN

USERDATA MOUNTED 20476 EXTERN

查看磁盘组的状态

SQL>select b.name,path,a.state,a.mount_status,a.header_status from v$asm_disk a,v$asm_diskgroup b where a.group_number=b.group_number

NAME PATH STATE MOUNT_STATUS HEADER_STATUS

-------------------- -------------------- -------------------- --------------------- ------------------------------------

USERDATA /dev/sdd1 NORMAL CACHED MEMBER

ACFSDATA /dev/sdb1 NORMAL CACHED MEMBER

DATA /dev/sdc1 NORMAL CACHED MEMBER

查看磁盘信息

Disk /dev/sdd: 21.5 GB, 21474836480 bytes, 41943040 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x7bf56a92

Device Boot Start End Blocks Id System

/dev/sdd1 2048 41943039 20970496 83 Linux

尝试重新mount磁盘组

SQL> alter diskgroup usedata mount;

alter diskgroup usedata mount

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15017: diskgroup "USEDATA" cannot be mounted

ORA-15040: diskgroup is incomplete

下面尝试mount正常的磁盘组

SQL> alter diskgroup ACFSDATA mount;

alter diskgroup ACFSDATA mount

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15017: diskgroup "ACFSDATA" cannot be mounted

ORA-15013: diskgroup "ACFSDATA" is already mounted

显然二者的提示不同，问题磁盘组报错15040，下马是是该错误解释

[oracle@rac2 ~]$ oerr ora 15040

15040, 00000, "diskgroup is incomplete"

// *Cause: Some of the disks comprising a diskgroup were not present.

// *Action: Check the hardware to ensure that all disks are functional. Also

// check that the setting of the ASM_DISKSTRING initialization

// parameter has not changed. Alternatively, for normal or high

// redundancy diskgroups, use MOUNT FORCE to offline missing disks

// as part of mounting the diskgroup.

找不到ASM磁盘了，但是第一个节点该磁盘组是正常的，因为只有一个ASM磁盘，我们排除是ASM磁盘头的问题

应该是第二个节点与操作系统之间有问题，也就是实例2无法识别这个ASM磁盘

节点2 userdata磁盘组dismount后的状态

SQL> l

1* select b.name,path,a.state,a.mount_status,a.header_status from v$asm_disk a,v$asm_diskgroup b where a.group_number=b.group_number

SQL> /

NAME PATH STATE MOUNT_STATUS HEADER_STATUS

-------------------- -------------------- ------------------------ --------------------- ------------------------------------

USERDATA /dev/sde1 NORMAL CLOSED CANDIDATE

USERDATA /dev/sdg1 NORMAL CLOSED CANDIDATE

USERDATA /dev/sdg NORMAL CLOSED CANDIDATE

USERDATA /dev/sdb NORMAL CLOSED CANDIDATE

USERDATA /dev/sdc NORMAL CLOSED CANDIDATE

USERDATA /dev/sdf NORMAL CLOSED CANDIDATE

USERDATA /dev/sde NORMAL CLOSED CANDIDATE

USERDATA /dev/sda2 NORMAL CLOSED CANDIDATE

USERDATA /dev/sdd NORMAL CLOSED CANDIDATE

USERDATA /dev/sda1 NORMAL CLOSED CANDIDATE

ACFSDATA /dev/sdb1 NORMAL CACHED MEMBER

DATA /dev/sdc1 NORMAL CACHED MEMBER

12 rows selected.

节点2多了很多candidate盘，并且磁盘组userdata没有/dev/sdd1的盘为member状态。

节点1 userdata磁盘组dismount后的状态（该节点正常）

select 'alter diskgroup '||b.name||' drop disk '||path from v$asm_disk a,v$asm_diskgroup b where a.group_number=b.group_number;

SQL> select b.name,path,a.state,a.mount_status,a.header_status from v$asm_disk a,v$asm_diskgroup b where a.group_number=b.group_number

NAME PATH STATE MOUNT_STATUS HEADER_STATUS

-------------------- ------------------------------ ------------------------ --------------------- ------------------------------------

USERDATA /dev/sdd1 NORMAL CLOSED MEMBER

USERDATA /dev/sdf1 NORMAL CLOSED FORMER

ACFSDATA /dev/sdb1 NORMAL CACHED MEMBER

DATA /dev/sdc1 NORMAL CACHED MEMBER

显然两个节点对应的设备不同，从candidate含义可以知道多了备选的ASM磁盘，于是查询rules规则文件

[oracle@rac2 rules.d]$ ls -lrt *asm*

-rw-r--r-- 1 root root 712 Nov 1 13:16 99-my-asmdevices.rules

-rw-r--r-- 1 root root 1684 Apr 20 17:09 99-oracle-asmdevices.rules

发现节点2有两个规则，文件，突然记起来之前测试了一个自动生成rules文件的脚本，脚本内容如下，也就解释了为何出现这个问题。

[oracle@rac2 rules.d]$ cat 99-oracle-asmdevices.rules