Oracle 11g RAC CRS-4535/ORA-15077

    新安装了Oracle 11g rac之后,不知道是什么原因导致第二个节点上的crsd无法启动?其错误消息是CRS-4535: Cannot communicate with Cluster Ready Services。其具体的错误信息还需要查看crsd.log日志才知道。

1、环境
 [root@linux2 ~]# cat /etc/issue
 Enterprise Linux Enterprise Linux Server release 5.5 (Carthage)
 Kernel \r on an \m
 
 [root@linux2 bin]# ./crsctl query crs activeversion
 Oracle Clusterware active version on the cluster is [11.2.0.1.0]
 #注意下文中描述中使用了grid与root用户操作不同的对象。
 
2、错误症状
 [root@linux2 bin]# ./crsctl check crs
 CRS-4638: Oracle High Availability Services is online
 CRS-4535: Cannot communicate with Cluster Ready Services   #CRS-4535
 CRS-4529: Cluster Synchronization Services is online
 CRS-4533: Event Manager is online

 [root@linux2 bin]# ps -ef | grep d.bin   #下面的查询中没有crsd.bin
 root      3886     1  1 09:50 ?        00:00:11 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
 grid      3938     1  0 09:51 ?        00:00:04 /u01/app/11.2.0/grid/bin/oraagent.bin
 grid      4009     1  0 09:51 ?        00:00:00 /u01/app/11.2.0/grid/bin/gipcd.bin
 grid      4014     1  0 09:51 ?        00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
 grid      4028     1  0 09:51 ?        00:00:02 /u01/app/11.2.0/grid/bin/gpnpd.bin
 root      4040     1  0 09:51 ?        00:00:03 /u01/app/11.2.0/grid/bin/cssdmonitor
 root      4058     1  0 09:51 ?        00:00:04 /u01/app/11.2.0/grid/bin/cssdagent
 root      4060     1  0 09:51 ?        00:00:00 /u01/app/11.2.0/grid/bin/orarootagent.bin
 grid      4090     1  2 09:51 ?        00:00:15 /u01/app/11.2.0/grid/bin/ocssd.bin 
 grid      4094     1  0 09:51 ?        00:00:02 /u01/app/11.2.0/grid/bin/diskmon.bin -d -f
 root      4928     1  0 09:51 ?        00:00:00 /u01/app/11.2.0/grid/bin/octssd.bin reboot
 grid      4945     1  0 09:51 ?        00:00:02 /u01/app/11.2.0/grid/bin/evmd.bin
 root      6514  5886  0 10:00 pts/1    00:00:00 grep d.bin

 [root@linux2 bin]# ./crsctl stat res -t -init
 --------------------------------------------------------------------------------
 NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
 --------------------------------------------------------------------------------
 Cluster Resources
 --------------------------------------------------------------------------------
 ora.asm
       1        ONLINE  ONLINE       linux2                   Cluster Reconfigura 
                                                              tion                
 ora.crsd
       1        ONLINE  OFFLINE       #crsd处于offline状态                                              
 ora.cssd
       1        ONLINE  ONLINE       linux2                                       
 ora.cssdmonitor
       1        ONLINE  ONLINE       linux2                                       
 ora.ctssd
       1        ONLINE  ONLINE       linux2                   OBSERVER            
 ora.diskmon
       1        ONLINE  ONLINE       linux2                                       
 ora.drivers.acfs
       1        ONLINE  OFFLINE      #acfs处于offline状态                                             
 ora.evmd
       1        ONLINE  ONLINE       linux2                                       
 ora.gipcd
       1        ONLINE  ONLINE       linux2                                       
 ora.gpnpd
       1        ONLINE  ONLINE       linux2                                       
 ora.mdnsd
       1        ONLINE  ONLINE       linux2            
 
 #下面查看crsd对应的日志文件
 [grid@linux2 ~]$ view $ORACLE_HOME/log/linux2/crsd/crsd.log
 
 2013-01-05 10:28:27.107: [GIPCXCPT][1768145488] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], 
  ret gipcretSuccess (0)
 2013-01-05 10:28:27.107: [  OCRASM][1768145488]proprasmo: Error in open/create file in dg [OCR_VOTE] #打开磁盘组错误
 [  OCRASM][1768145488]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
 ORA-15077: could not locate ASM instance serving a required diskgroup  #出现了ORA错误
 
 2013-01-05 10:28:27.107: [  OCRASM][1768145488]proprasmo: kgfoCheckMount returned [7]
 2013-01-05 10:28:27.107: [  OCRASM][1768145488]proprasmo: The ASM instance is down    #实例处于关闭状态
 2013-01-05 10:28:27.107: [  OCRRAW][1768145488]proprioo: Failed to open [+OCR_VOTE]. Returned proprasmo() with [26].
   Marking location as UNAVAILABLE.
 2013-01-05 10:28:27.107: [  OCRRAW][1768145488]proprioo: No OCR/OLR devices are usable  #OCR/OLR设备不可用
 2013-01-05 10:28:27.107: [  OCRASM][1768145488]proprasmcl: asmhandle is NULL
 2013-01-05 10:28:27.107: [  OCRRAW][1768145488]proprinit: Could not open raw device
 2013-01-05 10:28:27.107: [  OCRASM][1768145488]proprasmcl: asmhandle is NULL
 2013-01-05 10:28:27.107: [  OCRAPI][1768145488]a_init:16!: Backend init unsuccessful : [26]
 2013-01-05 10:28:27.107: [  CRSOCR][1768145488] OCR context init failure.  Error: PROC-26: Error while accessing the 
  physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
 ORA-15077: could not locate ASM instance serving a required diskgroup
 ] [7]
 2013-01-05 10:28:27.107: [    CRSD][1768145488][PANIC] CRSD exiting: Could not init OCR, code: 26
 2013-01-05 10:28:27.107: [    CRSD][1768145488] Done.

 [root@linux2 bin]# ps -ef | grep pmon   #查看pmon进程,此处也表明ASM实例没有启动
 root      7447  7184  0 10:48 pts/2    00:00:00 grep pmon
  
  #从上面的分析可知,应该是ASM实例没有启动的原因导致了crsd进程无法启动

3、解决  
 [grid@linux2 ~]$ asmcmd          
 Connected to an idle instance.
 ASMCMD> startup                  #启动asm实例
 ASM instance started
 
 Total System Global Area  283930624 bytes
 Fixed Size                  2212656 bytes
 Variable Size             256552144 bytes
 ASM Cache                  25165824 bytes
 ASM diskgroups mounted
 ASMCMD> exit

 #Author : Robinson
 #Blog   : http://blog.csdn.net/robinson_0612
 
 #再次查看集群资源的状态
 [root@linux2 bin]# ./crsctl stat res -t -init
 --------------------------------------------------------------------------------
 NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
 --------------------------------------------------------------------------------
 Cluster Resources
 --------------------------------------------------------------------------------
 ora.asm
       1        ONLINE  ONLINE       linux2                   Started             
 ora.crsd
       1        ONLINE  INTERMEDIATE linux2                                       
 ora.cssd
       1        ONLINE  ONLINE       linux2                                       
 ora.cssdmonitor
       1        ONLINE  ONLINE       linux2                                       
 ora.ctssd
       1        ONLINE  ONLINE       linux2                   OBSERVER            
 ora.diskmon
       1        ONLINE  ONLINE       linux2                                       
 ora.drivers.acfs
       1        ONLINE  OFFLINE                                                   
 ora.evmd
       1        ONLINE  ONLINE       linux2                                       
 ora.gipcd
       1        ONLINE  ONLINE       linux2                                       
 ora.gpnpd
       1        ONLINE  ONLINE       linux2                                       
 ora.mdnsd
       1        ONLINE  ONLINE       linux2                     

 #启动acfs
 [root@linux2 bin]# ./crsctl start res ora.drivers.acfs -init
 CRS-2672: Attempting to start 'ora.drivers.acfs' on 'linux2'
 CRS-2676: Start of 'ora.drivers.acfs' on 'linux2' succeeded

 #之后所有的状态都处于online状态             
 [root@linux2 bin]# ./crsctl stat res -t -init
 --------------------------------------------------------------------------------
 NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
 --------------------------------------------------------------------------------
 Cluster Resources
 --------------------------------------------------------------------------------
 ora.asm
       1        ONLINE  ONLINE       linux2                   Started             
 ora.crsd
       1        ONLINE  ONLINE       linux2                                       
 ora.cssd
       1        ONLINE  ONLINE       linux2                                       
 ora.cssdmonitor
       1        ONLINE  ONLINE       linux2                                       
 ora.ctssd
       1        ONLINE  ONLINE       linux2                   OBSERVER            
 ora.diskmon
       1        ONLINE  ONLINE       linux2                                       
 ora.drivers.acfs
       1        ONLINE  ONLINE       linux2                                       
 ora.evmd
       1        ONLINE  ONLINE       linux2                                       
 ora.gipcd
       1        ONLINE  ONLINE       linux2                                       
 ora.gpnpd
       1        ONLINE  ONLINE       linux2                                       
 ora.mdnsd
       1        ONLINE  ONLINE       linux2    

有关grid相关故障链接:       Troubleshooting CRSD Start up Issue [ID 1323698.1] How to Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1]

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏琯琯博客

docker-resources资源汇集相关项目博文

docker资源汇总。英文版本链接 资源汇集 书籍 第一本Docker书 (7.4分) Docker —— 从入门到实践 (内容一般) The Docker B...

5157
来自专栏Golang语言社区

使用Docker和热加载运行Go API

This is a quick discussion of how to set up a local development environment for ...

1311
来自专栏王亚昌的专栏

How to build your own ubuntu image with docker?

docker run -d -p 222:22 ubuntu-sshd-admin

1152
来自专栏木制robot技术杂谈

Ubuntu 使用 Docker 安装 Gitlab

最近帮公司重新搭建了 Gitlab,中间遇到了一些坑,折腾了不少时间,在此记录供大家参考。

3614
来自专栏乐沙弥的世界

CRS-1006 , CRS-0215 故障一例

    安装好sles 10 sp3 + Oracle 10g RAC之后,在配置监听器时,总是提示主机bo2dbp上的监听服务已经在运行,忽略错误之后手动在b...

693
来自专栏圣杰的专栏

.NET Core+MySql+Nginx 容器化部署

1. 引言 上两节我们通过简单的demo学习了docker的基本操作。这一节我们来一个进阶学习,完成ASP.NET Core + MySql + Nginx的容...

4388
来自专栏吴伟祥

Xshell如何连接Docker容器 顶

3824
来自专栏杨建荣的学习笔记

使用shell定制awr脚本(r3笔记第32天)

大家在做性能问题诊断的时候,awr是不可或缺的工具,使用?/rdbms/admin/awrrpt.sql可能大家使用的多了,可能有时候感觉输入参数还是有些太繁琐...

2884
来自专栏bboysoul

自己动手做一个最小的docker镜像

其实有人学了很久还是把docker当虚拟机来使用,但是docker其实和虚拟机是完全不一样的,如何理解这一区别呢,我觉得自己动手做一个docker的hello ...

951
来自专栏康怀帅的专栏

在开发环境使用 Docker

本文是对官方文档的总结与备注。 官方文档:https://docs.docker.com/develop/ 根据官方文档的层次,分为 容器 (Container...

7234

扫码关注云+社区

领取腾讯云代金券