CRS-1006 , CRS-0215 故障一例

    安装好sles 10 sp3 + Oracle 10g RAC之后，在配置监听器时，总是提示主机bo2dbp上的监听服务已经在运行，忽略错误之后手动在bo2dbp节点上启，总是收到TNS-12545: Connect failed because target host or object does not exist错误信息。后来发现节点bo2dbp的vip总是漂移到另一节点bo2dbs上，原来这才是罪魁祸首 ??????

1、集群环境netca之后，节点bo2dbp监听无法启动    手动尝试启动监听仍不成功，查看了listener.ora以及监听日志未发现任何异常，查看主机/etc/hosts配置也正常    无奈之下，删除监听器 oracle@bo2dbp:~> crs_stat -t    Name           Type           Target    State     Host           ------------------------------------------------------------    ora....BP.lsnr application    ONLINE    OFFLINE                  ora.bo2dbp.gsd application    ONLINE    ONLINE    bo2dbp         ora.bo2dbp.ons application    ONLINE    ONLINE    bo2dbp         ora.bo2dbp.vip application    ONLINE    ONLINE    bo2dbs         ora....BS.lsnr application    ONLINE    ONLINE    bo2dbs         ora.bo2dbs.gsd application    ONLINE    ONLINE    bo2dbs         ora.bo2dbs.ons application    ONLINE    ONLINE    bo2dbs         ora.bo2dbs.vip application    ONLINE    ONLINE    bo2dbs

2、节点bo2dbp.vip被漂移到bo2dbs oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_stat -t    Name           Type           Target    State     Host           ------------------------------------------------------------    ora.bo2dbp.gsd application    ONLINE    ONLINE    bo2dbp         ora.bo2dbp.ons application    ONLINE    ONLINE    bo2dbp         ora.bo2dbp.vip application    ONLINE    ONLINE    bo2dbs         ora.bo2dbs.gsd application    ONLINE    ONLINE    bo2dbs         ora.bo2dbs.ons application    ONLINE    ONLINE    bo2dbs         ora.bo2dbs.vip application    ONLINE    ONLINE    bo2dbs 

3、尝试为bo2dbp进行relocate也不听使唤 oracle@bo2dbp:~/product/10.2.0/crs_1/bin> ./crs_relocate ora.bo2dbp.vip -c bo2dbp    Attempting to stop `ora.bo2dbp.vip` on member `bo2dbs`    Stop of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.    Attempting to start `ora.bo2dbp.vip` on member `bo2dbp`    Start of `ora.bo2dbp.vip` on member `bo2dbp` failed.    Attempting to start `ora.bo2dbp.vip` on member `bo2dbs`    Start of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.    CRS-0217: Could not relocate resource 'ora.bo2dbp.vip'.

4、crs_stop 出马 oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_stop ora.bo2dbp.vip    Attempting to stop `ora.bo2dbp.vip` on member `bo2dbs`    Stop of `ora.bo2dbp.vip` on member `bo2dbs` succeeded. oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_start ora.bo2dbp.vip    Attempting to start `ora.bo2dbp.vip` on member `bo2dbp`    Start of `ora.bo2dbp.vip` on member `bo2dbp` failed.    Attempting to start `ora.bo2dbp.vip` on member `bo2dbs`    Start of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.

   我晕，又漂移到bo2dbs节点

5、狠一点，直接将节点2bo2dbs的crs停掉，这下没得漂了吧 :)    bo2dbs:/u01/app/oracle/product/10.2.0/crs_1/bin # ./crsctl stop crs    Stopping resources. This could take several minutes.    Successfully stopped CRS resources.    Stopping CSSD.    Shutting down CSS daemon.    Shutdown request successfully issued.

6、露出马脚 oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_start ora.bo2dbp.vip    Attempting to start `ora.bo2dbp.vip` on member `bo2dbp`    Start of `ora.bo2dbp.vip` on member `bo2dbp` failed.    Remote start for `ora.bo2dbp.vip` failed on member `bo2dbs`    CRS-1006: No more members to consider    CRS-0215: Could not start resource 'ora.bo2dbp.vip'.

7、check log

oracle@bo2dbp:~/product/10.2.0/crs_1/log/bo2dbp/racg> tail -50 ora.bo2dbp.vip.log    2012-08-30 16:18:55.942: [  RACG][4250161648] [20571][4250161648][ora.bo2dbp.vip]: end for resource = ora.bo2dbp.vip, action = start, status = 1, time = 6.390s    2012-08-30 16:19:30.968: [  RACG][4041843184] [20878][4041843184][ora.bo2dbp.vip]: checkIf: Default gateway is not defined (host=bo2dbp)    Interface eth0 checked failed (host=bo2dbp)    Invalid parameters, or failed to bring up VIP (host=bo2dbp)

   原来醉鬼祸首尽然是没有缺省网关，我倒倒倒......    记得之前的安装就碰到过在节点2上没有缺省网关导致vipca不能继续的情形。这次又碰上了。汗。。。 oracle@bo2dbp:~> cat /etc/sysconfig/network/routes    cat: /etc/sysconfig/network/routes: No such file or directory oracle@bo2dbp:~> ssh bo2dbs cat /etc/sysconfig/network/routes    default 192.168.7.254 - -    使用root帐户为其追加网关后，一切ok. oracle@bo2dbs:~> crs_stat -t    Name           Type           Target    State     Host           ------------------------------------------------------------    ora.bo2dbp.gsd application    ONLINE    ONLINE    bo2dbp         ora.bo2dbp.ons application    ONLINE    ONLINE    bo2dbp         ora.bo2dbp.vip application    ONLINE    ONLINE    bo2dbp         ora.bo2dbs.gsd application    ONLINE    ONLINE    bo2dbs         ora.bo2dbs.ons application    ONLINE    ONLINE    bo2dbs         ora.bo2dbs.vip application    ONLINE    ONLINE    bo2dbs  

8、问题环境    sles 10/sp3 + Oracle 10g(10.2.0.4)

9、小结:    a、细心细心再细心    b、根据相应的日志快速定位故障