合 DBCA建库rac环境不能创建rac库问题处理
- 前言部分
- 导读和注意事项
- 故障分析及解决过程
- 故障环境介绍
- 故障发生现象及报错信息
- 故障分析及解决过程
- 模拟让故障重现
- 关于重建Inventory目录的2种办法
- 方法一：$ORACLE_HOME/oui/bin/attachHome.sh
- 方法二：集群下统一创建
- 方法三：直接修改/u01/app/oraInventory/ContentsXML/inventory.xml的内容
- 故障处理总结
- rac环境不能创建rac库
- 其它dbca静默建库常见错误的解决办法
- The Oracle system identifier(SID) "xxx" already exists. Specify another SID
- The cluster database(DB_NAME) "racdb" already exists. Please specify another DB_NAME.
- DBCA建库报错CRS-2566 PRCR-1071 PRCR-1006
- 用到的SQL集合
【故障处理】DBCA建库诡异问题处理--rac环境不能创建rac库
前言部分
导读和注意事项
各位技术爱好者，看完本文后，你可以掌握如下的技能，也可以学到一些其它你所不知道的知识，~O(∩_∩)O~：
① dbca静默创建rac库
② Inventory目录作用及其2种重建方法（重点）
③ rac环境dbca工具不能创建rac库的解决办法
④ dbca静默建库常见问题处理
⑤ 重建CRS集群环境执行root.sh脚本
故障分析及解决过程
故障环境介绍
| 项目 | source db |
|---|---|
| db 类型 | RAC |
| db version | 11.2.0.4 |
| db 存储 | ASM |
| OS版本及kernel版本 | AIX 64位 6.1.0.0 |
故障发生现象及报错信息
小麦苗采用dbca -silent+nodeinifo创建rac库，原来用的好好的命令，结果在这个rac环境上创建出来的库成了单实例的，很是郁闷，，，，且看故障发生的现象：
ZFZHLHRDB1:oracle:/oracle>dbca -silent -createDatabase -templateName General_Purpose.dbc -responseFile NO_VALUE \
> -gdbname raclhr -sid raclhr \
> -sysPassword oracle -systemPassword lhr \
> -datafileDestination 'DATA/' -recoveryAreaDestination 'DATA/' \
> -redoLogFileSize 50 \
> -storageType ASM -asmsnmpPassword lhr -diskGroupName 'DATA' \
> -characterset ZHS16GBK -nationalCharacterSet AL16UTF16 \
> -sampleSchema true \
> -automaticMemoryManagement true -totalMemory 2048 \
> -databaseType OLTP \
> -emConfiguration NONE \
> -nodeinfo ZFZHLHRDB1,ZFZHLHRDB2
Copying database files
1% complete
3% complete
10% complete
17% complete
24% complete
35% complete
Creating and starting Oracle instance
37% complete
42% complete
47% complete
52% complete
53% complete
56% complete
58% complete
Registering database with Oracle Restart
64% complete
Completing Database Creation
68% complete
71% complete
75% complete
85% complete
96% complete
100% complete
Look at the log file "/oracle/app/oracle/cfgtoollogs/dbca/raclhr/raclhr.log" for further details.
ZFZHLHRDB1:oracle:/oracle>more /oracle/app/oracle/cfgtoollogs/dbca/raclhr/raclhr.log
Copying database files
DBCA_PROGRESS : 1%
DBCA_PROGRESS : 3%
DBCA_PROGRESS : 10%
DBCA_PROGRESS : 17%
DBCA_PROGRESS : 24%
DBCA_PROGRESS : 31%
DBCA_PROGRESS : 35%
Creating and starting Oracle instance
DBCA_PROGRESS : 37%
DBCA_PROGRESS : 42%
DBCA_PROGRESS : 47%
DBCA_PROGRESS : 52%
DBCA_PROGRESS : 53%
DBCA_PROGRESS : 56%
DBCA_PROGRESS : 58%
Registering database with Oracle Restart
DBCA_PROGRESS : 64%
Completing Database Creation
DBCA_PROGRESS : 68%
DBCA_PROGRESS : 71%
DBCA_PROGRESS : 75%
DBCA_PROGRESS : 85%
DBCA_PROGRESS : 96%
DBCA_PROGRESS : 100%
Database creation complete. For details check the logfiles at:
/oracle/app/oracle/cfgtoollogs/dbca/raclhr.
Database Information:
Global Database Name:raclhr
System Identifier(SID):raclhr
ZFZHLHRDB1:oracle:/oracle>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue Jul 5 17:17:42 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SYS@raclhr> show parameter cluster
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_database boolean FALSE
cluster_database_instances integer 1
cluster_interconnects string
大家看到这里的安装百分比是1%->3%->10%-17%.....,而rac库创建的时候进度百分比应该是1%->3%->9%-15%.....,如下图所示：
Copying database files
1% complete
3% complete
9% complete
15% complete
21% complete
30% complete
Creating and starting Oracle instance
32% complete
36% complete
40% complete
44% complete
45% complete
48% complete
50% complete
Creating cluster database views
52% complete
70% complete
Completing Database Creation
73% complete
76% complete
85% complete
94% complete
100% complete
Look at the log file "/oracle/app/oracle/cfgtoollogs/dbca/raclhr/raclhr.log" for further details.
若是11.2.0.3环境的话：
RAC过程：
1% complete
3% complete
30% complete
Creating and starting Oracle instance
32% complete
36% complete
40% complete
44% complete
45% complete
48% complete
50% complete
Creating cluster database views
52% complete
70% complete
Completing Database Creation
73% complete
76% complete
85% complete
94% complete
100% complete
单实例过程：
1% complete
3% complete
35% complete
Creating and starting Oracle instance
37% complete
42% complete
47% complete
52% complete
53% complete
56% complete
58% complete
Registering database with Oracle Restart
64% complete
Completing Database Creation
68% complete
71% complete
75% complete
85% complete
96% complete
100% complete
接下来小麦苗尝试了如下的几种方式：①dbca采用响应文件的形式，②dbca图形界面的方式，都是同样的情况。
我的环境dbca图形界面的第一张图：
而正常rac环境下dbca图形界面的第一步图应该是：
以上都是目前出现的问题，初步预估是哪个配置文件出错了，dbca不能判断集群环境。
故障分析及解决过程
起初小麦苗以为是集群出现了问题，在万般无奈的情况下执行root.sh重建了集群，按照以前的经验，这个终极大招执行后集群肯定可以恢复到正常的情况下，结果呢？结果还是令小麦苗很伤心，情况依旧没有解决。重新执行root.sh的命令如下（注：重新执行root.sh并不会清掉db的数据）：
ocrconfig -showbackup
ocrconfig -manualbackup
$ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force -verbose
ls -l $ORACLE_BASE/Clusterware/ckptGridHA*
find $ORACLE_HOME/gpnp/* -type f
find $ORACLE_HOME/gpnp/* -type f -exec rm -rf {} \;
$ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force -verbose -lastnode -keepdg
ls -l $ORACLE_BASE/Clusterware/ckptGridHA*
find $ORACLE_HOME/gpnp/* -type f
find $ORACLE_HOME/gpnp/* -type f -exec rm -rf {} \;
ipcs
$ORACLE_HOME/root.sh
/oracle/app/11.2.0/grid/perl/bin/perl -I/oracle/app/11.2.0/grid/perl/lib -I/oracle/app/11.2.0/grid/crs/install /oracle/app/11.2.0/grid/crs/install/rootcrs.pl
ipcs
$ORACLE_HOME/root.sh
---crs的配置文件
---$ORACLE_HOME/crs/install/crsconfig_params
-- GRID
export DISPLAY=22.188.216.132:0.0
$ORACLE_HOME/crs/config/config.sh
这个终极大招已经使用过了，问题没有解决只能说明不是权限的问题，也不是集群的问题，而是DB环境的问题。
于是硬着头皮分析分析dbca的日志，dbca创建库的时候日志在：11g:$ORACLE_BASE/cfgtoollogs/dbca , 10g:$ORACLE_HOME/cfgtoollogs/dbca
ZFZHLHRDB1:oracle:/oracle>cd /oracle/app/oracle/cfgtoollogs/dbca/raclhr
total 15592
-rw-r----- 1 oracle dba 1454 Jul 05 16:56 CloneRmanRestore.log
-rw-r----- 1 oracle dba 182 Jul 05 16:52 DetectOption.log
-rw-r----- 1 oracle dba 43191 Jul 05 16:59 cloneDBCreation.log
-rw-r----- 1 oracle dba 48 Jul 05 16:59 lockAccount.log
-rw-r----- 1 oracle dba 18557 Jul 05 16:59 mkplug_v3_raclhr.log
-rw-r----- 1 oracle dba 1630 Jul 05 16:59 postDBCreation.log
-rw-r----- 1 oracle dba 16 Jul 05 16:59 postScripts.log
-rw-r----- 1 oracle dba 718 Jul 05 16:51 raclhr.log
-rw-r----- 1 oracle dba 738 Jul 05 17:00 raclhr0.log
-rw-r----- 1 oracle dba 1476 Jul 05 17:00 raclhr_creation_checkpoint.xml
-rw-r----- 1 oracle dba 1454 Jul 05 16:55 rmanRestoreDatafiles.sql
-rw-r----- 1 oracle dba 7716864 Jul 05 16:56 tempControl.ctl
-rw-r----- 1 oracle dba 155662 Jul 05 17:00 trace.log
-rw-r----- 1 oracle asmadmin 2258 Jul 05 16:57 tts_example_imp.log
里边有trace.log文件，拿到手，搜了下cluster，发现一个错误：
[main] [ 2016-07-05 17:37:30.694 GMT+08:00 ] [HAUtils.getDefaultListenerProperties:1666] PRCR-1001 : Resource ora.LISTENER.lsnr does not exist
PRCR-1001 : Resource ora.LISTENER.lsnr does not exist
at oracle.cluster.impl.common.SoftwareModuleImpl.crsResource(SoftwareModuleImpl.java:776)
at oracle.cluster.impl.nodeapps.ListenerImpl.crsResource(ListenerImpl.java:1107)
ora.LISTENER.lsnr该资源不存在，那就手动创建该资源：
ZFZHLHRDB1:grid:/home/grid>netca -silent -responsefile $ORACLE_HOME/assistants/netca/netca.rsp
Parsing command line arguments:
Parameter "silent" = true
Parameter "responsefile" = /oracle/app/11.2.0/grid/assistants/netca/netca.rsp
Done parsing command line arguments.
Oracle Net Services Configuration:
Profile configuration complete.
ListenerException: Could not create Listener: TNS-04414: File error
Error: null
Check the trace file for details: /oracle/app/grid/cfgtoollogs/netca/trace_GridHome1-1607055PM5133.log
Oracle Net Services configuration failed. The exit code is 1
报错了，应该是监听已经存在了，重新添加进crs中：
ZFZHLHRDB1:grid:/oracle/app/11.2.0/grid/network/admin>crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
ora.LISTENER_LHRDG.lsnr
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
ora.asm
ONLINE ONLINE ZFZHLHRDB1 Started
ONLINE ONLINE ZFZHLHRDB2 Started
ora.gsd
OFFLINE OFFLINE ZFZHLHRDB1
OFFLINE OFFLINE ZFZHLHRDB2
ora.net1.network
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
ora.ons
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
ora.registry.acfs
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ZFZHLHRDB1
ora.cvu
1 ONLINE ONLINE ZFZHLHRDB2
ora.myrac.db
1 ONLINE ONLINE ZFZHLHRDB1 Open
ora.oc4j
1 ONLINE ONLINE ZFZHLHRDB2
ora.oraesdb.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE OFFLINE
ora.oraeskdb.db
1 ONLINE ONLINE ZFZHLHRDB1 Open,Readonly
2 OFFLINE OFFLINE Instance Shutdown
ora.scan1.vip
1 ONLINE ONLINE ZFZHLHRDB1
ora.ZFZHLHRDB1.vip
1 ONLINE ONLINE ZFZHLHRDB1
ora.ZFZHLHRDB2.vip
1 ONLINE ONLINE ZFZHLHRDB2
ZFZHLHRDB1:grid:/oracle/app/11.2.0/grid/network/admin>srvctl add listener -l LISTENER -p 1521 -o $ORACLE_HOME
ZFZHLHRDB1:grid:/oracle/app/11.2.0/grid/network/admin>crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
ora.LISTENER.lsnr
OFFLINE OFFLINE ZFZHLHRDB1
OFFLINE OFFLINE ZFZHLHRDB2
ora.LISTENER_LHRDG.lsnr
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
ora.asm
ONLINE ONLINE ZFZHLHRDB1 Started
ONLINE ONLINE ZFZHLHRDB2 Started
ora.gsd
OFFLINE OFFLINE ZFZHLHRDB1
OFFLINE OFFLINE ZFZHLHRDB2
ora.net1.network
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
ora.ons
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
ora.registry.acfs
ONLINE ONLINE ZFZHLHRDB1
ONLINE ONLINE ZFZHLHRDB2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ZFZHLHRDB1
ora.cvu
1 ONLINE ONLINE ZFZHLHRDB2
ora.oc4j
1 ONLINE ONLINE ZFZHLHRDB2
ora.oraesdb.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE OFFLINE
ora.oraeskdb.db
1 ONLINE ONLINE ZFZHLHRDB1 Open,Readonly
2 OFFLINE OFFLINE Instance Shutdown
ora.scan1.vip
1 ONLINE ONLINE ZFZHLHRDB1
ora.ZFZHLHRDB1.vip
1 ONLINE ONLINE ZFZHLHRDB1
ora.ZFZHLHRDB2.vip
1 ONLINE ONLINE ZFZHLHRDB2
ZFZHLHRDB1:grid:/home/grid>crsctl stop res ora.LISTENER.lsnr
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'ZFZHLHRDB2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'ZFZHLHRDB1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'ZFZHLHRDB1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'ZFZHLHRDB2' succeeded
ZFZHLHRDB1:grid:/home/grid>crsctl start res ora.LISTENER.lsnr
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'ZFZHLHRDB2'
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'ZFZHLHRDB1'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'ZFZHLHRDB1' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'ZFZHLHRDB2' succeeded
原本以为就是这个原因导致的，结果添加完ora.LISTENER.lsnr后还是单实例的，郁闷。。。更郁闷的是第二次创建后，trace文件中已经没有任何错误了，无论搜error或fail都搜不到，但是创建的数据库还是单实例，，，第二次创建的日志：
有种深深的挫败感，于是解决同事碰到的那个DG问题，参考：http://blog.itpub.net/26736162/viewspace-2121688/。时间就这样过了2天，到了7月8号，手头又没什么事情了，想起来dbca这个问题，想来想去还是得从日志入手，老子一行一行的看，总能看出点东西来，但这次我比对的看，就是找一个dbca可以创建rac的环境，生成日志来分析差异，结果令我震惊了。
当小麦苗看到第5行的日志的时候，发现一个null的问题，Current Version From Inventory: null，截图如下：
错误环境截图：
而正常环境的rac是这个样子的：
可以看到正常环境下，可以获取到Homeinfo和Current Version From Inventory的值，当看到Inventory这个词的时候我就想我已经找到问题的答案了。应该是Inventory这个目录出现了问题，标准点的说法应该是/oracle/app/oraInventory/ContentsXML/inventory.xml这个文件的内容有问题了。Inventory目录存放的是Oracle软件安装的目录信息，Oracle升级也需要这个目录，执行opatch lspatches也需要这个目录。oraInventory目录的位置在/etc/oraInst.loc中记录，11g中我们可以通过$ORACLE_HOME/oui/bin/attachHome.sh来重建这个目录。
在其中的1个节点上分别用oracle和grid来执行：
The user is root. Oracle Universal Installer cannot continue installation if the user is root.
: A file or directory in the path name does not exist.
ZFLHRDB1:root:/oracle/app/oraInventory>su - grid
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 8192 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/app/oraInventory
'AttachHome' was successful.
ZFLHRDB1:grid:/home/grid>cd /oracle/app/oraInventory/ContentsXML
ZFLHRDB1:grid:/oracle/app/oraInventory/ContentsXML>more inv*
\<?xml version="1.0" standalone="yes" ?>
\<!-- Copyright (c) 1999, 2013, Oracle and/or its affiliates.
All rights reserved. -->
\<!-- Do not modify the contents of this file by hand. -->
\<INVENTORY>
\<VERSION_INFO>
\<SAVED_WITH>11.2.0.4.0\</SAVED_WITH>
\<MINIMUM_VER>2.1.0.6.0\</MINIMUM_VER>
\</VERSION_INFO>
\<HOME_LIST>
\<HOME NAME="Ora11g_gridinfrahome1" LOC="/oracle/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
\<NODE_LIST>
\<NODE NAME="ZFLHRDB1"/>
\<NODE NAME="ZFLHRDB2"/>
\</NODE_LIST>
\</HOME>
\</HOME_LIST>
\<COMPOSITEHOME_LIST>
\</COMPOSITEHOME_LIST>
\</INVENTORY>
ZFLHRDB1:root:/oracle/app/oraInventory>su - oracle
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 8192 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/app/oraInventory
'AttachHome' was successful.
ZFLHRDB1:oracle:/oracle>cd /oracle/app/oraInventory/ContentsXML
ZFLHRDB1:oracle:/oracle/app/oraInventory/ContentsXML>more inv*
\<?xml version="1.0" standalone="yes" ?>
\<!-- Copyright (c) 1999, 2013, Oracle and/or its affiliates.
All rights reserved. -->
\<!-- Do not modify the contents of this file by hand. -->
\<INVENTORY>
\<VERSION_INFO>
\<SAVED_WITH>11.2.0.4.0\</SAVED_WITH>
