Oracle10.2.0.4/5版本数据库无法启动dbconsole问题解决

在2011年新装了oracle10.2.0.5数据库，在建库最后em创建时候报错。Starting Oracle Enterprise Manager 10g Database Control...

在2011年新装了Oracle10.2.0.5数据库，在建库最后em创建时候报错。

Starting Oracle Enterprise Manager 10g Database Control
............................................................................................. failed.
------------------------------------------------------------------
Logs are generated in directory /myhost/sysman/log
Jan 2, 2011 7:27:26 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
WARNING: Error executing /myhost/bin/emctl start dbconsole
Jan 2, 2011 7:27:26 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Error starting Database Control
Refer to the log file at /myhost/dbua/d4/upgrade/emConfig.log for more details.
Jan 2, 2011 7:27:26 PM oracle.sysman.emcp.EMConfig perform
CONFIG: Stack Trace:
oracle.sysman.emcp.exception.EMConfigException: Error starting Database Control
at oracle.sysman.emcp.EMDBPostConfig.performUpgrade(EMDBPostConfig.java:763)
at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:232)
at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:193)
at oracle.sysman.emcp.EMConfig.perform(EMConfig.java:184)
at oracle.sysman.assistants.util.em.EMConfiguration.run(EMConfiguration.java:436)
at oracle.sysman.assistants.util.em.EMConfigStep.executeImpl(EMConfigStep.java:140)
at oracle.sysman.assistants.util.step.BasicStep.execute(BasicStep.java:210)
at oracle.sysman.assistants.util.step.BasicStep.callStep(BasicStep.java:251)
at oracle.sysman.assistants.dbma.backend.EMConfigStep.executeStepImpl(EMConfigStep.java:104)
at oracle.sysman.assistants.dbma.backend.SummarizableStep.executeImpl(SummarizableStep.java:175)
at oracle.sysman.assistants.util.step.BasicStep.execute(BasicStep.java:210)
at oracle.sysman.assistants.util.step.Step.execute(Step.java:140)
at oracle.sysman.assistants.util.step.StepContext$ModeRunner.run(StepContext.java:2488)
at java.lang.Thread.run(Thread.java:534)

查看 ORACLE_HOME\_\sysman\log\emagent.trc文件有如下报错信息：

2011-01-09 09:36:56 Thread-51125136 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://myhost:1158/em/upload/: retStatus=-1
2011-01-09 09:36:56 Thread-51125136 ERROR ssl: Open wallet failed, ret = 28750
2011-01-09 09:36:56 Thread-51125136 ERROR ssl: nmehlenv_openWallet failed
2011-01-09 09:36:56 Thread-51125136 ERROR http: 15: Unable to initialize ssl connection with server, aborting connection attempt
通过查询support发现是因为The Root Certificate Authority (CA) expired on 31-DEC-2010 00:00:0 root授权信息过期导致，oracle官方给出的解决办法是打Patch 8350262，然后按照oracle给出的建议操作即可解决，，具体解决步骤如下：

Recovering from Configuration Errors on a Single Instance Database
1. Ignore any errors and continue with the installation or upgrade. The database will be created without errors.

2. Apply Patch 8350262 to your Oracle Home installation using OPatch.

opatch apply

Invoking OPatch 10.2.0.4.2

Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.

Oracle Home : /myhost
Central Inventory : /scratch/pchebrol/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /myhost/oui
Log file location : /myhost/cfgtoollogs/opatch/opatch2011-01-02_11-00-00AM.log

ApplySession applying interim patch '8350262' to OH '/myhost'

Running prerequisite checks...

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '8350262' for restore. This might take a while...
Backing up files affected by the patch '8350262' for rollback. This might take a while...

Patching component oracle.sysman.agent.core, 10.2.0.4.0a...
Updating jar file "/myhost/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
Updating jar file "/myhost/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
Updating jar file "/myhost/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/RootCert.class"
Updating jar file "/myhost/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/SecConstants.class"
Updating jar file "/myhost/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
Updating jar file "/myhost/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
Updating jar file "/myhost/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/util/RootCert.class"
Updating jar file "/myhost/sysman/jlib/emd_java.jar" with "/sysman/jlib/emd_
java.jar/oracle/sysman/eml/sec/util/SecConstants.class"
ApplySession adding interim patch '8350262' to inventory

Verifying the update...
Inventory check OK: Patch ID 8350262 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 8350262 are present in Oracle Home.

OPatch succeeded.

3. After applying the patch, force stop the Database Control (dbconsole) process using the killDBConsole script bundled with the patch. Note that the dbconsole process cannot be stopped using the emctl stop dbconsole command, as EMCTL is unable to connect to the process.

To execute the killDBConsole script:

Set the ORACLE_HOME and ORACLE_SID environment variables.
Execute /killDBConsole.
Note for Windows Platform Only:

It is not necessary to force stop the dbconsole process on the Windows platform, because the process will
already be in a stopped state at the end of the failed configuration attempt.

The killDBConsole script output is shown below:

$ /killDBConsole
ORACLE_HOME=/myhost/db_1
ORACLE_SID=caem31
State directory = /myhost/db_1/staxd10_caem31
WatchDog PID = 802932
DBconsole PID = 577716
EMAgent PID = 512156
Killing WatchDog (pid=802932) ...
Successfully killed process 802932
Killing DBConsole (pid=577716) ...
Successfully killed process 577716
Killing EMAgent (pid=512156) ...
Successfully killed process 512156

4. Re-secure Database Control with the following command:

/bin/emctl secure dbconsole -reset

You will be prompted twice to confirm that the Root key must be overwritten. In both cases, enter upper-case "Y" as the response. Any other response (including lower-case "y") will cause the command to terminate without completing. If this happens, the command can be re-invoked.

$ ./emctl secure dbconsole -reset
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
https://myhost:5501/em/console/aboutApplication
Enter Enterprise Manager Root Password :
DBCONSOLE already stopped... Done.
Agent is already stopped... Done.
Securing dbconsole... Started.
Checking Repository... Done.
Checking Em Key... Done.
Checking Repository for an existing Enterprise Manager Root Key...
WARNING! An Enterprise Manager Root Key already exists in
the Repository. This operation will replace your Enterprise
Manager Root Key.
All existing Agents that use HTTPS will need to be
reconfigured if you proceed. Do you wish to continue and
overwrite your Root Key
(Y/N) ?
Y
Are you sure ? Reset of the Enterprise Manager Root Key
will mean that you will need to reconfigure each Agent
that is associated with this OMS before they will be
able to upload any data to it. Monitoring of Targets
associated with these Agents will be unavailable until
after they are reconfigured.
(Y/N) ?
Y
Generating Enterprise Manager Root Key (this takes a minute)... Done.Fetching Root Certificate from
the Repository... Done.
Updating HTTPS port in emoms.properties file... Done.
Generating Java Keystore... Done.
Securing OMS ... Done.
Generating Oracle Wallet Password for Agent.... Done.
Generating wallet for Agent ... Done.
Copying the wallet for agent use... Done.
Storing agent key in repository... Done.
Storing agent key for agent ... Done.
Configuring Agent...
Configuring Agent for HTTPS in DBCONSOLE mode... Done.
EMD_URL set in /myhost/myhost/sysman/config/emd.properties
Done.
Configuring Key store.. Done.
Securing dbconsole... Sucessful.

5. Re-start Database Control with the following command:

/bin/emctl start dbconsole

Recovering from Configuration Errors in an Oracle Real Application Clusters (RAC) Environment
1. Ignore any errors and continue with the upgrade, so that the database is upgraded without errors.

2. Apply Patch 8350262 to your Oracle Home installation. Note that the OPatch utility will apply the patch to all nodes in the cluster, as shown below:

../OPatch/opatch apply
Invoking OPatch 10.2.0.4.2

Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.

Oracle Home : /myhost/oracle/product/10.2.0/db_1
Central Inventory : /myhost/app/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /myhost/oracle/product/10.2.0/db_1/oui
Log file location : /myhost/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2011-01-01_
21-30-27PM.log

ApplySession applying interim patch '8350262' to OH '/myhost/oracle/product/10.2.0/db_1'

Running prerequisite checks...

OPatch detected the node list and the local node from the inventory. OPatch will patch the local
system then propagate the patch to the remote nodes.

Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '8350262' for restore. This might take a while...
Backing up files affected by the patch '8350262' for rollback. This might take a while...

Patching component oracle.sysman.agent.core, 10.2.0.4.0a...
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with
“/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/RootCert.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emCORE.jar" with
"/sysman/jlib/emCORE.jar/oracle/sysman/eml/sec/util/SecConstants.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with
"/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/fsc/FSWalletUtil.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with
"/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/rep/RepWalletUtil.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with
"/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/RootCert.class"
Updating jar file "/myhost/oracle/product/10.2.0/db_1/sysman/jlib/emd_java.jar" with
"/sysman/jlib/emd_java.jar/oracle/sysman/eml/sec/util/SecConstants.class"
ApplySession adding interim patch '8350262' to inventory

Verifying the update...
Inventory check OK: Patch ID 8350262 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 8350262 are present in Oracle Home.

Patching in rolling mode.

Updating nodes 'myhost'
Apply-related files are:
FP = :/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_
files.txt"
DP = "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_
dirs.txt"
MP = "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/make_
cmds.txt"
RC = "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/remote_
cmds.txt"

Instantiating the file "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_
44/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/myhost/oracle/product/10.2.0/db_
1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/myhost/oracle/product/10.2.0/db_1/.patch_storage/8350262_Sep_14_2010_04_59_
44/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/myhost/oracle/product/10.2.0/db_
1/.patch_storage/8350262_Sep_14_2010_04_59_44/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...

OPatch succeeded.

3. After applying the patch, force stop the Database Control (dbconsole) process by executing the
killDBConsole script bundled with the patch on each node in the cluster. Note that the dbconsole
process cannot be stopped using the emctl stop dbconsole command, as EMCTL is unable to connect
to the process.

To execute the killDBConsole script:

Set the ORACLE_HOME and ORACLE_SID environment variables.
Execute /killDBConsole
Note for Windows Platform Only:
It is not necessary to force stop the dbconsole process on the Windows platform, because the process will
already be in a stopped state at the end of the failed configuration attempt.

The killDBConsole script output is shown below:

$ /killDBConsole
ORACLE_HOME=/myhost/catest/db_1
ORACLE_SID=caem31
State directory = /myhost/catest/db_1/staxd10_caem31
WatchDog PID = 802932
DBconsole PID = 577716
EMAgent PID = 512156
Killing WatchDog (pid=802932) ...
Successfully killed process 802932
Killing DBConsole (pid=577716) ...
Successfully killed process 577716
Killing EMAgent (pid=512156) ...
Successfully killed process 512156

4. Re-secure Database Control on the first cluster node with the following command:

/bin/emctl secure dbconsole -reset

You will be prompted twice to confirm that the Root key must be overwritten. In both cases, enter upper-case "Y" as the response. Any other response (including lower-case "y") will cause the command to terminate without completing. If this happens, the command can be re-invoked.

$ ./emctl secure dbconsole -reset
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
https://myhost:5501/em/console/aboutApplication
Enter Enterprise Manager Root Password :
DBCONSOLE already stopped... Done.
Agent is already stopped... Done.
Securing dbconsole... Started.
Checking Repository... Done.
Checking Em Key... Done.
Checking Repository for an existing Enterprise Manager Root Key...
WARNING! An Enterprise Manager Root Key already exists in
the Repository. This operation will replace your Enterprise
Manager Root Key.
All existing Agents that use HTTPS will need to be
reconfigured if you proceed. Do you wish to continue and
overwrite your Root Key
(Y/N) ?
Y
Are you sure ? Reset of the Enterprise Manager Root Key
will mean that you will need to reconfigure each Agent
that is associated with this OMS before they will be
able to upload any data to it. Monitoring of Targets
associated with these Agents will be unavailable until
after they are reconfigured.
(Y/N) ?
Y
Generating Enterprise Manager Root Key (this takes a minute)... Done.Fetching Root Certificate from
the Repository... Done.
Updating HTTPS port in emoms.properties file... Done.
Generating Java Keystore... Done.
Securing OMS ... Done.
Generating Oracle Wallet Password for Agent.... Done.
Generating wallet for Agent ... Done.
Copying the wallet for agent use... Done.
Storing agent key in repository... Done.
Storing agent key for agent ... Done.
Configuring Agent...
Configuring Agent for HTTPS in DBCONSOLE mode... Done.
EMD_URL set in /myhost/sysman/config/emd.properties
Done.
Configuring Key store.. Done.
Securing dbconsole... Sucessful.

5. Re-secure Database Control on the remaining cluster nodes with the following command. Note that the -reset switch is not included with this command:

/bin/emctl secure dbconsole

[myhost bin]$ ./emctl secure dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
https://myhost:1158/em/console/aboutApplication
Enter Enterprise Manager Root password :
Enter a Hostname for this OMS : myhost
DBCONSOLE already stopped... Done.
Agent is already stopped... Done.
Securing dbconsole... Started.
Checking Repository... Done.
Checking Em Key... Done.
Checking Repository for an existing Enterprise Manager Root Key... Done.
Fetching Root Certificate from the Repository... Done.
Updating HTTPS port in emoms.properties file... Done.
Generating Java Keystore... Done.
Securing OMS ... Done.
Generating Oracle Wallet Password for Agent.... Done.
Generating wallet for Agent ... Done.
Copying the wallet for agent use... Done.
Storing agent key in repository... Done.
Storing agent key for agent ... Done.
Configuring Agent...
Configuring Agent for HTTPS in DBCONSOLE mode... Done.
EMD_URL set in /myhost/oracle/product/10.2.0/db_1/myhost/sysman/c
onfig/emd.properties
Done.
Configuring Key store.. Done.
Securing dbconsole... Sucessful.

6. Re-start Database Control by executing the following command on each node in the cluster:

/bin/emctl start dbconsole