First critical patch update for 12c was released Oct 15 2013. This post looks at the difference in patching 12c RAC environment (with role separation) compared to 11.2 environment. The environment used for patching is the environment that was upgraded from 11.2 to 12c.
First thing to notice is the name of the patch. On the readme.html that is included in the patch it is referred to as "Oracle Grid Infrastructure System Patch" instead of "Oracle Grid Infrastructure Patch Set Update" (More jargon to converse with!). However in the PSU and CPU availability document (1571391.1) it is still referred to as PSU (GI 12.1.0.1.1 PSU Patch 17272829). "GI System Patch" is used throughout the readme.html document so it's pretty safe to assume that's how the 12c patches going to be referred from now on.
Opatch auto option has been merged into one single command called "opatchauto".
However it is still possible to apply the patch manually. But at the time of this post (16/10/2013) the document with instruction for manual patch apply/rollback (1591616.1) is not available on MOS though the readme.html mentions it (shouldn't this be available before patches are released?). When this become available follow it for manual patch apply. In mean time as a workaround generateSteps option could be used to list the steps used by opatchauto
Next issue is related to patch location. Readme.html mentions to use "PATH_TO_PATCH_DIRECTORY" in the opatchauto command. PATH_TO_PATCH_DIRECTORY is the location where the patch was unzipped. This is same as the 11.2. However this location is not recognized by the opatchauto command and complains of the missing bundle.xml file.
Also worth noting is that along with opatchauto keyword apply must be given without it a syntax error occurs
The readme.html for GI system patch doesn't list any post installation task such as loading modified SQLs. This is automatically run as part of the patch apply. Once the patch is applied on the last node of the RAC the registry history is updated
Full output of running the opaatchauto is given below
On a system with PDBs that have dynamic services created for them, stopping RAC step will have the following output listing the service
Apply has the option of analyze which says
Unlike the RAC environment single instance database requires running the "loading modified SQLs" manually. 12c provides the datapatch tool for this purpose unlike in 11.2 where catbundle script was run for the same purpose. All databases (CDB and PDB) are updated.
Useful metalink notes
Known Patching Issues for the Oct 15 PSU, Oracle Database 12c R1 using opatchauto and EM [ID 1592252.1]
First thing to notice is the name of the patch. On the readme.html that is included in the patch it is referred to as "Oracle Grid Infrastructure System Patch" instead of "Oracle Grid Infrastructure Patch Set Update" (More jargon to converse with!). However in the PSU and CPU availability document (1571391.1) it is still referred to as PSU (GI 12.1.0.1.1 PSU Patch 17272829). "GI System Patch" is used throughout the readme.html document so it's pretty safe to assume that's how the 12c patches going to be referred from now on.
Opatch auto option has been merged into one single command called "opatchauto".
However it is still possible to apply the patch manually. But at the time of this post (16/10/2013) the document with instruction for manual patch apply/rollback (1591616.1) is not available on MOS though the readme.html mentions it (shouldn't this be available before patches are released?). When this become available follow it for manual patch apply. In mean time as a workaround generateSteps option could be used to list the steps used by opatchauto
/opt/app/12.1.0/grid/OPatch/opatchauto apply /usr/local/patch/17272829 -ocmrf ocm.rsp -generateStepsOPatch 12.1.0.1.2 or later is needed to apply this patch. Installing new OPatch on GI_HOME causes the following
unzip p6880880_121010_Linux-x86-64.zipFile PatchSearch.xml is to be copied (or unzipped) to GI_HOME outside the OPatch directory and since GI_HOME has restrictive permission unzipping as grid user causes the above error. The file could be copied manually as root user into GI_HOME or ignore the error (this caused no issue when installing the patch). Looking inside the PatchSearch.xml file it seem this might be used to get the OPatch from MOS (has urls of MOS and OPatch including CSI number). No such issue installing the new OPatch on ORACLE_HOME.
..
inflating: OPatch/operr
error: cannot create PatchSearch.xml
Permission denied
Next issue is related to patch location. Readme.html mentions to use "PATH_TO_PATCH_DIRECTORY" in the opatchauto command. PATH_TO_PATCH_DIRECTORY is the location where the patch was unzipped. This is same as the 11.2. However this location is not recognized by the opatchauto command and complains of the missing bundle.xml file.
[grid@rhel6m2 patches]$ pwdSo using the location where the patch was unzipped doesn't work unlike 11.2. Give the full path to the patch directory
/usr/local/patches <<-- this becomes the PATH_TO_PATCH_DIRECTORY (same as 11.2 as shown here)
[grid@rhel6m2 patches]$ ls
p17027533_121010_Linux-x86-64.zip
[grid@rhel6m2 patches]$ unzip p17027533_121010_Linux-x86-64.zip
[grid@rhel6m2 patches]$ su <-- preparing to run opatchauto as root user
[root@rhel6m2 patches]# /opt/app/12.1.0/grid/OPatch/opatchauto apply /usr/local/patches -ocmrf ocm.rsp
Parameter Validation: Successful
Patch Collection failed: Invalid patch location "/usr/local/patches" as there is no bundle.xml file in it or its parent directory.
opatchauto failed with error code 2.
[root@rhel6m2 patches]# /opt/app/12.1.0/grid/OPatch/opatchauto apply /usr/local/patches/17272829 -ocmrf ocm.rspApply of patch progress.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /opt/app/12.1.0/grid
opatchauto log file: /opt/app/12.1.0/grid/cfgtoollogs/opatchauto/17272829/opatch_gi_2013-10-16_15-39-34_deploy.log
Parameter Validation: Successful
...
Also worth noting is that along with opatchauto keyword apply must be given without it a syntax error occurs
[root@rhel6m1 patches]# /opt/app/12.1.0/grid/OPatch/opatchauto /usr/local/patches/17272829 -ocmrf ocm.rspSection 2.3 on the readme.html does mention apply keyword in the commands but in 2.4 Patch installation section the apply key word missing. This is another difference compared to 11.2 where there was no apply key word when opatch auto option was used. Rollback commands on section 2.7 are also incorrectly listed. Correct rollback commands are listed on section 2.3.
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Syntax Error... Unrecognized Command or Option (/usr/local/patches/17272829): 1st argument must be one of the following:
apply
rollback
version
..
The readme.html for GI system patch doesn't list any post installation task such as loading modified SQLs. This is automatically run as part of the patch apply. Once the patch is applied on the last node of the RAC the registry history is updated
SQL> select * from dba_registry_history;SQL apply is logged in dba_registry_sqlpatch table
ACTION_TIME ACTION NAMESPACE VERSION ID BUNDLE_SER COMMENTS
------------------------------ ---------- ---------- ---------- ---------- ---------- ------------------------------
12-AUG-13 04.28.26.378432 PM UPGRADE SERVER 12.1.0.1.0 Upgraded from 11.2.0.3.0
12-AUG-13 04.34.09.496894 PM APPLY SERVER 12.1.0.1 0 PSU Patchset 12.1.0.0.0
16-OCT-13 04.05.54.514261 PM APPLY SERVER 12.1.0.1 1 PSU PSU 12.1.0.1.1
SQL> show con_nameEach PDB will also have its own log file entry in the dba_registry_sqlpatch view
CON_NAME
------------------------------
CDB$ROOT
SQL> select * from dba_registry_sqlpatch;
PATCH_ID ACTION STATUS ACTION_TIME DESCRIPTIO LOGFILE
---------- ---------- --------------- ------------------------------ ---------- --------------------------------------------------------------------------------
17027533 APPLY SUCCESS 16-OCT-13 05.54.42.295071 PM sqlpatch /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_apply_CDB12C_
CDBROOT_2013Oct16_17_51_30.log
SQL> alter session set container=pdb12c;Even the pdb$seed database could be queried this way to confirm that it is also updated with the SQL changes made by the patch. Any new PDB created using the seed PDB also gets these modification and no patch post installation work is necessary.
Session altered.
SQL> show con_name
CON_NAME
------------------------------
PDB12C
SQL> select * from dba_registry_sqlpatch;
PATCH_ID ACTION STATUS ACTION_TIME DESCRIPTIO LOGFILE
---------- ---------- --------------- ------------------------------ ---------- --------------------------------------------------------------------------------
17027533 APPLY END 16-OCT-13 05.54.44.488402 PM sqlpatch /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_apply_CDB12C_
PDB12C_2013Oct16_17_51_49.log
Full output of running the opaatchauto is given below
[root@rhel6m1 17272829]# /opt/app/12.1.0/grid/OPatch/opatchauto apply `pwd` -ocmrf ../ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /opt/app/12.1.0/grid
opatchauto log file: /opt/app/12.1.0/grid/cfgtoollogs/opatchauto/17272829/opatch_gi_2013-10-16_14-23-50_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/opt/app/12.1.0/grid
RAC home(s):
/opt/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /usr/local/patches/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/opt/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: std11g2
Applying patch(es) to "/opt/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/usr/local/patches/17272829/17027533" successfully applied to "/opt/app/oracle/product/12.1.0/dbhome_1".
Patch "/usr/local/patches/17272829/17077442" successfully applied to "/opt/app/oracle/product/12.1.0/dbhome_1".
Stopping CRS ... Successful
Applying patch(es) to "/opt/app/12.1.0/grid" ...
Patch "/usr/local/patches/17272829/17027533" successfully applied to "/opt/app/12.1.0/grid".
Patch "/usr/local/patches/17272829/17077442" successfully applied to "/opt/app/12.1.0/grid".
Patch "/usr/local/patches/17272829/17303297" successfully applied to "/opt/app/12.1.0/grid".
Starting CRS ... Successful
Starting RAC (/opt/app/oracle/product/12.1.0/dbhome_1) ... Successful
SQL changes, if any, are applied successfully on the following database(s): std11g2
Apply Summary:
Following patch(es) are successfully installed:
GI Home: /opt/app/12.1.0/grid: 17027533, 17077442, 17303297
RAC Home: /opt/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
On a system with PDBs that have dynamic services created for them, stopping RAC step will have the following output listing the service
Stopping RAC (/opt/app/oracle/product/12.1.0/dbhome_1) ... SuccessfulIf there are no services created for the PDBs then only the CDB is mentioned in the output
Following database(s) were stopped and will be restarted later during the session: -pdbsvc,cdb12c
Stopping RAC (/opt/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: cdb12c
Apply has the option of analyze which says
-analyzeEven though it says "runs all the required prerequisite checks to confirm the patchability" this seem not be the case. Analyze could suceed and actual patch apply could fail.
This option runs all the required prerequisite checks to confirm
the patchability of the system without actually patching or
affecting the system in any way.
[root@rhel12c2 patch]# /opt/app/12.1.0/grid/OPatch/opatchauto apply /usr/local/patch/17272829 -ocmrf ocm.rsp -analyzeLog files list the failed steps and has commands that could be manually executed.
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /opt/app/12.1.0/grid
opatchauto log file: /opt/app/12.1.0/grid/cfgtoollogs/opatchauto/17272829/opatch_gi_2013-10-17_11-28-37_analyze.log
NOTE: opatchauto is running in ANALYZE mode. There will be no change to your system.
Parameter Validation: Successful
Grid Infrastructure home:
/opt/app/12.1.0/grid
RAC home(s):
/opt/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /usr/local/patch/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Analyzing patch(es) on "/opt/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/usr/local/patch/17272829/17027533" successfully analyzed on "/opt/app/oracle/product/12.1.0/dbhome_1" for apply.
Patch "/usr/local/patch/17272829/17077442" successfully analyzed on "/opt/app/oracle/product/12.1.0/dbhome_1" for apply.
Analyzing patch(es) on "/opt/app/12.1.0/grid" ...
Patch "/usr/local/patch/17272829/17027533" successfully analyzed on "/opt/app/12.1.0/grid" for apply.
Patch "/usr/local/patch/17272829/17077442" successfully analyzed on "/opt/app/12.1.0/grid" for apply.
Patch "/usr/local/patch/17272829/17303297" successfully analyzed on "/opt/app/12.1.0/grid" for apply.
SQL changes, if any, are analyzed successfully on the following database(s): cdb12c
Apply Summary:
Following patch(es) are successfully analyzed:
GI Home: /opt/app/12.1.0/grid: 17027533, 17077442, 17303297
RAC Home: /opt/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
opatchauto succeeded.
<<------ Running of actual patch command ----------->>
[root@rhel12c2 patch]# /opt/app/12.1.0/grid/OPatch/opatchauto apply /usr/local/patch/17272829 -ocmrf ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /opt/app/12.1.0/grid
opatchauto log file: /opt/app/12.1.0/grid/cfgtoollogs/opatchauto/17272829/opatch_gi_2013-10-17_11-32-12_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/opt/app/12.1.0/grid
RAC home(s):
/opt/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /usr/local/patch/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/opt/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: -pdbsvc,cdb12c
Applying patch(es) to "/opt/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/usr/local/patch/17272829/17027533" successfully applied to "/opt/app/oracle/product/12.1.0/dbhome_1".
Patch "/usr/local/patch/17272829/17077442" successfully applied to "/opt/app/oracle/product/12.1.0/dbhome_1".
Stopping CRS ... Successful
Applying patch(es) to "/opt/app/12.1.0/grid" ...
Command "/opt/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_patchList -local -invPtrLoc /opt/app/12.1.0/grid/oraInst.loc -oh /opt/app/12.1.0/grid -silent -ocmrf /usr/local/patch/ocm.rsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file Location for the failed command: /opt/app/12.1.0/grid/cfgtoollogs/opatch/opatch2013-10-17_11-39-04AM_1.log
[WARNING] The local database instance 'cdb12c2' from '/opt/app/oracle/product/12.1.0/dbhome_1' is not running. SQL changes, if any, will not be applied. Please refer to the log file for more details.
For more details, please refer to the log file "/opt/app/12.1.0/grid/cfgtoollogs/opatchauto/17272829/opatch_gi_2013-10-17_11-32-12_deploy.debug.log".
Apply Summary:
Following patch(es) are successfully installed:
RAC Home: /opt/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
Following patch(es) failed to be installed:
GI Home: /opt/app/12.1.0/grid: 17027533, 17077442, 17303297
opatchauto failed with error code 2.
-------------------Following steps still need to be executed-------------------Executing the first command shows how much free disk space must be available before the patch apply
/opt/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_patchList -local -invPtrLoc /opt/app/12.1.0/grid/oraInst.loc -oh /opt/app/12.1.0/grid -silent -ocmrf /usr/local/patch/ocm.rsp (TRIED BUT FAILED)
/opt/app/12.1.0/grid/rdbms/install/rootadd_rdbms.sh
/usr/bin/perl /opt/app/12.1.0/grid/crs/install/rootcrs.pl -postpatch
[grid@rhel12c2 patch]$ /opt/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_patchList -local -invPtrLoc /opt/app/12.1.0/grid/oraInst.loc -oh /opt/app/12.1.0/grid -silent -ocmrf /usr/local/patch/ocm.rsp
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /opt/app/12.1.0/grid
Central Inventory : /opt/app/oraInventory
from : /opt/app/12.1.0/grid/oraInst.loc
OPatch version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Log file location : /opt/app/12.1.0/grid/cfgtoollogs/opatch/opatch2013-10-17_11-42-52AM_1.log
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(10578.277MB) is not available.
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file location: /opt/app/12.1.0/grid/cfgtoollogs/opatch/opatch2013-10-17_11-42-52AM_1.log
OPatch failed with error code 73
Unlike the RAC environment single instance database requires running the "loading modified SQLs" manually. 12c provides the datapatch tool for this purpose unlike in 11.2 where catbundle script was run for the same purpose. All databases (CDB and PDB) are updated.
[oracle@rhel6m1 OPatch]$ ./datapatch -verboseEach container could be queried to check the status of the apply.
SQL Patching tool version 12.1.0.1.0 on Mon Oct 21 16:58:15 2013
Copyright (c) 2013, Oracle. All rights reserved.
Connecting to database...OK
Determining current state...
Currently installed SQL Patches:
PDB CDB$ROOT:
PDB PDB$SEED:
PDB PDB12C:
PDB PDB12CDI:
Currently installed C Patches: 17027533
For the following PDBs: CDB$ROOT
Nothing to roll back
The following patches will be applied: 17027533
For the following PDBs: PDB$SEED
Nothing to roll back
The following patches will be applied: 17027533
For the following PDBs: PDB12C
Nothing to roll back
The following patches will be applied: 17027533
For the following PDBs: PDB12CDI
Nothing to roll back
The following patches will be applied: 17027533
Adding patches to installation queue...
Installing patches...
Validating logfiles...
Patch 17027533 apply (pdb CDB$ROOT): SUCCESS
logfile: /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_apply_ENT12C_CDBROOT_2013Oct21_16_58_30.log (no errors)
Patch 17027533 apply (pdb PDB$SEED): SUCCESS
logfile: /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_apply_ENT12C_PDBSEED_2013Oct21_16_59_06.log (no errors)
Patch 17027533 apply (pdb PDB12C): SUCCESS
logfile: /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_apply_ENT12C_PDB12C_2013Oct21_16_59_32.log (no errors)
Patch 17027533 apply (pdb PDB12CDI): SUCCESS
logfile: /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_apply_ENT12C_PDB12CDI_2013Oct21_16_59_55.log (no errors)
SQL Patching tool complete on Mon Oct 21 17:00:30 2013
SQL> show con_name
CON_NAME
------------------------------
CDB$ROOT
SQL> select * from dba_registry_sqlpatch;
PATCH_ID ACTION STATUS ACTION_TIME DESCRIPTIO LOGFILE
---------- --------------- --------------- ---------------------------- ---------- ----------------------------------------------------------------------
17027533 APPLY SUCCESS 21-OCT-13 05.00.27.856979 PM sqlpatch /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_app
ly_ENT12C_CDBROOT_2013Oct21_16_58_30.log
SQL> ALTER SESSION SET container = pdb$seed;
Session altered.
SQL> show con_name
CON_NAME
------------------------------
PDB$SEED
SQL> select * from dba_registry_sqlpatch;
PATCH_ID ACTION STATUS ACTION_TIME DESCRIPTIO LOGFILE
---------- --------------- --------------- ---------------------------- ---------- ----------------------------------------------------------------------
17027533 APPLY SUCCESS 21-OCT-13 05.00.29.488402 PM sqlpatch /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_app
ly_ENT12C_PDBSEED_2013Oct21_16_59_06.log
SQL> ALTER SESSION SET container = pdb12c;
Session altered.
SQL> show con_name
CON_NAME
------------------------------
PDB12C
SQL> select * from dba_registry_sqlpatch;
PATCH_ID ACTION STATUS ACTION_TIME DESCRIPTIO LOGFILE
---------- --------------- --------------- ---------------------------- ---------- ----------------------------------------------------------------------
17027533 APPLY SUCCESS 21-OCT-13 05.00.30.823562 PM sqlpatch /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_app
ly_ENT12C_PDB12C_2013Oct21_16_59_32.log
SQL> ALTER SESSION SET container = pdb12cdi;
Session altered.
SQL> show con_name
CON_NAME
------------------------------
PDB12CDI
SQL> select * from dba_registry_sqlpatch;
PATCH_ID ACTION STATUS ACTION_TIME DESCRIPTIO LOGFILE
---------- --------------- --------------- ---------------------------- ---------- ----------------------------------------------------------------------
17027533 APPLY SUCCESS 21-OCT-13 05.00.30.996406 PM sqlpatch /opt/app/oracle/product/12.1.0/dbhome_1/sqlpatch/17027533/17027533_app
ly_ENT12C_PDB12CDI_2013Oct21_16_59_55.log
Useful metalink notes
Known Patching Issues for the Oct 15 PSU, Oracle Database 12c R1 using opatchauto and EM [ID 1592252.1]