Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 12.2.0.1

This post lists the steps for upgrading an Oracle restart environment (Single instance non-CDB on ASM) from 11.2.0.4 to 12.2.0.1. The 11.2.0.4 setup was on RHEL 6.4. The current versions of the GI are

crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]

crsctl query has softwareversion
Oracle High Availability Services version on the local node is [11.2.0.4.0]

The Oracle restart environment used is a role separate setup where GI is installed as grid user while Oracle software is installed Oracle user. The GI and Oracle homes had the latest PSU applied (Database Patch Set Update : 11.2.0.4.171017 (26392168)). With this patch set applied orachk pre-upgrade did not list any patches to be applied before the upgrade. If the current patch set is different then additional patches may need to be applied before the upgrade. Refer 2180188.1 and earlier post of 11.2.0.4 to 12.2 RAC upgrade post for these patches.
Unlike the previous versions, 12.2 grid infrastructure requires unzipping it into a grid home. There's no GI home selection on the OUI. Create the new directory path for 12.2 GI and unzip the GI bundle.

mkdir -p /opt/app/oracle/product/12.2.0/grid
cp ~/linuxx64_12201_grid_home.zip /opt/app/oracle/product/12.2.0/grid
cd /opt/app/oracle/product/12.2.0/grid
unzip linuxx64_12201_grid_home.zip 

cluvfy from the 12.2 GI could be used to further check the system setup. This option is not for upgrade check but could be used to check readiness of oracle restart installation.

$ ./runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: rhel6m1:/usr,rhel6m1:/var,rhel6m1:/etc,rhel6m1:/sbin,rhel6m1:/tmp ...PASSED
Verifying User Existence: grid ...
  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: binutils-2.20.51.0.2 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libgcc-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.4.7 (x86_64) ...PASSED
Verifying Package: sysstat-9.0.4 ...PASSED
Verifying Package: gcc-4.4.7 ...PASSED
Verifying Package: gcc-c++-4.4.7 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.81 ...PASSED
Verifying Package: glibc-2.12 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.12 (x86_64) ...PASSED
Verifying Package: libaio-0.3.107 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.107 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-5.43-1 ...PASSED
Verifying Package: net-tools-1.60-110 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.

The failure encountered is due to under size swap which could be ignored.
ASM will be upgraded as part of the upgrade process. Therefore stop the database before starting the GI upgrade. Upgrade process will prompt to stop the database during gridSetup run as well.

srvctl stop database -d std11g2

Start the GI upgrade by running the gridSetup.sh from the grid home.

$ ./gridSetup.sh

This will start the GI installer and select the upgrade GI option. OS groups for authentication into ASM. Oracle base location could be selected but not the GI Home location. This will be fixed, based on where the GI software is extracted. Summary page and progress. When prompted execute the rootupgrade.sh

# /opt/app/oracle/product/12.2.0/grid/rootupgrade.sh
Performing root user operation.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m1/crsconfig/roothas_2017-10-19_05-13-24PM.log
2017/10/19 17:13:26 CLSRSC-363: User ignored prerequisites during installation

ASM has been upgraded and started successfully.

Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel6m1 successfully pinned.
2017/10/19 17:17:38 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'
CRS-4123: Oracle High Availability Services has been started.

2017/10/19 17:19:36 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p first'
2017/10/19 17:20:01 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p last'

rhel6m1     2017/10/19 17:20:04     /opt/app/oracle/product/12.2.0/grid/cdata/rhel6m1/backup_20171019_172004.olr     0

rhel6m1     2015/03/16 14:14:51     /opt/app/oracle/product/11.2.0/grid_4/cdata/rhel6m1/backup_20150316_141451.olr     -

rhel6m1     2015/03/11 17:57:18     /opt/app/oracle/product/11.2.0/grid_1/cdata/rhel6m1/backup_20150311_175718.olr     -
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/10/19 17:21:35 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

The GI versions after the upgrade

crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [12.2.0.1.0]
crsctl query has softwareversion
Oracle High Availability Services version on the local node is [12.2.0.1.0]

The next step of the upgrade is to install 12.2 database software. 12.2 introduced a new OS group for RAC DB management. If this is used then add the user group to both oracle and grid. Not adding the group to grid user result in an issue which is mentioned in 12.1.0.2 to 12.2.1.0 single instance upgrade post.

# groupadd racdba
# usermod -g oinstall -G dba,oper,asmoper,asmdba,asmadmin,backupdba,dgdba,kmdba,racdba oracle
# id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),505(asmadmin),506(asmdba),507(asmoper),509(dgdba),510(kmdba),511(backupdba),512(racdba)
# usermod -g oinstall -G dba,asmoper,asmdba,asmadmin,racdba grid
id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),502(dba),505(asmadmin),506(asmdba),507(asmoper),512(racdba)

As GI home was install first some of the directories may not have write permission for oracle user. Change permission of those directories. In this case it was the 12.2.0 directory in the GI home path.

drwxr-xr-x. 3 grid oinstall 4096 Oct 19 16:44 12.2.0
$ chmod 775 12.2.0
drwxrwxr-x. 3 grid oinstall 4096 Oct 19 16:44 12.2.0

Install DB software selecting "software only" option. RACDBA group selected for RAC administration.SummaryOnce the 12.2 DB software is installed next step is to upgrade the DB. 12.2 provides a preupgrade check jar file which need to be run using current oracle home (old version). Preupgrade.jar is available in 12.2_ORACLE_HOME/rdbms/admin

$ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
12.2.0.1.0

Upgrade-To version: 12.2.0.1.0

=======================================
Status of the database prior to upgrade
=======================================

      Database Name:  STD11G2
     Container Name:  Not Applicable in Pre-12.1 database
       Container ID:  Not Applicable in Pre-12.1 database
            Version:  11.2.0.4.0
         Compatible:  11.2.0.4.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  14
  Database log mode:  ARCHIVELOG
           Readonly:  FALSE
            Edition:  R1

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Enterprise Manager Repository   [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID


==============
BEFORE UPGRADE
==============

  Run /preupgrade_fixups.sql to complete all
  of the BEFORE UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
   + Adjust TABLESPACE SIZES as needed.
                                                Auto      12.2.0.1.0
     Tablespace                        Size     Extend    Min Size    Action
     ----------                     ----------  --------  ----------  ------

     SYSAUX                             600 MB  DISABLED      551 MB  None
     SYSTEM                             700 MB  ENABLED       806 MB  None
     TEMP                                20 MB  ENABLED       150 MB  None
     UNDOTBS1                           225 MB  ENABLED       400 MB  None

     Note that 12.2.0.1.0 minimum sizes are estimates.
     If you plan to upgrade multiple pluggable databases concurrently,
     then you must ensure that the UNDO tablespace size is equal to at least
     the number of pluggable databases that you upgrade concurrently,
     multiplied by that minimum.  Failing to allocate sufficient space can
     cause the upgrade to fail.

   + (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

     The database contains 2 objects in the recycle bin.

     The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
   + Remove the EM repository.

     - Copy the $ORACLE_HOME/rdbms/admin/emremove.sql script from the target
     12.2.0.1.0 ORACLE_HOME into the source 11.2.0.4.0 ORACLE_HOME.

     Step 1: If database control is configured, stop EM Database Control,
     using the following command

       $> emctl stop dbconsole

     Step 2: Connect to the database using the SYS account AS SYSDBA

       SET ECHO ON;
       SET SERVEROUTPUT ON;
      @emremove.sql

     Without the set echo and serveroutput commands, you will not be able to
     follow the progress of the script.

     The database has an Enterprise Manager Database Control repository.

     Starting with Oracle Database 12c, the local Enterprise Manager Database
     Control does not exist anymore. The repository will be removed from your
     database during the upgrade.  This step can be manually performed before
     the upgrade to reduce downtime.

   + Run 11.2.0.4.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
     objects.  You can view the individual invalid objects with

       SET SERVEROUTPUT ON;
       EXECUTE DBMS_PREUP.INVALID_OBJECTS;

     1 objects are INVALID.

     There should be no INVALID objects in SYS/SYSTEM or user schemas before
     database upgrade.

   + Backup the existing ACLs and their assignments for reference. Use the
     new DBMS_NETWORK_ACL_ADMIN interfaces and dictionary views to administer
     network privileges after upgrade.

     The database contains network ACLs with privileges that will be migrated
     to a new format in 12c.

     Network access control list (ACL) privileges in 11g will be migrated to
     a new format in 12c. As part of the migration, new
     DBMS_NETWORK_ACL_ADMIN interfaces and dictionary views are provided, and
     privileges in the existing ACLs will be converted to the new format with
     new ACL names. The old ACL names, DBMS_NETWORK_ACL_ADMIN interfaces and
     dictionary views may continue to be used but are deprecated and their
     use is discouraged. For further information, refer to My Oracle Support
     note number 2078710.1.

   + Review and remove any unnecessary EVENTS.

     The database contains events.

     There are events set that should be removed before upgrade, unless your
     application vendors and/or Oracle Support state differently.  Changes
     will need to be made in the spfile.

   + (AUTOFIXUP) Gather stale data dictionary statistics prior to database
     upgrade in off-peak time using:

      EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Dictionary statistics do not exist or are stale (not up-to-date).

     Dictionary statistics help the Oracle optimizer find efficient SQL
     execution plans and are essential for proper upgrade timing. Oracle
     recommends gathering dictionary statistics in the last 24 hours before
     database upgrade.

     For information on managing optimizer statistics, refer to the 11.2.0.4
     Oracle Database Performance Tuning Guide.

   + Directly grant ADMINISTER DATABASE TRIGGER privilege to the owner of the
     trigger or drop and re-create the trigger with a user that was granted
     directly with such. You can list those triggers using "SELECT OWNER,
     TRIGGER_NAME FROM DBA_TRIGGERS WHERE BASE_OBJECT_TYPE=''DATABASE'' AND
     OWNER NOT IN (SELECT GRANTEE FROM DBA_SYS_PRIVS WHERE
     PRIVILEGE=''ADMINISTER DATABASE TRIGGER'')"

     There is one or more database triggers whose owner does not have the
     right privilege on the database.

     The creation of database triggers must be done by users granted with
     ADMINISTER DATABASE TRIGGER privilege. Privilege must have been granted
     directly.

=============
AFTER UPGRADE
=============

  Run /postupgrade_fixups.sql to complete all
  of the AFTER UPGRADE action items below marked with '(AUTOFIXUP)'.

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
   + Upgrade the database time zone version using the DBMS_DST package.

     The database is using timezone datafile version 14 and the target
     12.2.0.1.0 database ships with timezone datafile version 26.

     Oracle recommends using the most recent timezone data.  For further
     information, refer to My Oracle Support Note 1585343.1.

   + (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

     Oracle recommends gathering dictionary statistics after upgrade.

     Dictionary statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans. After a
     database upgrade, statistics need to be re-gathered as there can now be
     tables that have significantly changed during the upgrade or new tables
     that do not have statistics gathered yet.

   + Gather statistics on fixed objects two weeks after the upgrade using the
     command:

       EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

     This recommendation is given for all preupgrade runs.

     Fixed object statistics provide essential information to the Oracle
     optimizer to help it find efficient SQL execution plans.  Those
     statistics are specific to the Oracle Database release that generates
     them, and can be stale upon database upgrade.

Preupgrade generated files:
    /opt/app/oracle/cfgtoollogs/std11g2/preupgrade/preupgrade_fixups.sql
    /opt/app/oracle/cfgtoollogs/std11g2/preupgrade/postupgrade_fixups.sql

Before the upgrade the DB component status and versions, plus the timezone file version.

 COMP_ID    COMP_NAME                                VERSION         STATUS
---------- ---------------------------------------- --------------- --------
XDB        Oracle XML Database                      11.2.0.4.0      VALID
CONTEXT    Oracle Text                              11.2.0.4.0      VALID
OWM        Oracle Workspace Manager                 11.2.0.4.0      VALID
CATALOG    Oracle Database Catalog Views            11.2.0.4.0      VALID
CATPROC    Oracle Database Packages and Types       11.2.0.4.0      VALID


FILENAME                VERSION
-------------------- ----------
timezlrg_14.dat              14

Read the output from preupgrade.jar and carry out necessary pre-upgrade tasks. Some are included in the preupgrade_fixups.sql while others may required to be carried out manually. Once pre-upgrade tasks are completed run dbua from the 12.2 home to begin the upgrade.
Select the DB for upgrade. In this case type is indicated as SI (single instance) Select the upgrade option. The timezone is upgraded along with the DB upgrade. Select the listener. In this case the listener port is a non-default port. Upgrade summary and progress Post upgrade summary Run the post upgrade script

@postupgrade_fixups.sql

DB components versions and status after the upgrade

COMP_ID    COMP_NAME                                VERSION    STATUS
---------- ---------------------------------------- ---------- --------
CATALOG    Oracle Database Catalog Views            12.2.0.1.0 VALID
CATPROC    Oracle Database Packages and Types       12.2.0.1.0 VALID
OWM        Oracle Workspace Manager                 12.2.0.1.0 VALID
CONTEXT    Oracle Text                              12.2.0.1.0 VALID
XDB        Oracle XML Database                      12.2.0.1.0 VALID

Timezone file version after upgrade

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_26.dat              26          0

As mentioned in 11.2.0.4 to 12.2.0.1 RAC upgrade post, during the upgrade new user accounts are added to DB with default passwords. Change the default password of these user accounts.

Check the registry history view for the upgrade entry.

ACTION_TIME                    ACTION               NAMESPACE   VERSION            ID COMMENTS
------------------------------ -------------------- ----------- ---------- ---------- ----------------------------------
11-MAR-15 06.48.08.733830 PM   APPLY                SERVER      11.2.0.3            0 Patchset 11.2.0.2.0
12-MAR-15 11.54.22.212281 AM   APPLY                SERVER      11.2.0.3           13 PSU 11.2.0.3.13
16-MAR-15 03.09.19.557408 PM   VIEW INVALIDATE                                8289601 view invalidation
16-MAR-15 03.09.19.634383 PM   UPGRADE              SERVER      11.2.0.4.0            Upgraded from 11.2.0.3.0
16-MAR-15 03.09.43.891843 PM   APPLY                SERVER      11.2.0.4            0 Patchset 11.2.0.2.0
10-AUG-15 04.28.56.297048 PM   APPLY                SERVER      11.2.0.4            7 PSU 11.2.0.4.7
19-OCT-17 04.22.55.222389 PM   APPLY                SERVER      11.2.0.4       171017 PSU 11.2.0.4.171017
                               BOOTSTRAP            DATAPATCH   12.2.0.1              RDBMS_12.2.0.1.0_LINUX.X64_170125
19-OCT-17 06.40.33.227204 PM   VIEW INVALIDATE                                8289601 view invalidation
19-OCT-17 06.42.41.868410 PM   UPGRADE              SERVER      12.2.0.1.0            Upgraded from 11.2.0.4.0

Upgrade the database and ASM compatible parameters

alter system set compatible='12.2.0.1.0' scope=spfile sid='*';

srvctl stop database -d std11g2

alter diskgroup flash SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup DATA  SET attribute 'compatible.asm'='12.2.0.1.0';
alter diskgroup flash set attribute 'compatible.rdbms'='12.2.0.1.0';
alter diskgroup data set attribute 'compatible.rdbms'='12.2.0.1.0';

GROUP_NUMBER NAME                 VALUE
------------ -------------------- ---------------
           1 compatible.asm       12.2.0.1.0
           1 compatible.rdbms     12.2.0.1.0
           2 compatible.asm       12.2.0.1.0
           2 compatible.rdbms     12.2.0.1.0

srvctl start database -d std11g2

Finally run orachk and cluvfy for check post upgrade state

./orachk -u -o post

cluvfy stage -post hacfg

Verifying Oracle Restart Integrity ...PASSED
Verifying OLR Integrity ...PASSED

Post-check for Oracle Restart configuration was successful.

Useful metalink notes
ACFS Drivers Install reports CLSRSC-400: A system reboot is required to continue installing [ID 2025056.1]
Patches to apply before upgrading Oracle GI and DB to 12.2.0.1 [ID 2180188.1]
12.2 Grid Infrastructure Installation: What's New [ID 2024946.1]
How to Upgrade to/Downgrade from Grid Infrastructure 12.2 and Known Issues [ID 2240959.1]

Related Posts
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 12.1.0.2 to 12.2.0.1
Upgrading Single Instance on ASM from 11.2.0.3 to 11.2.0.4
Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2