Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 19c (19.6) on RHEL 7

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.DATA.dg               ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-7-244
ora.FRA.dg                ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-7-244
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             ip-172-31-7-244
ora.asm                   ora.asm.type              ONLINE             ONLINE             ip-172-31-7-244
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             ip-172-31-7-244
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-7-244
ora.ons                   ora.ons.type              OFFLINE            OFFLINE
ora.test.db               ora.database.type         ONLINE             ONLINE             ip-172-31-7-244

$ crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]

$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [11.2.0.4.0]

opatch lspatches

30503372;OJVM PATCH SET UPDATE 11.2.0.4.200114
30298532;Database Patch Set Update : 11.2.0.4.200114 (30298532)
29938455;OCW Patch Set Update : 11.2.0.4.191015 (29938455)

mkdir -p /opt/app/oracle/product/19.x.0/grid
unzip giGold_19.6.zip -d /opt/app/oracle/product/19.x.0/grid

./runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: ip-172-31-7-244:/usr,ip-172-31-7-244:/var,ip-172-31-7-244:/etc,ip-172-31-7-244:/sbin,ip-172-31-7-244:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.

Failures were encountered during execution of CVU verification request "stage -pre hacfg".

srvctl stop database -d test

$ ./gridSetup.sh

/opt/app/oracle/product/19.x.0/grid/rootupgrade.sh
Performing root user operation.
...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-7-244/crsconfig/roothas_2020-01-21_04-17-39PM.log
2020/01/21 16:17:40 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2020/01/21 16:17:42 CLSRSC-363: User ignored prerequisites during installation
2020/01/21 16:17:43 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.
2020/01/21 16:17:45 CLSRSC-595: Executing upgrade step 3 of 12: 'GenSiteGUIDs'.
2020/01/21 16:17:45 CLSRSC-595: Executing upgrade step 4 of 12: 'SetupOSD'.
2020/01/21 16:17:46 CLSRSC-595: Executing upgrade step 5 of 12: 'PreUpgrade'.

ASM has been upgraded and started successfully.

2020/01/21 16:21:40 CLSRSC-595: Executing upgrade step 6 of 12: 'UpgradeAFD'.
2020/01/21 16:21:40 CLSRSC-595: Executing upgrade step 7 of 12: 'UpgradeOLR'.
clscfg: EXISTING configuration version 0 detected.
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
2020/01/21 16:21:48 CLSRSC-595: Executing upgrade step 8 of 12: 'UpgradeOCR'.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-7-244 successfully pinned.
2020/01/21 16:21:51 CLSRSC-595: Executing upgrade step 9 of 12: 'CreateOHASD'.
2020/01/21 16:21:52 CLSRSC-595: Executing upgrade step 10 of 12: 'ConfigOHASD'.
2020/01/21 16:21:52 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2020/01/21 16:22:40 CLSRSC-595: Executing upgrade step 11 of 12: 'UpgradeSIHA'.

ip-172-31-7-244     2020/01/21 16:24:11     /opt/app/oracle/crsdata/ip-172-31-7-244/olr/backup_20200121_162411.olr     1748936676

ip-172-31-7-244     2020/01/21 14:25:40     /opt/app/oracle/product/11.2.0/grid/cdata/ip-172-31-7-244/backup_20200121_142540.olr     -
2020/01/21 16:24:11 CLSRSC-595: Executing upgrade step 12 of 12: 'InstallACFS'.
2020/01/21 16:26:36 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

$ crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]

$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [19.0.0.0.0]

cluvfy stage -post hacfg

Verifying Oracle Restart Integrity ...PASSED
Verifying OLR Integrity ...PASSED

Post-check for Oracle Restart configuration was successful.

# groupadd racdba
# usermod -g oinstall -G dba,oper,asmoper,asmdba,asmadmin,backupdba,dgdba,kmdba,racdba oracle
# id oracle
uid=501(oracle) gid=502(oinstall) groups=502(oinstall),501(dba),503(oper),504(asmoper),505(asmdba),506(asmadmin),1001(backupdba),1002(dgdba),1003(kmdba),1004(racdba)

mkdir -p /opt/app/oracle/product/19.x.0/dbhome_1
unzip ohGold_19.6.zip -d /opt/app/oracle/product/19.x.0/dbhome_1

# /opt/app/oracle/product/19.x.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/oracle/product/19.x.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Oracle Trace File Analyzer (TFA - Standalone Mode) is available at :
    /opt/app/oracle/product/19.x.0/dbhome_1/bin/tfactl

Note :
1. tfactl will use TFA Service if that service is running and user has been granted access
2. tfactl will configure TFA Standalone Mode only if user has no access to TFA Service or TFA is not installed

$ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
19.0.0.0.0 Build: 1 on 2020-01-22T11:03:49

Upgrade-To version: 19.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  TEST
     Container Name:  Not Applicable in Pre-12.1 database
       Container ID:  Not Applicable in Pre-12.1 database
            Version:  11.2.0.4.0
     DB Patch Level:  PSU 11.2.0.4.200114
         Compatible:  11.2.0.4.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  14
  Database log mode:  ARCHIVELOG
           Readonly:  FALSE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Enterprise Manager Repository   [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  1.  (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

      The database contains 2 objects in the recycle bin.

      The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
  2.  Remove the EM repository.

      - Copy the $ORACLE_HOME/rdbms/admin/emremove.sql script from the target
      19 ORACLE_HOME into the source 11.2.0.4.0 ORACLE_HOME.

      Step 1: If database control is configured, stop EM Database Control,
      using the following command

        $> emctl stop dbconsole

      Step 2: Connect to the database using the SYS account AS SYSDBA

        SET ECHO ON;
        SET SERVEROUTPUT ON;
        @emremove.sql

      Without the set echo and serveroutput commands, you will not be able to
      follow the progress of the script.

      The database has an Enterprise Manager Database Control repository.

      Starting with Oracle Database 12c, the local Enterprise Manager Database
      Control does not exist anymore. The repository will be removed from your
      database during the upgrade.  This step can be manually performed before
      the upgrade to reduce downtime.

  3.  (AUTOFIXUP) Directly grant ADMINISTER DATABASE TRIGGER privilege to the
      owner of the trigger or drop and re-create the trigger with a user that
      was granted directly with such. You can list those triggers using: SELECT
      OWNER, TRIGGER_NAME FROM DBA_TRIGGERS WHERE
      TRIM(BASE_OBJECT_TYPE)='DATABASE' AND OWNER NOT IN (SELECT GRANTEE FROM
      DBA_SYS_PRIVS WHERE PRIVILEGE='ADMINISTER DATABASE TRIGGER').

      There is one or more database triggers whose owner does not have the
      right privilege on the database.

      The creation of database triggers must be done by users granted with
      ADMINISTER DATABASE TRIGGER privilege. Privilege must have been granted
      directly.

  4.  (AUTOFIXUP) Gather statistics on fixed objects prior the upgrade.

      None of the fixed object tables have had stats collected.

      Gathering statistics on fixed objects, if none have been gathered yet, is
      recommended prior to upgrading.

      For information on managing optimizer statistics, refer to the 11.2.0.4
      Oracle Database Performance Tuning Guide.

  INFORMATION ONLY
  ================
  5.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      TEMP                                20 MB       150 MB
      UNDOTBS1                           380 MB       411 MB

      Minimum tablespace sizes for upgrade are estimates.

  6.  Check the Oracle Backup and Recovery User's Guide for information on how
      to manage an RMAN recovery catalog schema.

      If you are using a version of the recovery catalog schema that is older
      than that required by the RMAN client version, then you must upgrade the
      catalog schema.

      It is good practice to have the catalog schema the same or higher version
      than the RMAN client version you are using.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database TEST
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following

    SQL>@/opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade_fixups.sql

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  7.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 14 and the target 19 release
      ships with time zone file version 32.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 19 Oracle Database
      Globalization Support Guide.

  8.  (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  9.  Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 11.2.0.4
      Oracle Database Performance Tuning Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database TEST
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following

    SQL>@/opt/app/oracle/cfgtoollogs/test/preupgrade/postupgrade_fixups.sql


==================
PREUPGRADE SUMMARY
==================
  /opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade.log
  /opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade_fixups.sql
  /opt/app/oracle/cfgtoollogs/test/preupgrade/postupgrade_fixups.sql

Execute fixup scripts as indicated below:

Before upgrade:

Log into the database and execute the preupgrade fixups
@/opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade_fixups.sql

After the upgrade:

Log into the database and execute the postupgrade fixups
@/opt/app/oracle/cfgtoollogs/test/preupgrade/postupgrade_fixups.sql

Preupgrade complete: 2020-01-22T11:03:49

SQL> @/opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade_fixups.sql
Executing Oracle PRE-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 1
Generated on:            2020-01-22 11:03:47

For Source Database:     TEST
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    1.  purge_recyclebin          YES         None.
    2.  em_present                YES         None.
    3.  trgowner_no_admndbtrg     YES         None.
    4.  pre_fixed_objects         YES         None.
    5.  tablespaces_info          NO          Informational only.
                                              Further action is optional.
    6.  rman_recovery_version     NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database is not ready
for upgrade.  To resolve the outstanding issues, start by reviewing
the preupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.

ORA-38880 signalled during: ALTER DATABASE MOUNT

38880, 00000, "Cannot advance compatibility from %s to %s due to guaranteed restore points"
// *Cause: Flashback database cannot undo the advance of database
//         compatibility.  Therefore, one cannot advance the compatibility
//         of the database while there are guaranteed restore points in the
//         database.
// *Action: Drop all guaranteed restore points first and retry, or delay
//          the advance of database compatibility to a later time.

@/opt/app/oracle/cfgtoollogs/test/preupgrade/postupgrade_fixups.sql

Executing Oracle POST-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 1
Generated on:            2020-01-22 11:13:31

For Source Database:     TEST
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    3.  old_time_zones_exist      YES         None.
    4.  post_dictionary           YES         None.
    5.  post_fixed_objects        NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database upgrade is not
fully complete.  To resolve the outstanding issues, start by reviewing
the postupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.

Session altered.

COMP_NAME                           STATUS     VERSION         VERSION_FULL
----------------------------------- ---------- --------------- ---------------
Oracle Database Catalog Views       VALID      19.0.0.0.0      19.6.0.0.0
Oracle Database Packages and Types  VALID      19.0.0.0.0      19.6.0.0.0
Oracle Real Application Clusters    OPTION OFF 19.0.0.0.0      19.6.0.0.0
Oracle Workspace Manager            VALID      19.0.0.0.0      19.6.0.0.0
Oracle Text                         VALID      19.0.0.0.0      19.6.0.0.0
Oracle XML Database                 VALID      19.0.0.0.0      19.6.0.0.0

ACTION_TIME                    ACTION     NAMESPACE  VERSION         COMMENTS
------------------------------ ---------- ---------- --------------- ------------------------------------------------------------
21-JAN-20 03.30.26.273084 PM   APPLY      SERVER     11.2.0.4        Patchset 11.2.0.2.0
21-JAN-20 04.07.12.218350 PM   APPLY      SERVER     11.2.0.4        PSU 11.2.0.4.200114
                               BOOTSTRAP  DATAPATCH  19              RDBMS_19.6.0.0.0DBRU_LINUX.X64_191217
22-JAN-20 11.50.55.143749 AM   RU_APPLY   SERVER     19.0.0.0.0      Patch applied on 19.6.0.0.0: Release_Update - 191217155004
22-JAN-20 11.52.39.733756 AM   UPGRADE    SERVER     19.0.0.0.0      Upgraded from 11.2.0.4.0 to 19.6.0.0.0

SQL> select * from v$timezone_file;

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_32.dat              32          0

SQL> alter system set compatible='19.0.0.0.0' scope=spfile;
shutdown immediate;

SQL> alter diskgroup FRA SET attribute 'compatible.asm'='19.0.0.0.0';
SQL> alter diskgroup DATA  SET attribute 'compatible.asm'='19.0.0.0.0';
SQL> alter diskgroup fra set attribute 'compatible.rdbms'='19.0.0.0.0';
SQL> alter diskgroup data set attribute 'compatible.rdbms'='19.0.0.0.0';

SQL> select g.name,a.name,a.value from v$asm_diskgroup g, v$asm_attribute a where g.group_number=a.group_number and a.name like '%compat%';

NAME                 NAME                 VALUE
-------------------- -------------------- --------------------
DATA                 compatible.asm       19.0.0.0.0
DATA                 compatible.rdbms     19.0.0.0.0
FRA                  compatible.asm       19.0.0.0.0
FRA                  compatible.rdbms     19.0.0.0.0

2020/01/23 12:33:30 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2020/01/23 12:33:33 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.

CRS-6705: Oracle Clusterware Release Version ('19.0.0.0.0') does not match Software Version ('11.2.0.4.0'). Oracle Clusterware cannot be started.

CRS-4000: Command Start failed, or completed with errors.

2020/01/23 12:33:50 CLSRSC-318: Failed to start Oracle OHASD service

Died at /u01/app/oracle/product/19.0.0.0/grid/crs/install/crsupgrade.pm line 6269.