Upgrading RHEL 7 OS in a 11gR2 (11.2.0.4) and 12cR1 (12.1.0.2) Oracle Restart Environments

As per Oracle documentation "You must relink the Oracle Clusterware and Oracle ASM binaries every time you apply an operating system patch or after an operating system upgrade." There are two earlier posts which shows upgrading of RHEL5 and RHEL6 OS in a 11gR2 RAC environment. This post shows relinking of GI and Oracle homes after upgrading of RHEL 7 in 11.2.0.4 and 12.1.0.2 single instance (Oracle restart) environments.
1. The current RHEL version is 7.1 and the kernel is

uname -r
3.10.0-123.el7.x86_64

2. Before the OS upgrade stop the HAS and disable the auto start on reboot.

crsctl stop has
crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.

3. Comment the entries in oracle-ohasd.service

cat /etc/systemd/system/oracle-ohasd.service
# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup

[Unit]
#Description=Oracle High Availability Services
#After=syslog.target

[Service]
#ExecStart=/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
#Type=simple
#Restart=always

[Install]
#WantedBy=multi-user.target graphical.target

4. Upgrade the RHEL 7 OS. Kernel and RHEL7 version after the upgrade are

uname -r
3.10.0-327.el7.x86_64
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)

5. Uncomment the entires in oracle-ohasd.service file and enable auto start of HAS

crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.

Until this point the steps are common for both 11.2.0.4 and 12.1.0.2.

Relinking 11.2.0.4 GI and Oracle Homes
1.Relinking Oracle Home as oracle user. Make sure the relink binary in the PATH is from Oracle home

which relink
/opt/app/oracle/product/11.2.0/dbhome_4/bin/relink

Run relink all

cd $ORACLE_HOME/bin
./relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_4/install/relink.log

At the end of the relink the log file have the following

test ! -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle ||\
   mv -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_4/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle

2. Relinkning GI Home
Set ORACLE_HOME=$GI_HOME
As root unlock the GI Home. Using the RHEL 7 provided perl version will result in following error

# perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/app/oracle/product/11.2.0/grid/crs/install) at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
Compilation failed in require at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.

Solution for this was mentioned on the changing of hostname post which is to use oracle provided perl

# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Successfully unlock /opt/app/oracle/product/11.2.0/grid

3. As grid user run the relink

$ORACLE_HOME/bin/relink
writing relink log to: /opt/app/oracle/product/11.2.0/grid/install/relink.log

The log file will have the following entry at the end

tail /opt/app/oracle/product/11.2.0/grid/install/relink.log

log tail
test ! -f /opt/app/oracle/product/11.2.0/grid/bin/oracle ||\
   mv -f /opt/app/oracle/product/11.2.0/grid/bin/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracleO
mv /opt/app/oracle/product/11.2.0/grid/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/grid/bin/oracle

4. As root patch the GI Home

# cd $ORACLE_HOME/rdbms/install
# ./rootadd_rdbms.sh

$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -patch
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Broadcast message from systemd-journald@rhel7s.codegen.net (Thu 2016-02-18 16:50:14 GMT):
dracut[16456]: dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed

Message from syslogd@rhel7s at Feb 18 16:50:14 ...
 dracut:dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed
ACFS driver install actions failed
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
Timed out waiting for ohasd to start.

During the patching the HAS stack start will fail, this is expected.
5. Reboot the host and HAS stack will be up and running as before

Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.FRA.dg                          ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
ora.std11g2.db                      ora.database.type              ONLINE             ONLINE             rhel7s

Relinking 12.1.0.2 GI and Oracle Homes
1. Relinking Oracle home as oracle user. Make sure the relink binary in PATH is from Oracle home. Run relink all to relink the oracle home.

relink all
writing relink log to: /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log

The log file end with the following

tail /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log
test ! -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle ||\
   mv -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracleO
mv /opt/app/oracle/product/12.1.0/dbhome_2/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle

2. As root unlock the GI Home by using the $GI_HOME/crs/install/roothas.sh. Unlike 11.2.0.4 there are no issues in unlocking GI home in 12c

./roothas.sh -unlock
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
2016/02/19 18:52:17 CLSRSC-347: Successfully unlock /opt/app/oracle/product/12.1.0/grid

3. As grid user run the relink.

which relink
/opt/app/oracle/product/12.1.0/grid/bin/relink
$ relink
writing relink log to: /opt/app/oracle/product/12.1.0/grid/install/relink.log

The log file will end with the following

tail /opt/app/oracle/product/12.1.0/grid/install/relink.log

test ! -f /opt/app/oracle/product/12.1.0/grid/bin/oracle ||\
   mv -f /opt/app/oracle/product/12.1.0/grid/bin/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracleO
mv /opt/app/oracle/product/12.1.0/grid/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/grid/bin/oracle

4. As root patch the GI Home

# $GI_HOME/rdbms/install/rootadd_rdbms.sh

./roothas.sh -patch
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
2016/02/19 18:59:50 CLSRSC-199: Timed out waiting for OHASD to start

The command '/opt/app/oracle/product/12.1.0/grid/perl/bin/perl -I/opt/app/oracle/product/12.1.0/grid/perl/lib -I/opt/app/oracle/product/12.1.0/grid/crs/install /opt/app/oracle/product/12.1.0/grid/crs/install/roothas.pl -patch' execution failed

5. Similar to 11.2.0.4, start of the HAS stack will fail, this is expected. Reboot the host and HAS will be up and running.

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.DATA.dg               ora.diskgroup.type        ONLINE             ONLINE             rhel7
ora.FRA.dg                ora.diskgroup.type        ONLINE             ONLINE             rhel7
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             rhel7
ora.asm                   ora.asm.type              ONLINE             ONLINE             rhel7
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             rhel7
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             rhel7
ora.ons                   ora.ons.type              OFFLINE            OFFLINE
ora.se2db.db              ora.database.type         ONLINE             ONLINE             rhel7

This concludes the steps for relinking GI and Oracle homes after RHEL 7 OS upgrade.

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment
Upgrading ASMLib and OS in 11gR1 RAC Environment