Quantcast
Channel: A! Help
Viewing all 315 articles
Browse latest View live
↧

Local Undo in 12.2 PDB in RAC

February 14, 2019, 12:00 am
$
0
0
In 12.1 all PDBs shared a single undo tablespace. With 12.2 oracle introduced local undo, where each container has it's own undo tablespace on each instance it is opened. Which mode is currently used could be checked with following.
SQL> select * from database_properties where property_name='LOCAL_UNDO_ENABLED';

PROPERTY_NAME        PROPERTY_VALUE  DESCRIPTION
-------------------- --------------- ------------------------------
LOCAL_UNDO_ENABLED   TRUE            true if local undo is enabled
It is possible to move from local undo mode to shared undo mode and vice versa.

In RAC instances this means there would be undo tablespaces per instance and per PDB. However, due to automatic naming convention used this may not be obvious until datafile path is examined. Below example was on a base release of 12.2 with Patch 26124078 applied [ID 2497491.1].

RAC CDB consists of two instances.
SQL> select inst_id,instance_name from gv$instance;

   INST_ID INSTANCE_NAME
---------- ----------------
         1 cdb12c1
         2 cdb12c2
Currently there are no PDBs except the seed.
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
The current undo tablespaces are their datafile locations are as follows
    CON_ID TBLSPACE   NAME
---------- ---------- --------------------------------------------------------------------------------
         1 UNDOTBS1   +DATA/ORCL/DATAFILE/undotbs1.279.1000054779
         1 UNDOTBS2   +DATA/ORCL/DATAFILE/undotbs2.289.1000054817
         2 UNDOTBS1   +DATA/ORCL/81B657D10A1B2EE8E0535D00A8C08053/DATAFILE/undotbs1.295.1000054779
Each instance has it's own undo tablespace which is the expected setup on RAC.
SQL> select inst_id,con_id,name,value from gv$parameter where name='undo_tablespace' order by 1;

   INST_ID     CON_ID NAME                      VALUE
---------- ---------- ------------------------- --------------------
         1          1 undo_tablespace           UNDOTBS1
         2          1 undo_tablespace           UNDOTBS2


When creating a PDB and opening it which will create the undo tablespace for remote instance's PDB.
SQL> create pluggable database pdb3 admin user pdb1 identified by pdb1 default tablespace users ;

Pluggable database created.

SQL> alter pluggable database pdb3 open instances=all;

Pluggable database altered.
The alert log on the remote instance shows the creation of this local undo tablespace for the PDB on that instance.
PDB3(3):Opening pdb with no Resource Manager plan active
PDB3(3):CREATE SMALLFILE UNDO TABLESPACE undo_2 DATAFILE '+DATA' SIZE 193986560 AUTOEXTEND ON NEXT 5242880 MAXSIZE 34359721984 ONLINE
2019-02-13T11:35:09.410262+00:00
PDB3(3):[4337] Successfully onlined Undo Tablespace 5.
PDB3(3):Completed: CREATE SMALLFILE UNDO TABLESPACE undo_2 DATAFILE '+DATA' SIZE 193986560 AUTOEXTEND ON NEXT 5242880 MAXSIZE 34359721984 ONLINE
Pluggable database PDB3 opened read write
Login to the PDB and query the instance specific local undo tablespaces.
SQL> alter session set container=pdb3;

Session altered.

SQL> select inst_id,con_id,name,value from gv$parameter where name='undo_tablespace' order by 1;

   INST_ID     CON_ID NAME                      VALUE
---------- ---------- ------------------------- ----------
         1          3 undo_tablespace           UNDOTBS1
         2          3 undo_tablespace           UNDO_2
It appears that the local undo tablespace for PDB on instance 1 uses the same as the CDB$root's undo tablespaces "UNDOTBS1" this is not the case. The datafile for this undo tablesapce shows it's different to root's undo tablespace.
SQL>  select t.con_id,t.name tblspace,d.name from  v$tablespace t, v$datafile d where  d.con_id=t.con_id and t.ts#=d.ts# and t.name like 'UND%' order by 1,2;

    CON_ID TBLSPACE   NAME
---------- ---------- -------------------------------------------------------------------------------
         3 UNDOTBS1   +DATA/ORCL/81C59E3616323828E0535D00A8C059DD/DATAFILE/undotbs1.288.1000121653
         3 UNDO_2     +DATA/ORCL/81C59E3616323828E0535D00A8C059DD/DATAFILE/undo_2.285.1000121707
This shows that each PDB in each instance gets a local UNDO tablespace.

Useful metalink notes
Undo Modes in 12.2 Multitenant Databases - Local and Shared Modes [ID 2169828.1]
Different levels of encrypted undo tablespace across instances PDB In a 12.2 RAC - OCI . [ID 2497491.1]
↧
Search

Changing ORACLE_BASE, ORACLE_HOME and GI_HOME in a Oracle Restart Setup

March 7, 2019, 7:31 am
$
0
0
This post gives the steps for changing the location of ORACLE_BASE, ORACLE_HOME, oracle Inventory and Grid Infrastructure in a Oracle restart setup. The current and new paths for these items are given in the table below.

ItemCurrent LocationFuture Location
ORACLE_BASE/opt/app/oracle/u01/app/oracle
ORACLE_HOME/opt/app/oracle/product/11.2.0/dbhome_1/u01/app/oracle/product/11.2.0/dbhome_1
GI_HOME/opt/app/oracle/product/12.1.0/grid/u01/app/oracle/product/12.1.0/grid
Oracle Inventory/opt/app/oraInventory/u01/app/oraInventory

As could be seen from the locations the GI home is a 12.1 while the DB runs out of a 11.2 (11.2.0.4) home.

1. It is assumed the mount point /u01 exists. Create the two base directories first, that is the Oracle base and oracle inventory directories and set the necessary permissions.
# cd /u01/

# mkdir -p app/oracle
# mkdir -p app/oraInventory

# chmod 775 oracle
# chmod 770 oraInventory

# chown oracle:oinstall oracle
# chown grid:oinstall oraInventory
2. Stop the database and the HA service. 
srvctl stop database -d westdb
crsctl stop has
3. Detach the current GI home from the inventory.
[grid@west bin]$ ./runInstaller -silent -waitforcompletion -detachHome ORACLE_HOME='/opt/app/oracle/product/12.1.0/grid'
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4097 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'DetachHome' was successful.
Check the GI home was removed from the inventory by checking in the inventory.xml
<HOME NAME="OraGI12Home1" LOC="/opt/app/oracle/product/12.1.0/grid" TYPE="O" IDX="1"REMOVED="T"/>
4. Create the future GI Home location.
mkdir -p /u01/app/oracle/product/12.1.0/
Copy the current grid folder to the future location.
cd /opt/app/oracle/product/12.1.0
cp -pR grid /u01/app/oracle/product/12.1.0/
5. Clone the GI Home in the new location. Pass the new Oracle base, GI home and oracle inventory locations to the clone script.
cd /u01/app/oracle/product/12.1.0/grid/clone/bin
$ perl clone.pl -silent ORACLE_BASE=/u01/app/oracle ORACLE_HOME=/u01/app/oracle/product/12.1.0/grid 
ORACLE_HOME_NAME=OraGI12Home1 INVENTORY_LOCATION=/u01/app/oraInventory CRS=true

./runInstaller -clone -waitForCompletion  "ORACLE_BASE=/u01/app/oracle""ORACLE_HOME=/u01/app/oracle/product/12.1.0/grid""ORACLE_HOME_NAME=OraGI12Home1""INVENTORY_LOCATION=/u01/app/oraInventory" -silent -paramFile /u01/app/oracle/product/12.1.0/grid/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 17853 MB    Passed
Checking swap space: must be greater than 500 MB.   Actual 4097 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-03-07_06-42-57PM. Please wait ...You can find the log of this install session at:
 /u01/app/oraInventory/logs/cloneActions2019-03-07_06-42-57PM.log
..................................................   5% Done.
..................................................   10% Done.
..................................................   15% Done.
..................................................   20% Done.
..................................................   25% Done.
..................................................   30% Done.
..................................................   35% Done.
..................................................   40% Done.
..................................................   45% Done.
..................................................   50% Done.
..................................................   55% Done.
..................................................   60% Done.
..................................................   65% Done.
..................................................   70% Done.
..................................................   75% Done.
..................................................   80% Done.
..................................................   85% Done.
..........Could not backup file /u01/app/oracle/product/12.1.0/grid/root.sh to /u01/app/oracle/product/12.1.0/grid/root.sh.ouibak
Could not backup file /u01/app/oracle/product/12.1.0/grid/rootupgrade.sh to /u01/app/oracle/product/12.1.0/grid/rootupgrade.sh.ouibak

Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.
The cloning of OraGI12Home1 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2019-03-07_06-42-57PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   95% Done.

As a root user, execute the following script(s):
        1. /u01/app/oraInventory/orainstRoot.sh
        2. /u01/app/oracle/product/12.1.0/grid/root.sh



..................................................   100% Done.
You have new mail in /var/spool/mail/grid
6. After running the orainstRoot.sh the inventory location gets updated in the /etc/oraInst.loc
/u01/app/oraInventory/orainstRoot.sh

cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
7. Running the root.sh will generate a log file which will have commands to run to create a Oracle restart setup or a cluster setup.
Check /u01/app/oracle/product/12.1.0/grid/install/root_west.domain.net_2019-03-07_18-45-23.log for the output of root script

# tail -f /u01/app/oracle/product/12.1.0/grid/install/root_west.domain.net_2019-03-07_18-45-23.log
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/oracle/product/12.1.0/grid/perl/bin/perl -I/u01/app/oracle/product/12.1.0/grid/perl/lib -I/u01/app/oracle/product/12.1.0/grid/crs/install /u01/app/oracle/product/12.1.0/grid/crs/install/roothas.pl
8. Before running the command mentioned in the log file, the existing Oracle restart configuration need to be de-configured. If not following error will occur.
Using configuration parameter file: /u01/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
2019/03/07 18:46:12 CLSRSC-350: Cannot configure two CRS instances on the same cluster

2019/03/07 18:46:14 CLSRSC-352: CRS is already configured on this node for the CRS home location /opt/app/oracle/product/12.1.0/grid
To de-configure run the following command.
# /u01/app/oracle/product/12.1.0/grid/perl/bin/perl roothas.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
2019/03/07 18:47:50 CLSRSC-337: Successfully deconfigured Oracle Restart stack
9. Run the command to create the Oracle restart setup.
# /u01/app/oracle/product/12.1.0/grid/perl/bin/perl -I/u01/app/oracle/product/12.1.0/grid/perl/lib -I/u01/app/oracle/product/12.1.0/grid/crs/install /u01/app/oracle/product/12.1.0/grid/crs/install/roothas.pl
Using configuration parameter file: /u01/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node west successfully pinned.
2019/03/07 18:48:26 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'

west     2019/03/07 13:18:58     /u01/app/oracle/product/12.1.0/grid/cdata/west/backup_20190307_131858.olr     459864538
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'west'
CRS-2673: Attempting to stop 'ora.evmd' on 'west'
CRS-2677: Stop of 'ora.evmd' on 'west' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'west' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/03/07 18:50:11 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
10. Check the GI home added to inventory in new inventory location.
<HOME NAME="OraGI12Home1" LOC="/u01/app/oracle/product/12.1.0/grid" TYPE="O" IDX="1" CRS="true"/>
11. Add listener and ASM to the Oracle restart config.
srvctl add listener -l listener -o /u01/app/oracle/product/12.1.0/grid -p 1521
srvctl start listener -l listener
srvctl add asm -l listener -p +data/asm/ASMPARAMETERFILE/REGISTRY.253.881942589 -d "/dev/sd*"
srvctl start asm

 crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       west                     STABLE
ora.FLASH.dg
               ONLINE  ONLINE       west                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       west                     STABLE
ora.asm
               ONLINE  ONLINE       west                     Started,STABLE
ora.ons
               OFFLINE OFFLINE      west                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       west                     STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       west                     STABLE
--------------------------------------------------------------------------------


12. Next step is to move the Oracle home to new location. As this is a role separated setup, write permission must be granted to oracle user for admin and diag directories inside Oracle base.
cd $ORACLE_BASE
chmod 775 admin diag
Create the new Oracle home location and copy the current Oracle home (dbhome_1) to new location.
# cd /u01/app/oracle/product
# mkdir -p 11.2.0

# cd /opt/app/oracle/product/11.2.0
# cp -pR dbhome_1 /u01/app/oracle/product/11.2.0/
13. Clone the DB Home
$ cd /u01/app/oracle/product/11.2.0/dbhome_1/clone/

/u01/app/oracle/product/11.2.0/dbhome_1/perl/bin/perl clone.pl ORACLE_BASE="/u01/app/oracle/" ORACLE_HOME="/u01/app/oracle/product/11.2.0/dbhome_1" OSDBA_GROUP=dba OSOPER_GROUP=oper -defaultHomeName

./runInstaller -clone -waitForCompletion  "ORACLE_BASE=/u01/app/oracle/""ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1""oracle_install_OSDBA=dba""oracle_install_OSOPER=oper" -defaultHomeName  -defaultHomeName "CRS=true" -silent -noConfig -nowait
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4097 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-03-07_07-21-14PM. Please wait ...Oracle Universal Installer, Version 11.2.0.4.0 Production
Copyright (C) 1999, 2013, Oracle. All rights reserved.

You can find the log of this install session at:
 /u01/app/oraInventory/logs/cloneActions2019-03-07_07-21-14PM.log
.................................................................................................... 100% Done.

Installation in progress (Thursday, 7 March 2019 19:21:31 o'clock IST)
...........................................................................                                                     75% Done.
Install successful

Linking in progress (Thursday, 7 March 2019 19:21:43 o'clock IST)
Link successful

Setup in progress (Thursday, 7 March 2019 19:22:43 o'clock IST)
Setup successful

End of install phases.(Thursday, 7 March 2019 19:23:08 o'clock IST)
WARNING:
The following configuration scripts need to be executed as the "root" user.
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts

The cloning of OraHome1 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2019-03-07_07-21-14PM.log' for more details.
Check the inventory is updated with new DB home location.
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0/dbhome_1" TYPE="O" IDX="2"/>
14. Update the /etc/oratab with the new Oracle home location. Add the database to the Oracle restart configuration.
srvctl add database -d westdb -o /u01/app/oracle/product/11.2.0/dbhome_1 -p +DATA/westdb/spfilewestdb.ora -a "data,flash"
srvctl start database -d westdb
Certain init parameters will refer to paths under previous Oracle base. Change them to reflect the current Oracle base. 
show parameter diagnostic_dest
diagnostic_dest                      string      /opt/app/oracle

alter system set diagnostic_dest='/u01/app/oracle' scope=both;
alter system set audit_file_dest='/u01/app/oracle/admin/westdb/adump' scope=spfile;

mkdir -p u01/app/oracle/admin/westdb/adump
15. Stop the database and the Oracle restart stack. Rename the old base location (/opt/app/) to something temporary (/opt/appx/) and start the database. If all steps are followed there shouldn't be any references to previous location. Current oracle base directory could also be found out with orabase.
$ orabase
/u01/app/oracle
Once certain no references to old locations remain those could be removed.
↧
↧

RAC DB on OCI PaaS

March 18, 2019, 7:04 am
$
0
0
Unlike other cloud vendors, in OCI it is possible to create a 2 node RAC without any third party software. The minimum shape needed for creating a 2 node RAC for virtual machine type is VM.Standard(x).2. This will automatically enables the node count spinner allowing it to be changed to 2 nodes (currently the max number of nodes). Additionally cluster name could be specified. Rest of the steps are similar to creating a single instance DB.
For this post 18.3 RAC DB was created in private subnet. The Database specifications are shown below.
The database details are shown below.
An app server on a public subnet on the same VCN is able to resolve the SCAN without any other user configurations. 
nslookup rac-scan.londbsubnet.lonvcn.oraclevcn.com
Server:         169.254.169.254
Address:        169.254.169.254#53

Non-authoritative answer:
Name:   rac-scan.londbsubnet.lonvcn.oraclevcn.com
Address: 172.21.0.7
Name:   rac-scan.londbsubnet.lonvcn.oraclevcn.com
Address: 172.21.0.8
Name:   rac-scan.londbsubnet.lonvcn.oraclevcn.com
Address: 172.21.0.6
The hostnames will be suffixed with node number when the cluster is created.
The subnet CIDR block is treated as the public network and second 192.168.* network is created for private interconnect.
oifcfg getif
eth0  172.21.0.0  global  public
eth1  192.168.16.0  global  cluster_interconnect,asm
The /etc/oratab files on all nodes are empty of any ASM or DB instance informations. These need to be added manually if needed.

The RAC configuration is role separated. Following roles exists for oracle and grid user. There are no separate groups for backupdba, kmdba, dgdba or racdba.
[opc@rac1 ~]$ id oracle
uid=101(oracle) gid=1001(oinstall) groups=1001(oinstall),1006(asmdba),1003(dba),1002(dbaoper)
[opc@rac1 ~]$ id grid
uid=102(grid) gid=1001(oinstall) groups=1001(oinstall),1006(asmdba),1005(asmoper),1004(asmadmin),1002(dbaoper)
There's no user equivalence between nodes for oracle or grid user.
cluvfy stage -post crsinst -n all

Verifying User Equivalence ...FAILED (PRVG-2019, PRKC-1191)
PRVF-4009 : User equivalence is not set for nodes: rac2

[oracle@rac1 ~]$ ssh rac2
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

[oracle@rac2 ~]$ ssh rac1
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

[grid@rac1 ~]$ ssh rac2
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

[grid@rac2 ~]$ ssh rac1
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Oracle documentation mentions (this documentation may not be relevant any more) that user equivalence exists for opc user. But that too didn't exists.
[opc@rac1 ~]$ ssh rac2
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

[opc@rac2 ~]$ ssh rac1
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).


The cluster does not use GNS. Below is the full list of resources at the end of the cluster creation.
Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type          ONLINE             ONLINE             rac1
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type          ONLINE             ONLINE             rac2
ora.DATA.COMMONSTORE.advm           ora.volume.type                ONLINE             ONLINE             rac1
ora.DATA.COMMONSTORE.advm           ora.volume.type                ONLINE             ONLINE             rac2
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rac1
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rac2
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rac1
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rac2
ora.LISTENER_SCAN1.lsnr             ora.scan_listener.type         ONLINE             ONLINE             rac1
ora.LISTENER_SCAN2.lsnr             ora.scan_listener.type         ONLINE             ONLINE             rac2
ora.LISTENER_SCAN3.lsnr             ora.scan_listener.type         ONLINE             ONLINE             rac2
ora.RECO.dg                         ora.diskgroup.type             ONLINE             ONLINE             rac1
ora.RECO.dg                         ora.diskgroup.type             ONLINE             ONLINE             rac2
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rac1
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rac2
ora.cdbrac_lhr1fq.db                ora.database.type              ONLINE             ONLINE             rac1
ora.cdbrac_lhr1fq.db                ora.database.type              ONLINE             ONLINE             rac2
ora.cvu                             ora.cvu.type                   ONLINE             ONLINE             rac2
ora.data.commonstore.acfs           ora.acfs.type                  ONLINE             ONLINE             rac1
ora.data.commonstore.acfs           ora.acfs.type                  ONLINE             ONLINE             rac2
ora.net1.network                    ora.network.type               ONLINE             ONLINE             rac1
ora.net1.network                    ora.network.type               ONLINE             ONLINE             rac2
ora.ons                             ora.ons.type                   ONLINE             ONLINE             rac1
ora.ons                             ora.ons.type                   ONLINE             ONLINE             rac2
ora.proxy_advm                      ora.proxy_advm.type            ONLINE             ONLINE             rac1
ora.proxy_advm                      ora.proxy_advm.type            ONLINE             ONLINE             rac2
ora.qosmserver                      ora.qosmserver.type            OFFLINE            OFFLINE
ora.rac1.vip                        ora.cluster_vip_net1.type      ONLINE             ONLINE             rac1
ora.rac2.vip                        ora.cluster_vip_net1.type      ONLINE             ONLINE             rac2
ora.scan1.vip                       ora.scan_vip.type              ONLINE             ONLINE             rac1
ora.scan2.vip                       ora.scan_vip.type              ONLINE             ONLINE             rac2
ora.scan3.vip                       ora.scan_vip.type              ONLINE             ONLINE             rac2
Similar to single instance DB RAC DB too consists of two ASM disk groups. OCR and vote is stored in the DATA disk group. The backup location for OCR is also the DATA disk group.
SQL> select name,total_mb,free_Mb from v$asm_diskgroup;

NAME                             TOTAL_MB    FREE_MB
------------------------------ ---------- ----------
DATA                               262144     253408
RECO                               262144     257380

cat /etc/oracle/ocr.loc
#Device/file +DATA getting replaced by device +DATA/dbSyslw4gbgsa/OCRFILE/registry.255.1002113311
ocrconfig_loc=+DATA/dbSyslw4gbgsa/OCRFILE/registry.255.1002113311


[grid@rac1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1e4fd3b7b9364fdfbf4dbf41ffde36d9 (/dev/DATADISK3) [DATA]

ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+DATA].
ASM disks are SCSI attached and permissions are set with udev rules.
SQL> select name,path from v$asm_disk;

NAME                           PATH
------------------------------ ------------------------------
RECODISK4                      /dev/RECODISK4
RECODISK3                      /dev/RECODISK3
RECODISK2                      /dev/RECODISK2
RECODISK1                      /dev/RECODISK1
DATA_0003                      /dev/DATADISK4
DATA_0000                      /dev/DATADISK3
DATA_0001                      /dev/DATADISK2
DATA_0002                      /dev/DATADISK1


[grid@rac1 admin]$ ls -l /dev/DATA*
lrwxrwxrwx 1 root root 3 Mar  5 14:58 /dev/DATADISK1 -> sdb
lrwxrwxrwx 1 root root 3 Mar  5 14:58 /dev/DATADISK2 -> sdc
lrwxrwxrwx 1 root root 3 Mar  5 14:58 /dev/DATADISK3 -> sdd
lrwxrwxrwx 1 root root 3 Mar  5 14:58 /dev/DATADISK4 -> sde


[grid@rac1 admin]$ ls -l /dev/disk/by-id/ | egrep "sdb|sdc|sdd|sde"
lrwxrwxrwx 1 root root  9 Mar  5 14:59 scsi-36012159faf2d4794be5ab57d8a5b7498 -> ../../sdc
lrwxrwxrwx 1 root root 10 Mar  5 12:32 scsi-36012159faf2d4794be5ab57d8a5b7498-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Mar  5 14:59 scsi-36022a3f8dffa46b4a57049af3a2dd275 -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar  5 12:32 scsi-36022a3f8dffa46b4a57049af3a2dd275-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Mar  5 14:59 scsi-360748b62328c4ff6bb8fe3e9e4f04826 -> ../../sdd
lrwxrwxrwx 1 root root 10 Mar  5 12:32 scsi-360748b62328c4ff6bb8fe3e9e4f04826-part1 -> ../../sdd1
lrwxrwxrwx 1 root root  9 Mar  5 14:59 scsi-360923cd17c134686829d791e2b4ff82f -> ../../sde
lrwxrwxrwx 1 root root 10 Mar  5 12:32 scsi-360923cd17c134686829d791e2b4ff82f-part1 -> ../../sde1

cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36022a3f8dffa46b4a57049af3a2dd275", SYMLINK+="DATADISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36012159faf2d4794be5ab57d8a5b7498", SYMLINK+="DATADISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360748b62328c4ff6bb8fe3e9e4f04826", SYMLINK+="DATADISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360923cd17c134686829d791e2b4ff82f", SYMLINK+="DATADISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360bc2c20ddc145ceafa98bc1bb1532d3", SYMLINK+="RECODISK1",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606feb49caf045598f66042c2831ad38", SYMLINK+="RECODISK2",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360ad6aca673e4d0a8d2d52aded3ee466", SYMLINK+="RECODISK3",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360f2aa11901e4736b65d54f96f000dcf", SYMLINK+="RECODISK4",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e328f14b14ddf9042d0eca9038593", SYMLINK+="localdisk",  OWNER="grid",  GROUP="asmadmin",  MODE="0660"
Manual intervention is needed to create the time synchronization between the two nodes. Although NTP service is running, it is not synchronized with an external time source.
Verifying Clock Synchronization ...FAILED
  Verifying Network Time Protocol (NTP) ...FAILED
    Verifying NTP daemon is synchronized with at least one external time source
    ...FAILED
    rac1: PRVG-13602 : NTP daemon is not synchronized with any external time
          source on node "rac1".
As the ntp.conf file is present, the ctss runs in observe mode.
[grid@rac1 ~]$ crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.
This also mentioned in oracle documentation where it is recommended "that you configure NTP on both nodes in a 2-node RAC DB System to synchronize time across the nodes. If you do not configure NTP, then Oracle Clusterware configures and uses the Cluster Time Synchronization Service (CTSS), and the cluster time might be out-of-sync with applications that use NTP for time synchronization".
The ntpd file is missing the -x option which prevents setting the clock backwards.
cat /etc/sysconfig/ntpd
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"
Once the time synchronization is established it could be verified with
cluvfy comp clocksync
Though the cluster was 18c, the oracle home is a read/write home. Not a read only home which was introduced in 18c.
echo $ORACLE_HOME
/u01/app/oracle/product/18.0.0.0/dbhome_1
[oracle@rac2 bin]$ ./orabasehome
/u01/app/oracle/product/18.0.0.0/dbhome_1
The RAC DB created has two log groups per thread.
SQL> select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME     CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
         1          1          3 1073741824        512          1 YES INACTIVE               1262776 05-MAR-19      1264208 05-MAR-19          0
         2          1          4 1073741824        512          1 NO  CURRENT                1264244 05-MAR-19   1.8447E+19 05-MAR-19          0
         3          2          5 1073741824        512          1 NO  CURRENT                1262779 05-MAR-19   1.8447E+19                    0
         4          2          4 1073741824        512          1 YES INACTIVE               1242884 05-MAR-19      1262779 05-MAR-19          0
The RAC DB is a administrator managed DB.
srvctl config database -db cdbrac_lhr1fq -a
Database unique name: cdbrac_lhr1fq
Database name: cdbrac
Oracle home: /u01/app/oracle/product/18.0.0.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/CDBRAC_LHR1FQ/PARAMETERFILE/spfile.269.1002115587
Password file: +DATA/CDBRAC_LHR1FQ/PASSWORD/pwdcdbrac_lhr1fq.259.1002114845
Domain: londbsubnet.lonvcn.oraclevcn.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: RECO,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
Database is enabled
Database is individually enabled on nodes:
Database is individually disabled on nodes:
OSDBA group: dba
OSOPER group: dbaoper
Database instances: cdbrac1,cdbrac2
Configured nodes: rac1,rac2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
Both DB System patching and DB patching happens in a rolling fashion. Other DB versions where OJVM patch is needed (non-rolling patch) may require total outage. But at least in this case with 18c the patch applying happened rolling fashion.


Similar to single instance DB, RAC DB can also be associated with a data guard configuration. The London region where the initial RAC DB was created had an issue creating a data guard association. Oracle support also confirmed that there are intermittent issues when creating a data guard for RAC DB. However, it was possible to create a RAC DB with data guard association in the Frankfurt region. Following screenshot is of this new RAC DB (as such some names are different).
The DG protection mode is maximum performance.
DGMGRL> show configuration

Configuration - raccdb_fra2bg_raccdb_fra359

  Protection Mode: MaxPerformance
  Members:
  raccdb_fra2bg - Primary database
    raccdb_fra359 - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 19 seconds ago)
Primary Database
DGMGRL> show database raccdb_fra2bg

Database - raccdb_fra2bg

  Role:               PRIMARY
  Intended State:     TRANSPORT-ON
  Instance(s):
    raccdb1
    raccdb2

Database Status:
SUCCESS
Standby Database has both instance up and running, with one being the apply instance.
DGMGRL> show database raccdb_fra359

Database - raccdb_fra359

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 1 second ago)
  Apply Lag:          0 seconds (computed 1 second ago)
  Average Apply Rate: 1.00 KByte/s
  Real Time Query:    ON
  Instance(s):
    raccdb1 (apply instance)
    raccdb2

Database Status:
SUCCESS
Switchover could be performed from the console. 
DGMGRL> show configuration

Configuration - raccdb_fra2bg_raccdb_fra359

  Protection Mode: MaxPerformance
  Members:
  raccdb_fra359 - Primary database
    raccdb_fra2bg - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 58 seconds ago)

Related Posts
Data Guard on OCI PaaS DB
↧

Updating the OS and Patching OCI PaaS DB Using DBCLI

March 26, 2019, 4:40 am
$
0
0
OCI provide a command line interface for managing the PaaS DB (DBCLI) beside the console. This post shows the steps for updating the OS and patching the DB System and the DB using DBCLI.

Updating the OS
To update the OS the database server needs access to the YUM repository. If the DB system is in a private subnet then configure a NAT gateway to access the YUM repository. It is assumed that NAT gateway is configured and DB server can access the internet or egress traffic is allowed only to YUM repository location.
1. Check the region where the DB server resides.
# curl -s http://169.254.169.254/opc/v1/instance/ |grep region
"region" : "eu-frankfurt-1",
2. In this case the region is eu-frankfurt-1. Replace the region name in below URL with actual region and get the yum repos file.
https://swiftobjectstorage.region_name.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/oci_dbaas_ol6repo

# wget https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/oci_dbaas_ol6repo -O /tmp/oci_dbaas_ol6repo
--2019-03-12 10:14:05--  https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/oci_dbaas_ol6repo
...
Saving to: `/tmp/oci_dbaas_ol6repo'
3. Get the version lock file. Version lock file is needed to control the update. An update will occur only if a versionlock file has a valid update available to apply to the DB system.
# wget https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/versionlock.list -O /tmp/versionlock.list
--2019-03-12 10:14:37--  https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/versionlock.list
...
Saving to: `/tmp/versionlock.list'
4. Copy the downloaded files to their respective locations in the DB server
cp /tmp/oci_dbaas_ol6repo /etc/yum.repos.d/ol6.repo

cp /tmp/versionlock.list /etc/yum/pluginconf.d/versionlock.list
5. Open the yum repository file and enable the repository for the region. During this testing it was found that Frankfurt region was enabled by default.

6. Note the current kernel version and run the update command.
uname -r
4.1.12-124.20.3.el6uek.x86_64

yum update
Any warning similar to following could be ignored.
Warning: RPMDB altered outside of yum.
** Found 3 pre-existing rpmdb problem(s), 'yum check' output follows:
oda-hw-mgmt-18.3.1.0.0_LINUX.X64_181212-1.x86_64 has missing requires of libnfsodm18.so()(64bit)
oda-hw-mgmt-18.3.1.0.0_LINUX.X64_181212-1.x86_64 has missing requires of perl(GridDefParams)
oda-hw-mgmt-18.3.1.0.0_LINUX.X64_181212-1.x86_64 has missing requires of perl(s_GridSteps)
Also towards the end of the update following error.
ol6_UEKR4/filelists_db                                                                                                                                                                                                | 1.6 MB     00:00
Uploading /boot/vmlinuz-4.1.12-124.25.1.el6uek.x86_64 to http://169.254.0.3/kernel

Error uploading /kernel: 502 - <html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>
In spite of this error the kernel gets updated as shown below

6. Once the update is complete reboot the server and check the kernel version.
# reboot
# uname -r
4.1.12-124.25.1.el6uek.x86_64



Patching the PaaS DB
Patching the DB is of two parts similar to patching using console. First is to patch the DB system (Grid Infrastructure) and then the DB.

Updating the DBCLI
Before the patching could begin check if there's a newer version of DBCLI and update DBCLI.
cliadm update-dbcli

Job details
----------------------------------------------------------------
                     ID:  d0a0e701-3c5a-4f72-a97e-455efd02cefd
            Description:  DcsCli patching
                 Status:  Created
                Created:  March 12, 2019 10:59:31 AM UTC
                Message:  Dcs cli will be updated
Wait couple of minutes and check the outcome of the update, (use the job id given)
dbcli describe-job -i d0a0e701-3c5a-4f72-a97e-455efd02cefd

Job details
----------------------------------------------------------------
                     ID:  d0a0e701-3c5a-4f72-a97e-455efd02cefd
            Description:  DcsCli patching
                 Status:  Success
                Created:  March 12, 2019 10:59:31 AM UTC
                Message:

Task Name           Start Time                          End Time                            Status
------------------- -------------------------------     ----------------------------------- ----------
dcs-cli upgrade     March 12, 2019 10:59:32 AM UTC      March 12, 2019 10:59:34 AM UTC      Success
Check the currently available patches. This command list available patches for all versions.
# dbcli describe-latestpatch
componentType   availableVersion
--------------- --------------------
gi              12.2.0.1.190115
gi              12.1.0.2.190115
gi              18.5.0.0.190115
db              11.2.0.4.190115
db              12.2.0.1.190115
db              12.1.0.2.190115
db              18.5.0.0.190115
To list the patching available for installed GI and DB run the following command.
# dbcli describe-component
System Version
---------------
18.3.3.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
GI                                        18.3.0.0.180717       18.5.0.0.190115
DB                                        18.3.0.0.180717       18.5.0.0.190115
Patching the DB System
To patch the DB system run update server command.
# dbcli update-server
{
"jobId" : "058707c8-a9ec-48f0-ba47-47e5aa4f65c6",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "March 12, 2019 11:05:00 AM UTC",
"resourceList" : [ ],
"description" : "Server Patching",
"updatedTime" : "March 12, 2019 11:05:01 AM UTC"
}
The dcs agent log could be used to monitor the patch apply in progress
cd /opt/oracle/dcs/log
tail -100f dcs-agent.log
Another method is to use the describe job option with the job id.
dbcli describe-job -i "058707c8-a9ec-48f0-ba47-47e5aa4f65c6"

Job details
----------------------------------------------------------------
                     ID:  058707c8-a9ec-48f0-ba47-47e5aa4f65c6
            Description:  Server Patching
 Status:  Running
                Created:  March 12, 2019 11:05:00 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create Patching Repository Directories   March 12, 2019 11:05:01 AM UTC      March 12, 2019 11:05:01 AM UTC      Success
Download latest patch metadata           March 12, 2019 11:05:01 AM UTC      March 12, 2019 11:05:01 AM UTC      Success
Update Patching Repository               March 12, 2019 11:05:02 AM UTC      March 12, 2019 11:05:02 AM UTC      Running
Once the patching is complete the status will change to success.
dbcli describe-job -i "058707c8-a9ec-48f0-ba47-47e5aa4f65c6"

Job details
----------------------------------------------------------------
                     ID:  058707c8-a9ec-48f0-ba47-47e5aa4f65c6
            Description:  Server Patching
                 Status:  Success
                Created:  March 12, 2019 11:05:00 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create Patching Repository Directories   March 12, 2019 11:05:01 AM UTC      March 12, 2019 11:05:01 AM UTC      Success
Download latest patch metadata           March 12, 2019 11:05:01 AM UTC      March 12, 2019 11:05:01 AM UTC      Success
Update Patching Repository               March 12, 2019 11:05:02 AM UTC      March 12, 2019 11:08:24 AM UTC      Success
Opatch updation                          March 12, 2019 11:08:25 AM UTC      March 12, 2019 11:08:30 AM UTC      Success
Patch conflict check                     March 12, 2019 11:08:31 AM UTC      March 12, 2019 11:10:31 AM UTC      Success
apply clusterware patch                  March 12, 2019 11:10:31 AM UTC      March 12, 2019 11:32:08 AM UTC      Success
Updating GiHome version                  March 12, 2019 11:32:08 AM UTC      March 12, 2019 11:32:35 AM UTC      Success
Update System version                    March 12, 2019 11:32:52 AM UTC      March 12, 2019 11:32:52 AM UTC      Success
Use the component describe command again to see the patched version. Since the latest patch was applied the available column will state "up-to-date"
dbcli describe-component
System Version
---------------
18.3.3.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
GI                                        18.5.0.0.190115       up-to-date
DB                                         18.3.0.0.180717       18.5.0.0.190115
Patching the DB
Patching of the DB happens by specifying the DB home to patch. For this the DB home ID is needed. To find the DB home ID use the list home command.
dbcli list-dbhomes

ID                                       Name                 DB Version             Home Location                                 Status
---------------------------------------- -------------------- ---------------------- --------------------------------------------- ----------
0f59f70a-167b-4d8a-84e3-6e7d7529b093     OraDB18000_home1     18.3.0.0.180717        /u01/app/oracle/product/18.0.0.0/dbhome_1     Configured
Run the home patching command specifying the DB Home ID.
# dbcli update-dbhome -i 0f59f70a-167b-4d8a-84e3-6e7d7529b093
{
"jobId" : "dae78453-e3a4-4b6e-ba6b-baaeb8902f2f",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "March 12, 2019 11:37:09 AM UTC",
"resourceList" : [ ],
"description" : "DB Home Patching: Home Id is 0f59f70a-167b-4d8a-84e3-6e7d7529b093",
"updatedTime" : "March 12, 2019 11:37:09 AM UTC"
}
Use the job describe command to find the status of the job.
# dbcli describe-job -i "dae78453-e3a4-4b6e-ba6b-baaeb8902f2f"

Job details
----------------------------------------------------------------
                     ID:  dae78453-e3a4-4b6e-ba6b-baaeb8902f2f
            Description:  DB Home Patching: Home Id is 0f59f70a-167b-4d8a-84e3-6e7d7529b093
Status:  Success
                Created:  March 12, 2019 11:37:09 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create Patching Repository Directories   March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:09 AM UTC      Success
Download latest patch metadata           March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:09 AM UTC      Success
checking GiHome version                  March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:09 AM UTC      Success
Update System version                    March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:09 AM UTC      Success
Update Patching Repository               March 12, 2019 11:37:09 AM UTC      March 12, 2019 11:37:55 AM UTC      Success
Validating dbHome available space        March 12, 2019 11:37:55 AM UTC      March 12, 2019 11:37:55 AM UTC      Success
Opatch updation                          March 12, 2019 11:37:56 AM UTC      March 12, 2019 11:38:00 AM UTC      Success
Patch conflict check                     March 12, 2019 11:38:00 AM UTC      March 12, 2019 11:41:00 AM UTC      Success
db upgrade                               March 12, 2019 11:41:00 AM UTC      March 12, 2019 11:59:06 AM UTC      Success
SqlPatch upgrade                         March 12, 2019 11:59:06 AM UTC      March 12, 2019 12:02:14 PM UTC      Success
updating the Database version            March 12, 2019 12:02:29 PM UTC      March 12, 2019 12:02:43 PM UTC      Success
The list component will show the DB is also up-to-date.
# dbcli describe-component
System Version
---------------
18.3.3.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
GI                                        18.5.0.0.190115       up-to-date
DB                                        18.5.0.0.190115       up-to-date
Querying the DB will show that sql patch registry is updated.
select PATCH_ID,PATCH_TYPE,ACTION,STATUS,SOURCE_VERSION,TARGET_VERSION,ACTION_TIME from dba_registry_sqlpatch;

  PATCH_ID PATCH_TYPE ACTION          STATUS                    SOURCE_VERSION  TARGET_VERSION  ACTION_TIME
---------- ---------- --------------- ------------------------- --------------- --------------- ------------------------------
  28090523 RU         APPLY           SUCCESS                   18.1.0.0.0      18.3.0.0.0      06-AUG-18 07.10.59.424957 PM
  27923415 INTERIM    APPLY           SUCCESS                   18.1.0.0.0      18.1.0.0.0      06-AUG-18 07.10.59.428991 PM
28822489 RU         APPLY           SUCCESS                   18.3.0.0.0      18.5.0.0.0      12-MAR-19 11.55.48.034138 AM
  29143230 INTERIM    APPLY           SUCCESS                   18.5.0.0.0      18.5.0.0.0      12-MAR-19 12.00.42.612755 PM
The patched version of the DB System and DB was not updated on the console view. The console showed the pre-patch version for the DB system and DB. One way to rectify this was to run the patch from the console. This will not reapply the patch. But during the pre-patch check the applied patch will be detected and console will update itself with the current patch information.
↧

Backup and Recover Using DBCLI on OCI PaaS DB

March 29, 2019, 12:00 am
$
0
0
OCI Paas DBs could be configured with automatic backup in which case OCI would take care of the backup. The database gets backed up to an object storage. But the backup files or the object storage bucket they are stored in is not visible from the console. This post shows the steps for backing up a PaaS DB to a user created object storage bucket using DBCLI. It is assumed a object store bucket is already created. For this post an object store bucket called "orabaks" was created. This will be used as the backup storage.

Backup using DBCLI
1. In order to use this object storage bucket it must be associated with an object store swift end point. Create object store swift end point as shown below. This uses tenancy and username used to login to the tenancy and the password for the user account. The end point URL must be changed to reflect the region where the DB resides. Refer previous post on dbcli command to find the region. 
# dbcli create-objectstoreswift -n orabackupstore -t tenancyName -u asanga@email.com -e https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1 -p
object store swift password:
{
"jobId" : "3fa9e4fd-9da1-413d-9bca-e2434afb9229",
"status" : "Created",
"message" : "Create object store swift",
"reports" : [ ],
"createTimestamp" : "March 13, 2019 11:12:26 AM UTC",
"resourceList" : [ {
"resourceId" : "cb78ef29-a447-4b2d-af17-5b110a6717cb",
"resourceType" : "ObjectStoreSwift",
"jobId" : "3fa9e4fd-9da1-413d-9bca-e2434afb9229",
"updatedTime" : "March 13, 2019 11:12:26 AM UTC"
  } ],
"description" : "create object store:orabackupstore",
"updatedTime" : "March 13, 2019 11:12:26 AM UTC"
}
# dbcli list-objectstoreswifts

ID                                       Name                 UserName                       TenantName                Url
---------------------------------------- -------------------- ------------------------------ ------------------------- ---------------------------------------------------------------------------
c9ffa569-7f15-4220-8524-0e6f294871a4     bnCGRmStQJ1DnEMR1p4a bnCGRmStQJ1DnEMR1p4a           dbbackupfra               https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1
cb78ef29-a447-4b2d-af17-5b110a6717cb     orabackupstore       asanga@email.com               tenancyName               https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/v1

2. Create a backup configuration using the object store swift ID created in the previous step and the object store name (orabaks). The backup destination is listed as object store and this could be changed to disk for disk backups, in which case backups will be created in the FRA.
# dbcli create-backupconfig -d OBJECTSTORE -c orabaks -o cb78ef29-a447-4b2d-af17-5b110a6717cb -w 1 -n mybakconfig1
{
"jobId" : "0ecd1ed3-658a-4374-ad42-105f80e5480d",
"status" : "Created",
"message" : "backup config creation",
"reports" : [ ],
"createTimestamp" : "March 13, 2019 11:21:05 AM UTC",
"resourceList" : [ {
"resourceId" : "09c01611-6a58-49d1-b9de-553594b73cfc",
"resourceType" : "BackupConfig",
"jobId" : "0ecd1ed3-658a-4374-ad42-105f80e5480d",
"updatedTime" : "March 13, 2019 11:21:05 AM UTC"
  } ],
"description" : "create backup config:mybakconfig1",
"updatedTime" : "March 13, 2019 11:21:05 AM UTC"
}

dbcli list-backupconfigs

ID                                       Name                 RecoveryWindow   CrosscheckEnabled   BackupDestination
---------------------------------------- -------------------- ---------------- ------------------- --------------------
2e6e46e9-9dca-4906-a2e1-27b8df727152     bnCGRmStQJ1DnEMR1p4a_BC 30               true                ObjectStore
09c01611-6a58-49d1-b9de-553594b73cfc     mybakconfig1         1                   true                ObjectStore


dbcli describe-backupconfig -i 09c01611-6a58-49d1-b9de-553594b73cfc
Backup Config details
----------------------------------------------------------------
                     ID: 09c01611-6a58-49d1-b9de-553594b73cfc
Name: mybakconfig1
      CrosscheckEnabled: true
         RecoveryWindow: 1
 BackupDestination: ObjectStore
  BackupLocation: orabaks
          ObjectStoreId: cb78ef29-a447-4b2d-af17-5b110a6717cb
            CreatedTime: March 13, 2019 11:21:05 AM UTC
            UpdatedTime: March 13, 2019 11:21:34 AM UTC

3. Next step is to associate the newly created database configuration with the database. For this the database id is needed. Use the following command to find the DB id and the ID of the current backup configuration associated with the DB.
dbcli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
690f6c09-7c48-4332-ab65-b86f932c1a33     apcdb      Si       18.5.0.0.190115      true       Oltp              ASM        Configured   0f59f70a-167b-4d8a-84e3-6e7d7529b093

dbcli describe-database -i 690f6c09-7c48-4332-ab65-b86f932c1a33

Database details
----------------------------------------------------------------
                     ID: 690f6c09-7c48-4332-ab65-b86f932c1a33
            Description: apcdb
                DB Name: apcdb
             DB Version: 18.5.0.0.190115
                DB Type: Si
             DB Edition: EE
                   DBID: 1068299060
 Instance Only Database: false
                    CDB: true
               PDB Name: cgpdb
    PDB Admin User Name: pdbuser
                  Class: Oltp
                  Shape:
                Storage: ASM
          DB Redundancy:
           CharacterSet: AL32UTF8
  National CharacterSet: AL16UTF16
               Language: AMERICAN
              Territory: AMERICA
                Home ID: 0f59f70a-167b-4d8a-84e3-6e7d7529b093
        Console Enabled: false
     Level 0 Backup Day: Sunday
     AutoBackup Enabled: false
BackupConfigId: 2e6e46e9-9dca-4906-a2e1-27b8df727152
                Created: March 11, 2019 9:06:44 AM UTC
         DB Domain Name: dbsubnet.fravcn.oraclevcn.com

4. To associate the new backup configuration with the database use the update database command.
dbcli update-database -i 690f6c09-7c48-4332-ab65-b86f932c1a33 -bi 09c01611-6a58-49d1-b9de-553594b73cfc
{
"jobId" : "597671db-4db5-4a1e-91af-325d8c9ae39e",
"status" : "Created",
"message" : "update database",
"reports" : [ ],
"createTimestamp" : "March 13, 2019 11:26:50 AM UTC",
"resourceList" : [ {
"resourceId" : "690f6c09-7c48-4332-ab65-b86f932c1a33",
"resourceType" : "DB",
"jobId" : "597671db-4db5-4a1e-91af-325d8c9ae39e",
"updatedTime" : "March 13, 2019 11:26:50 AM UTC"
  } ],
"description" : "update database : apcdb",
"updatedTime" : "March 13, 2019 11:26:50 AM UTC"
}

 dbcli describe-database -i 690f6c09-7c48-4332-ab65-b86f932c1a33

Database details
----------------------------------------------------------------
                     ID: 690f6c09-7c48-4332-ab65-b86f932c1a33
            Description: apcdb
                DB Name: apcdb
             DB Version: 18.5.0.0.190115
                DB Type: Si
             DB Edition: EE
                   DBID: 1068299060
 Instance Only Database: false
                    CDB: true
               PDB Name: cgpdb
    PDB Admin User Name: pdbuser
                  Class: Oltp
                  Shape:
                Storage: ASM
          DB Redundancy:
           CharacterSet: AL32UTF8
  National CharacterSet: AL16UTF16
               Language: AMERICAN
              Territory: AMERICA
                Home ID: 0f59f70a-167b-4d8a-84e3-6e7d7529b093
        Console Enabled: false
     Level 0 Backup Day: Sunday
     AutoBackup Enabled: false
BackupConfigId: 09c01611-6a58-49d1-b9de-553594b73cfc
                Created: March 11, 2019 9:06:44 AM UTC
         DB Domain Name: dbsubnet.fravcn.oraclevcn.com
5. Create a database backup by specifying the DB id. The backups files will be stored in the user created object store created (orabaks). The following command shows creating an archive log backup.
dbcli create-backup -bt ArchiveLog -i 690f6c09-7c48-4332-ab65-b86f932c1a33
{
"jobId" : "047bbf1e-2ef2-4efd-a0dd-663d0b2c4456",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "March 13, 2019 11:34:01 AM UTC",
"resourceList" : [ ],
"description" : "Create ArchiveLog Backup with TAG-auto for Db:apcdb in OSS:orabaks",
"updatedTime" : "March 13, 2019 11:34:01 AM UTC"
}

dbcli describe-job -i 047bbf1e-2ef2-4efd-a0dd-663d0b2c4456

Job details
----------------------------------------------------------------
                     ID:  047bbf1e-2ef2-4efd-a0dd-663d0b2c4456
            Description:  Create ArchiveLog Backup with TAG-auto for Db:apcdb in OSS:orabaks
                 Status:  Success
                Created:  March 13, 2019 11:34:01 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate backup config                   March 13, 2019 11:34:04 AM UTC      March 13, 2019 11:34:04 AM UTC      Success
Database container validation            March 13, 2019 11:34:04 AM UTC      March 13, 2019 11:34:05 AM UTC      Success
libopc existence check                   March 13, 2019 11:34:05 AM UTC      March 13, 2019 11:34:05 AM UTC      Success
Backup Validations                       March 13, 2019 11:34:05 AM UTC      March 13, 2019 11:34:54 AM UTC      Success
Recovery Window validation               March 13, 2019 11:34:54 AM UTC      March 13, 2019 11:35:17 AM UTC      Success
Archivelog deletion policy configuration March 13, 2019 11:35:17 AM UTC      March 13, 2019 11:35:37 AM UTC      Success
Archive log backup                       March 13, 2019 11:35:37 AM UTC      March 13, 2019 11:38:41 AM UTC      Success
Backup TDE Wallet files to ObjectStore   March 13, 2019 11:38:41 AM UTC      March 13, 2019 11:38:42 AM UTC      Success
The created backup files are visible inside the object storage bucket.

6. A RMAN backup report could be created using the DB id. The rman report will list the log files which would have finer details of the backup.
dbcli create-rmanbackupreport -i 690f6c09-7c48-4332-ab65-b86f932c1a33 -w detailed -rn archlogreport
{
"jobId" : "20f74edf-dcbf-4bd7-89cd-51358d2ab7d6",
"status" : "Created",
"message" : "Rman BackupReport creation.",
"reports" : [ ],
"createTimestamp" : "March 13, 2019 11:44:45 AM UTC",
"resourceList" : [ {
"resourceId" : "34dbed6a-4182-4fb5-a1b8-8918a799485f",
"resourceType" : "Report",
"jobId" : "20f74edf-dcbf-4bd7-89cd-51358d2ab7d6",
"updatedTime" : "March 13, 2019 11:44:45 AM UTC"
  } ],
"description" : "Create detailed Backup Report ",
"updatedTime" : "March 13, 2019 11:44:45 AM UTC"
}


dbcli list-rmanbackupreports

ID                                       Name                 ReportType DbId                                     createTime                          updatedTime
---------------------------------------- -------------------- ---------- ---------------------------------------- ----------------------------------- -----------------------------------
34dbed6a-4182-4fb5-a1b8-8918a799485f     archlogreport        detailed   690f6c09-7c48-4332-ab65-b86f932c1a33     March 13, 2019 11:44:45 AM UTC      March 13, 2019 11:45:36 AM UTC

describe-rmanbackupreport -i 34dbed6a-4182-4fb5-a1b8-8918a799485f
Backup Report details
----------------------------------------------------------------
                     ID: 34dbed6a-4182-4fb5-a1b8-8918a799485f
            Report Type: detailed
               Location: Node apdb: /opt/oracle/dcs/log/apdb/rman/bkup/apcdb_fra2g6/rman_list_backup_detail/2019-03-13/rman_list_backup_detail_2019-03-13_11-45-20.0508.log
            Database ID: 690f6c09-7c48-4332-ab65-b86f932c1a33
            CreatedTime: March 13, 2019 11:44:45 AM UTC

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
b5d96197-91ec-4319-abf7-09b39d70bafb     Create Regular-L0 Backup with TAG-auto for Db:apcdb in OSS:orabaks          March 13, 2019 12:01:36 PM UTC      Success
8bc91a59-2bee-4b9e-aa59-b73048adca38     Create Archivelog Backup with TAG-auto for Db:apcdb in OSS:orabaks          March 13, 2019 12:53:59 PM UTC      Success

Backup report could be deleted using the following command.
dbcli delete-rmanbackupreport -i 34dbed6a-4182-4fb5-a1b8-8918a799485f
To remove a backup configuration, if its assign to the DB then update the DB with a different backup configuration. Use the following command to delete the backup configuration by specifying backup configuration id.
dbcli delete-backupconfig -i 09c01611-6a58-49d1-b9de-553594b73cfc
To delete a object store swift use the following command, specifying object store swift id.
dbcli delete-objectstoreswift -i cb78ef29-a447-4b2d-af17-5b110a6717cb



Recover using DBCLI
Following steps shows point in time recovery using DBCLI to simulate a recovery from accidental drop of a table. DBCLI also provides other recovery types as well (Latest|SCN).
1. The table is droped using purge option.
SQL> conn asanga/asanga#1234$ABC@pdbsrvx
Connected.
SQL> ! date
Wed Mar 13 12:58:37 UTC 2019

SQL> drop table x purge;
Table dropped.
2. Use recover database command to execute a PITR. The recovery time stamp (specified by -r) is just before the table was dropped.
dbcli recover-database -i 690f6c09-7c48-4332-ab65-b86f932c1a33 -r 03/13/2019 12:58:35 -t PITR
{
"jobId" : "037b6bfd-1bc3-4c2e-af5e-1a22dbc05189",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "March 13, 2019 13:01:57 PM UTC",
"resourceList" : [ ],
"description" : "Create recovery-pitr : time '03/13/2019 12:58:35' for db : apcdb",
"updatedTime" : "March 13, 2019 13:01:57 PM UTC"
}
Monitor the recovery using the recovery job id given above.
dbcli describe-job -i 037b6bfd-1bc3-4c2e-af5e-1a22dbc05189

Job details
----------------------------------------------------------------
                     ID:  037b6bfd-1bc3-4c2e-af5e-1a22dbc05189
            Description:  Create recovery-pitr : time '03/13/2019 12:58:35' for db : apcdb
                 Status:  Success
                Created:  March 13, 2019 1:01:57 PM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database recovery validation             March 13, 2019 1:02:08 PM UTC       March 13, 2019 1:03:05 PM UTC       Success
Database recovery                        March 13, 2019 1:03:05 PM UTC       March 13, 2019 1:05:54 PM UTC       Success
Enable block change tracking             March 13, 2019 1:05:54 PM UTC       March 13, 2019 1:05:56 PM UTC       Success
Database opening                         March 13, 2019 1:05:56 PM UTC       March 13, 2019 1:06:53 PM UTC       Success
Database restart                         March 13, 2019 1:06:53 PM UTC       March 13, 2019 1:08:55 PM UTC       Success
Recovery metadata persistance            March 13, 2019 1:08:55 PM UTC       March 13, 2019 1:08:55 PM UTC       Success
Verify table is recovered
SQL> desc x
 Name     Null?    Type
 -------- ------- ----------------------------
 A                NUMBER
↧
↧

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Clusterware

April 1, 2019, 12:00 am
$
0
0
This post is not a step by step guide to installing 18c RAC but list the main differences in the installation process compared to previous versions and things to look out for. Similar to 12cR2 clusterware the 18c cluster installation is no longer done through runInstaller. The downloaded Oracle Grid Infrastructure image files are unzipped into the folder path where GI is to be installed. This only need to be done on the local node only. During installation, the software is copied and installed on all other nodes in the cluster. Unlike 12cR1, it is possible to specify a separate disk group for GIMR at the installation time. Oracle documentation says "You cannot migrate the GIMR from one disk group to another later". 12cR1 also mentioned the same but there were ways to move GIMR to different disk group after the installation. It has been not verified whether the same techniques could be used for 18c as well.
1. The OS version and kernel versions used for this RAC setup is shown below. Unlike the previous post on 12cR2 this post for 18c uses RHEL7
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

uname -r
3.10.0-693.el7.x86_64
2. Create the user groups and users similar to previous 12.1 installation. As said in an earlier post, 12.2 introduce a new user group for RAC management. Following from install guide "You must designate a group as the OSRACDBA group during database installation. Members of this group are granted the SYSRAC privileges to perform day–to–day administration of Oracle databases on an Oracle RAC cluster".
groupadd dba
groupadd oinstall
groupadd oper
groupadd asmoper
groupadd asmdba
groupadd asmadmin
groupadd backupdba
groupadd dgdba
groupadd kmdba
groupadd racdba

useradd -g oinstall -G dba,oper,asmdba,backupdba,dgdba,kmdba,racdba oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba grid
For user equivelancy ensure that user id and group id are same across the cluster nodes. The option -g could be used to ensure that same user/group id is used for creating users across all nodes. The nodes in this case are named rhel71 and rhel72.
[oracle@rhel71 ~]$ id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),1000(dba),1002(oper),1004(asmdba),1006(backupdba),1007(dgdba),1008(kmdba),1009(racdba)
[oracle@rhel71 ~]$ ssh rhel72 id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),1000(dba),1002(oper),1004(asmdba),1006(backupdba),1007(dgdba),1008(kmdba),1009(racdba)

[oracle@rhel71 ~]$ id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),1000(dba),1003(asmoper),1004(asmdba),1005(asmadmin)
[oracle@rhel71 ~]$ ssh rhel72 id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),1000(dba),1003(asmoper),1004(asmdba),1005(asmadmin)
3. If a separate disk group is used for GIMR then it needs to be at least 35GB in size. Following from installation doc "during installation of an Oracle Standalone Cluster, if you create the MGMT disk group for Grid Infrastructure Management Repository (GIMR), then the installer requires that you use a disk group with at least 35 GB of available space". A 40GB disk was created to store the GIMR and udev rules were created for granting necessary ownership and permission.
cat /etc/udev/rules.d/99-sd.rules
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a5c8787-e082e7a7",SYMLINK+="oracleasm/ocr1",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a72679c-ab23e15f",SYMLINK+="oracleasm/ocr2",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB8f43e75c-3e0bec6d",SYMLINK+="oracleasm/ocr3",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBf8475ab7-d401ec8b",SYMLINK+="oracleasm/gimr",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB34650f33-5ecdc029",SYMLINK+="oracleasm/data1",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBdb057b08-1733da4b",SYMLINK+="oracleasm/fra1",OWNER="grid",GROUP="asmadmin",MODE="0660"
4. Create the GI installation directory and unzip the downloaded install media into the directory.
mkdir -p /opt/app/18.x.0/grid
chown grid:oinstall /opt/app/18.x.0/grid
unzip ./LINUX.X64_180000_grid_home.zip -d /opt/app/18.x.0/grid
5. Run cluvfy to check the pre-reqs checks. The failure related to shm shown at the end could be ignored as per 2065603.1 and 25907259.8
[grid@rhel71 grid]$ ./runcluvfy.sh stage -pre crsinst -n rhel71,rhel72 -r 18 -osdba asmdba -osoper asmoper -asm -asmgrp asmadmin -presence local -asmdev /dev/oracleasm/data1,/dev/oracleasm/fra1,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3,/dev/oracleasm/gimr  -crshome /opt/app/18.x.0/grid -fixup  -verbose

Verifying Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        2.7806GB (2915620.0KB)    8GB (8388608.0KB)         failed
  rhel71        2.7806GB (2915620.0KB)    8GB (8388608.0KB)         failed
Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        2.6736GB (2803492.0KB)    50MB (51200.0KB)          passed
  rhel71        2.4872GB (2608032.0KB)    50MB (51200.0KB)          passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3.7246GB (3905532.0KB)    2.7806GB (2915620.0KB)    passed
  rhel71        3.7246GB (3905532.0KB)    2.7806GB (2915620.0KB)    passed
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/18.x.0/grid,rhel72:/sbin,rhel72:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel72        /             28.9385GB     25MB          passed
  /var              rhel72        /             28.9385GB     5MB           passed
  /etc              rhel72        /             28.9385GB     25MB          passed
  /opt/app/18.x.0/grid  rhel72        /             28.9385GB     6.9GB         passed
  /sbin             rhel72        /             28.9385GB     10MB          passed
  /tmp              rhel72        /             28.9385GB     1GB           passed
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/18.x.0/grid,rhel72:/sbin,rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhel71:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel71        /             18.9439GB     25MB          passed
  /var              rhel71        /             18.9439GB     5MB           passed
  /etc              rhel71        /             18.9439GB     25MB          passed
  /opt/app/18.x.0/grid  rhel71        /             18.9439GB     6.9GB         passed
  /sbin             rhel71        /             18.9439GB     10MB          passed
  /tmp              rhel71        /             18.9439GB     1GB           passed
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhel71:/tmp ...PASSED
Verifying User Existence: grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists(502)
  rhel71        passed                    exists(502)

  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmdba ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: oinstall(Primary) ...
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel72            yes           yes           yes           yes           passed
  rhel71            yes           yes           yes           yes           passed
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: asmoper ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3                         3,5                       passed
  rhel71        3                         3,5                       passed
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            hard          65536         65536         passed
  rhel71            hard          65536         65536         passed
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          65536         1024          passed
  rhel71            soft          65536         1024          passed
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            hard          16384         16384         passed
  rhel71            hard          16384         16384         passed
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          16384         2047          passed
  rhel71            soft          16384         2047          passed
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          10240         10240         passed
  rhel71            soft          10240         10240         passed
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        x86_64                    x86_64                    passed
  rhel71        x86_64                    x86_64                    passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3.10.0-693.el7.x86_64     3.10.0                    passed
  rhel71        3.10.0-693.el7.x86_64     3.10.0                    passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3010          3010          250           passed
  rhel72            3010          3010          250           passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            385280        385280        32000         passed
  rhel72            385280        385280        32000         passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3010          3010          100           passed
  rhel72            3010          3010          100           passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            128           128           128           passed
  rhel72            128           128           128           passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4398046511104  4398046511104  1492797440    passed
  rhel72            4398046511104  4398046511104  1492797440    passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4096          4096          4096          passed
  rhel72            4096          4096          4096          passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            1073741824    1073741824    1073741824    passed
  rhel72            1073741824    1073741824    1073741824    passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            6815744       6815744       6815744       passed
  rhel72            6815744       6815744       6815744       passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel72            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4194304       4194304       262144        passed
  rhel72            4194304       4194304       262144        passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4194304       4194304       4194304       passed
  rhel72            4194304       4194304       4194304       passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            1048576       1048576       262144        passed
  rhel72            1048576       1048576       262144        passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            2097152       2097152       1048576       passed
  rhel72            2097152       2097152       1048576       passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3145728       3145728       1048576       passed
  rhel72            3145728       3145728       1048576       passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        binutils-2.25.1-31.base.el7  binutils-2.23.52.0.1      passed
  rhel71        binutils-2.25.1-31.base.el7  binutils-2.23.52.0.1      passed
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
  rhel71        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libgcc(x86_64)-4.8.5-16.el7  libgcc(x86_64)-4.8.2      passed
  rhel71        libgcc(x86_64)-4.8.5-16.el7  libgcc(x86_64)-4.8.2      passed
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libstdc++(x86_64)-4.8.5-16.el7  libstdc++(x86_64)-4.8.2   passed
  rhel71        libstdc++(x86_64)-4.8.5-16.el7  libstdc++(x86_64)-4.8.2   passed
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libstdc++-devel(x86_64)-4.8.5-16.el7  libstdc++-devel(x86_64)-4.8.2  passed
  rhel71        libstdc++-devel(x86_64)-4.8.5-16.el7  libstdc++-devel(x86_64)-4.8.2  passed
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        sysstat-10.1.5-12.el7     sysstat-10.1.5            passed
  rhel71        sysstat-10.1.5-12.el7     sysstat-10.1.5            passed
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        gcc-c++-4.8.5-16.el7      gcc-c++-4.8.2             passed
  rhel71        gcc-c++-4.8.5-16.el7      gcc-c++-4.8.2             passed
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        ksh                       ksh                       passed
  rhel71        ksh                       ksh                       passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        make-3.82-23.el7          make-3.82                 passed
  rhel71        make-3.82-23.el7          make-3.82                 passed
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        glibc(x86_64)-2.17-196.el7  glibc(x86_64)-2.17        passed
  rhel71        glibc(x86_64)-2.17-196.el7  glibc(x86_64)-2.17        passed
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        glibc-devel(x86_64)-2.17-196.el7  glibc-devel(x86_64)-2.17  passed
  rhel71        glibc-devel(x86_64)-2.17-196.el7  glibc-devel(x86_64)-2.17  passed
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
  rhel71        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
  rhel71        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        nfs-utils-1.3.0-0.48.el7  nfs-utils-1.2.3-15        passed
  rhel71        nfs-utils-1.3.0-0.48.el7  nfs-utils-1.2.3-15        passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        smartmontools-6.2-8.el7   smartmontools-6.2-4       passed
  rhel71        smartmontools-6.2-8.el7   smartmontools-6.2-4       passed
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        net-tools-2.0-0.22.20131004git.el7  net-tools-2.0-0.17        passed
  rhel71        net-tools-2.0-0.22.20131004git.el7  net-tools-2.0-0.17        passed
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        compat-libstdc++-33(x86_64)-3.2.3-72.el7  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel71        compat-libstdc++-33(x86_64)-3.2.3-72.el7  compat-libstdc++-33(x86_64)-3.2.3  passed
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libxcb(x86_64)-1.12-1.el7  libxcb(x86_64)-1.11       passed
  rhel71        libxcb(x86_64)-1.12-1.el7  libxcb(x86_64)-1.11       passed
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libX11(x86_64)-1.6.5-1.el7  libX11(x86_64)-1.6.3      passed
  rhel71        libX11(x86_64)-1.6.5-1.el7  libX11(x86_64)-1.6.3      passed
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXau(x86_64)-1.0.8-2.1.el7  libXau(x86_64)-1.0.8      passed
  rhel71        libXau(x86_64)-1.0.8-2.1.el7  libXau(x86_64)-1.0.8      passed
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXi(x86_64)-1.7.9-1.el7  libXi(x86_64)-1.7.4       passed
  rhel71        libXi(x86_64)-1.7.9-1.el7  libXi(x86_64)-1.7.4       passed
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXtst(x86_64)-1.2.3-1.el7  libXtst(x86_64)-1.2.2     passed
  rhel71        libXtst(x86_64)-1.2.3-1.el7  libXtst(x86_64)-1.2.2     passed
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            6200          TCP           yes           successful
  rhel71            6200          TCP           yes           successful
  rhel72            6100          TCP           yes           successful
  rhel71            6100          TCP           yes           successful
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...PASSED
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            42424         TCP           yes           successful
  rhel71            42424         TCP           yes           successful
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel72                                passed
  rhel71                                passed
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
  rhel71        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel71                                passed
  rhel72                                passed
  Verifying Hosts File ...PASSED

Interface information for node "rhel71"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s3 192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.102   08:00:27:BD:32:39 1500
 enp0s8 192.168.1.93    192.168.1.0     0.0.0.0         192.168.0.102   08:00:27:BD:A9:0A 1500

Interface information for node "rhel72"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s3 192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.102   08:00:27:31:0F:1F 1500
 enp0s8 192.168.1.94    192.168.1.0     0.0.0.0         192.168.0.102   08:00:27:98:22:A6 1500

Check: MTU consistency of the subnet "192.168.1.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel71            enp0s8        192.168.1.93  192.168.1.0   1500
  rhel72            enp0s8        192.168.1.94  192.168.1.0   1500

Check: MTU consistency of the subnet "192.168.0.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel71            enp0s3        192.168.0.93  192.168.0.0   1500
  rhel72            enp0s3        192.168.0.94  192.168.0.0   1500
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel71[enp0s8:192.168.1.93]     rhel72[enp0s8:192.168.1.94]     yes

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel71[enp0s3:192.168.0.93]     rhel72[enp0s3:192.168.0.94]     yes
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast or broadcast check ...PASSED
Verifying Device Checks for ASM ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
  Verifying ASM device sharedness check ...
    Verifying Shared Storage Accessibility:/dev/oracleasm/gimr,/dev/oracleasm/fra1,/dev/oracleasm/data1,/dev/oracleasm/ocr3,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2 ...PASSED

  Device                                Device Type
  ------------------------------------  ------------------------
  /dev/oracleasm/ocr3                   Disk
  /dev/oracleasm/data1                  Disk
  /dev/oracleasm/ocr2                   Disk
  /dev/oracleasm/fra1                   Disk
  /dev/oracleasm/ocr1                   Disk
  /dev/oracleasm/gimr                   Disk
  Verifying ASM device sharedness check ...PASSED
  Verifying Access Control List check ...
Checking consistency of device owner across all nodes...
Consistency check of device owner for "/dev/oracleasm/gimr" PASSED
Consistency check of device owner for "/dev/oracleasm/fra1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr3" PASSED
Consistency check of device owner for "/dev/oracleasm/data1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device group across all nodes...
Consistency check of device group for "/dev/oracleasm/gimr" PASSED
Consistency check of device group for "/dev/oracleasm/fra1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr3" PASSED
Consistency check of device group for "/dev/oracleasm/data1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/dev/oracleasm/gimr" PASSED
Consistency check of device permissions for "/dev/oracleasm/fra1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr3" PASSED
Consistency check of device permissions for "/dev/oracleasm/data1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr2" PASSED
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying Network Time Protocol (NTP) ...
  Verifying '/etc/ntp.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/etc/ntp.conf' ...PASSED
  Verifying '/etc/chrony.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/etc/chrony.conf' ...PASSED
  Verifying '/var/run/ntpd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/var/run/ntpd.pid' ...PASSED
  Verifying '/var/run/chronyd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/var/run/chronyd.pid' ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel72        0022                      0022                      passed
  rhel71        0022                      0022                      passed
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    does not exist
  rhel71        passed                    does not exist
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Time offset between nodes ...PASSED
Verifying resolv.conf Integrity ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel71                                passed
  rhel72                                passed

checking response for name "rhel72" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        192.168.0.66              IPv4                      passed

checking response for name "rhel71" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel71        192.168.0.66              IPv4                      passed
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Domain Sockets ...PASSED
Verifying /boot mount ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying Grid Infrastructure home path: /opt/app/18.x.0/grid ...
  Verifying '/opt/app/18.x.0/grid' ...PASSED
Verifying Grid Infrastructure home path: /opt/app/18.x.0/grid ...PASSED
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying File system mount options for path /var ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Physical Memory ...FAILED
rhel72: PRVF-7530 : Sufficient physical memory is not available on node
"rhel72" [Required physical memory = 8GB (8388608.0KB)]

rhel71: PRVF-7530 : Sufficient physical memory is not available on node
"rhel71" [Required physical memory = 8GB (8388608.0KB)]

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
6. From the GI_HOME directory run gridSetup.sh to begin the installation. Select new cluster configuration option.
Select standard cluster (this would create a flex cluster which is the default for all standard clusters).
Set cluster name and scan name. No GNS is setup or used for this installation.
Add node and node VIP names that will be used in the cluster configuration. Importantly all node roles are set as HUB nodes.
Select the network interfaces for public and ASM/Private interconnect.
The OCR and vote disks are stored in ASM but ASM could be configured either block devices or using NFS. This setup uses block devices.
A separate disk group is used to store GIMR.
Create the disk group for OCR files.
Create disk group for GIMR
Set the oracle base location. The GI install location is immutable.
The failed pre-reqs are ignorable for test setup.
Summary
7. When prompted run the root script. Following output from running root.sh on the first node.
./root.sh
Performing root user operation.

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel71/crsconfig/rootcrs_rhel71_2019-03-19_02-59-46PM.log
2019/03/19 15:00:12 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/03/19 15:00:12 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:00:56 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:00:56 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/03/19 15:00:57 CLSRSC-363: User ignored prerequisites during installation
2019/03/19 15:00:57 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/03/19 15:01:01 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/03/19 15:01:07 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/03/19 15:02:17 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/03/19 15:02:30 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/03/19 15:02:58 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/03/19 15:02:58 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/03/19 15:03:17 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/03/19 15:03:17 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/03/19 15:04:47 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/03/19 15:05:05 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:06:37 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/03/19 15:06:55 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel71'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel71'
CRS-2676: Start of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel71'
CRS-2676: Start of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel71'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel71'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel71'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel71'
CRS-2676: Start of 'ora.diskmon' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel71' succeeded

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190319PM030741.log for details.


2019/03/19 15:08:59 CLSRSC-482: Running command: '/opt/app/18.x.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-2672: Attempting to start 'ora.crf' on 'rhel71'
CRS-2672: Attempting to start 'ora.storage' on 'rhel71'
CRS-2676: Start of 'ora.storage' on 'rhel71' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel71'
CRS-2676: Start of 'ora.crsd' on 'rhel71' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk c500a19f07ec4f62bf8ac318166a95ae.
Successful addition of voting disk 9100c10cb7bb4fd1bffbe97b8dc91245.
Successful addition of voting disk 300c2bf8b7094fa6bfaf11e49f826c7e.
Successfully replaced voting disk group with +OCRDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   c500a19f07ec4f62bf8ac318166a95ae (/dev/oracleasm/ocr1) [OCRDG]
 2. ONLINE   9100c10cb7bb4fd1bffbe97b8dc91245 (/dev/oracleasm/ocr2) [OCRDG]
 3. ONLINE   300c2bf8b7094fa6bfaf11e49f826c7e (/dev/oracleasm/ocr3) [OCRDG]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel71'
CRS-2677: Stop of 'ora.crsd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel71'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel71'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel71'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel71'
CRS-2677: Stop of 'ora.crf' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel71'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel71'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel71'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel71'
CRS-2677: Stop of 'ora.ctssd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel71'
CRS-2677: Stop of 'ora.cssd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel71'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel71'
CRS-2677: Stop of 'ora.gipcd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/03/19 15:13:36 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel71'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel71'
CRS-2676: Start of 'ora.evmd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel71'
CRS-2676: Start of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel71'
CRS-2676: Start of 'ora.gipcd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel71'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel71'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel71'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel71'
CRS-2676: Start of 'ora.diskmon' on 'rhel71' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel71'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel71'
CRS-2676: Start of 'ora.ctssd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel71'
CRS-2676: Start of 'ora.asm' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel71'
CRS-2676: Start of 'ora.storage' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel71'
CRS-2676: Start of 'ora.crsd' on 'rhel71' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel71
CRS-6016: Resource auto-start has completed for server rhel71
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:16:19 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/03/19 15:16:19 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel71'
CRS-2676: Start of 'ora.asm' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.OCRDG.dg' on 'rhel71'
CRS-2676: Start of 'ora.OCRDG.dg' on 'rhel71' succeeded
2019/03/19 15:19:48 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190319PM031955.log for details.


2019/03/19 15:23:06 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Root script run on the second (last) node.
./root.sh
Performing root user operation.

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-03-19_03-25-21PM.log
2019/03/19 15:25:38 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/03/19 15:25:38 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:26:17 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:26:17 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/03/19 15:26:17 CLSRSC-363: User ignored prerequisites during installation
2019/03/19 15:26:17 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/03/19 15:26:19 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/03/19 15:26:19 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/03/19 15:26:23 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/03/19 15:26:23 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/03/19 15:26:25 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/03/19 15:26:28 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/03/19 15:26:28 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/03/19 15:26:31 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/03/19 15:26:31 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/03/19 15:26:34 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/03/19 15:26:35 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/03/19 15:27:53 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/03/19 15:27:56 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:29:13 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/03/19 15:29:15 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel72'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel72' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/03/19 15:29:28 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel72'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel72'
CRS-2676: Start of 'ora.mdnsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel72'
CRS-2676: Start of 'ora.gpnpd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel72'
CRS-2676: Start of 'ora.gipcd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel72'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel72'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel72'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel72'
CRS-2676: Start of 'ora.diskmon' on 'rhel72' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel72'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel72'
CRS-2676: Start of 'ora.ctssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel72'
CRS-2676: Start of 'ora.crsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-6017: Processing resource auto-start for servers: rhel72
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel71'
CRS-2672: Attempting to start 'ora.ons' on 'rhel72'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel71'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel72'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel72'
CRS-2676: Start of 'ora.ons' on 'rhel72' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel72' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.proxy_advm' on 'rhel71'
CRS-2672: Attempting to start 'ora.proxy_advm' on 'rhel72'
CRS-2676: Start of 'ora.proxy_advm' on 'rhel71' succeeded
CRS-2676: Start of 'ora.proxy_advm' on 'rhel72' succeeded
CRS-6016: Resource auto-start has completed for server rhel72
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:32:28 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/03/19 15:32:29 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
2019/03/19 15:33:03 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.
2019/03/19 15:33:13 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
8. Once the root scripts are run click OK on the configuration dialogue box to complete the creation of GIMR and other configuration assistants. At the end this clusterware installation would be completed.



9. Run cluvfy with post crsinst to check the installation.
[grid@rhel71 ~]$ cluvfy stage -post crsinst -allnodes

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying User Mask ...PASSED
Verifying Cluster Integrity ...PASSED
Verifying OCR Integrity ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Voting Disk ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Access control attributes for /var/tmp/.oracle ...PASSED
Verifying Access control attributes for %OCRCONFIGDIR%/maps ...PASSED

Post-check for cluster services setup was successful.
10. There are few cluster and local resources that are new in 18c compared to 12.2 Most of them relate the rapid home provisioning feature and are in disable state.
 crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GIMRDG.GHCHKPT.advm
               OFFLINE OFFLINE      rhel71                   STABLE
...
ora.gimrdg.ghchkpt.acfs
               OFFLINE OFFLINE      rhel71                   STABLE
               OFFLINE OFFLINE      rhel72                   STABLE
ora.helper
               OFFLINE OFFLINE      rhel71                   IDLE,STABLE
               OFFLINE OFFLINE      rhel72                   IDLE,STABLE
ora.proxy_advm
               ONLINE  ONLINE       rhel71                   STABLE
               ONLINE  ONLINE       rhel72                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
...
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
For example clusterware installation create a pre-configued ASM volume group call GHCHKPT which is to be used with rapid home provisioning framework.
ASMCMD> volinfo --all
Diskgroup Name: GIMRDG

         Volume Name: GHCHKPT
         Volume Device: /dev/asm/ghchkpt-493
         State: DISABLED
         Size (MB): 1536
         Resize Unit (MB): 512
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage: ACFS
         Mountpath: /opt/oracle/rhp_images/chkbase
Because of this volume group the proxy ASM instance will have an active and online status. These would be visible as APX{node number} instances. Oracle docs states "an Oracle ASM proxy instance is an Oracle instance running on a Hub node with a direct Oracle ASM client. An Oracle ASM proxy instance provides support for Oracle Automatic Storage Management Cluster File System (Oracle ACFS) and Oracle ASM Dynamic Volume Manager (Oracle ADVM)". If no other ASM volume groups are to be created or online then ASM proxy instances could be stopped and disabled.
[grid@rhel71 ~]$ srvctl stop asm -proxy -node rhel71
[grid@rhel71 ~]$ srvctl stop asm -proxy -node rhel72
[grid@rhel71 ~]$ srvctl status asm -proxy
ADVM proxy is not running
The full list of resources after the installation is shown below (before proxy instances were disabled).
Resource Name                       Type                      Target             State              Host
-------------                       ------                    -------            --------           ----------
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type     ONLINE             ONLINE             rhel71
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type     ONLINE             ONLINE             rhel72
ora.GIMRDG.GHCHKPT.advm             ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.GHCHKPT.advm             ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.dg                       ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.GIMRDG.dg                       ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.LISTENER.lsnr                   ora.listener.type         ONLINE             ONLINE             rhel71
ora.LISTENER.lsnr                   ora.listener.type         ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN1.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN2.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.LISTENER_SCAN3.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.MGMTLSNR                        ora.mgmtlsnr.type         ONLINE             ONLINE             rhel71
ora.OCRDG.dg                        ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.OCRDG.dg                        ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.asm                             ora.asm.type              ONLINE             ONLINE             rhel71
ora.asm                             ora.asm.type              ONLINE             ONLINE             rhel72
ora.asm                             ora.asm.type              OFFLINE            OFFLINE
ora.chad                            ora.chad.type             ONLINE             ONLINE             rhel71
ora.chad                            ora.chad.type             ONLINE             ONLINE             rhel72
ora.cvu                             ora.cvu.type              ONLINE             ONLINE             rhel71
ora.gimrdg.ghchkpt.acfs             ora.acfs.type             OFFLINE            OFFLINE
ora.gimrdg.ghchkpt.acfs             ora.acfs.type             OFFLINE            OFFLINE
ora.helper                          ora.helper.type           OFFLINE            OFFLINE
ora.helper                          ora.helper.type           OFFLINE            OFFLINE
ora.mgmtdb                          ora.mgmtdb.type           ONLINE             ONLINE             rhel71
ora.net1.network                    ora.network.type          ONLINE             ONLINE             rhel71
ora.net1.network                    ora.network.type          ONLINE             ONLINE             rhel72
ora.ons                             ora.ons.type              ONLINE             ONLINE             rhel71
ora.ons                             ora.ons.type              ONLINE             ONLINE             rhel72
ora.proxy_advm                      ora.proxy_advm.type       ONLINE             ONLINE             rhel71
ora.proxy_advm                      ora.proxy_advm.type       ONLINE             ONLINE             rhel72
ora.qosmserver                      ora.qosmserver.type       ONLINE             ONLINE             rhel71
ora.rhel71.vip                      ora.cluster_vip_net1.type ONLINE             ONLINE             rhel71
ora.rhel72.vip                      ora.cluster_vip_net1.type ONLINE             ONLINE             rhel72
ora.rhpserver                       ora.rhpserver.type        OFFLINE            OFFLINE
ora.scan1.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel72
ora.scan2.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel71
ora.scan3.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel71
11. The default backup location for OCR is the disk group used for GIMR
ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+GIMRDG].
12. The clusermode and versions are as follows.
$ crsctl get cluster mode status
Cluster is running in "flex" mode

$crsctl get node role config -all
Node 'rhel71' configured role is 'hub'
Node 'rhel72' configured role is 'hub'

$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [18.0.0.0.0]
The next step is to install the database software and craete teh RAC DB.

Related Posts
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)
↧

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Database

April 7, 2019, 12:00 am
$
0
0
Following the 18c clusterware installation the next step is to install database software and create the database. The database software installation is not much different from that of 12.2 database software installation. Even though the oracle installation guides states gcc and gcc-c++ are not required for RHEL, the cluvfy still searches for gcc-c++.
1. Create the Oracle home for the database software and unzip the installer into the directory.
cd $ORACLE_BASE
mkdir -p product/18.x.0/dbhome_1
unzip LINUX.X64_180000_db_home.zip -d /opt/app/oracle/product/18.x.0/dbhome_1/
2. Run cluvfy to check the pre dbinst state. The failures related to shm could be ignored
cluvfy stage -pre dbinst -allnodes -r 18

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying User Mask ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying ASM storage privileges for the user: oracle ...
  Verifying Group Membership: asmdba ...PASSED
Verifying ASM storage privileges for the user: oracle ...PASSED
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying ACFS device special file ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying Maximum locked memory check ...PASSED

Pre-check for database installation was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbinst".

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
3. Execute runInstaller from the Oracle home and select software only install option.
Other steps are similar to 12.2. However, one noticeable change is that in 18c the space available for Oracle base is also checked as part of pre-reqs. Any warning raised for test system could be ignored.
4. Once the database software is installed proceed to creating the database. As this is a role separate setup, certain folders need write permission for the oinstall group so oracle user could write into them.
ls -l
total 0
drwxr-x---.  5 grid oinstall  45 Mar 20 11:50 admin
drwxr-x---.  2 grid oinstall   6 Mar 20 11:51 audit

[grid@rhel71 oracle]$ chmod 770 admin audit

[grid@rhel71 oracle]$ ls -l
total 0
drwxrwx---.  5 grid oinstall  45 Mar 20 11:50 admin
drwxrwx---.  2 grid oinstall   6 Mar 20 11:51 audit

chmod 770 $ORACLE_BASE/cfgtoollogs/dbca
5. Run cluvfy could to check pre-requisite for RAC DB (output is without verbose option).
cluvfy stage -pre dbcfg -allnodes -d /opt/app/oracle/product/18.x.0/dbhome_1

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dgdba ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: racdba ...PASSED
Verifying Group Existence: backupdba ...PASSED
Verifying Group Existence: oper ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: backupdba ...PASSED
Verifying Group Membership: racdba ...PASSED
Verifying Group Membership: dgdba ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Group Membership: oper ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Current Group ID ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying File system mount options for path ORACLE_HOME ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)

Pre-check for database configuration was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbcfg".

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
6. Create the database using dbca.
7. Create PDBs are required and associate services with them so PDBs are auto started. Assuming PDB named pdb18c is created, add a database service associated with PDB with
srvctl add service -db ent18c -pdb pdb18c -service pdbsrv -preferred "ent18c1,ent18c2"
srvctl start service -db ent18c -s pdbsrv
8. Finally, the stopping of the ASM proxy instances has no relevance to the "flex" functioning of DB instances. If an ASM instances in one node goes down, these DB instances are able to use a remote ASM instance to access the disks. This could be verified with the following.
[grid@rhel71 ~]$  srvctl status asm -detail
ASM is running on rhel71,rhel72
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 2
Client names: ent18c1:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
ASM instance +ASM2 is running on node rhel72
Number of connected clients: 3
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel72.domain.net:_OCR:rhel7-cluster
The above output shows the clients connected to each ASM instance on each node. Currently ASM1 has 2 clients while ASM2 has 3 clients, *OCR is available as client on both ASM instances.
Then an ASM instance is stopped.
srvctl stop asm  -node rhel72  -force
When one of the ASM instances is stopped forcefully those clients start using the service of the available ASM instance
srvctl status asm -detail
ASM is running on rhel71
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 4
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c1:ent18c:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
As seen above, clients that previously clients of ASM2 instance are now using ASM1 instance to access ASM disks. (total client count is 4 not 5 due to not duplicating the OCR* client).
When ASM instance is restarted the clients automatically select the local ASM instance over remote ASM instances.
srvctl start asm  -node rhel72
srvctl status asm -detail
ASM is running on rhel71,rhel72
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 2
Client names: ent18c1:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
ASM instance +ASM2 is running on node rhel72
Number of connected clients: 3
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel72.domain.net:_OCR:rhel7-cluster
This concludes the setting up RAC DB on 18c.

Related Posts
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Database
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Database Software
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Creating CDB & PDB
↧

DIA-49802: missing read, write, or execute permission on specified ADR home directory

April 15, 2019, 2:25 am
$
0
0
Executing an 18c (18.5) Oracle restart reconfiguration after a hostname change resulted in following error.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-25-04AM.log
2019/03/27 08:25:06 CLSRSC-363: User ignored prerequisites during installation
Oracle Clusterware infrastructure error in OCRCONFIG (OS PID 8212): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

Oracle Clusterware infrastructure error in CLSCFG (OS PID 8223): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:26:01 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

2019/03/27 08:28:17 CLSRSC-214: Failed to start the resource 'ohasd'
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2019-03-27 08:26:28.832
[client(8848)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8848
2019-03-27 08:26:28.845
[client(8848)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:31.155
[client(8900)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8900
2019-03-27 08:26:31.168
[client(8900)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:33.140
[client(8950)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8950
2019-03-27 08:26:33.152
[client(8950)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:35.195
[client(9000)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9000
2019-03-27 08:26:35.208
[client(9000)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:37.231
[client(9050)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9050
2019-03-27 08:26:37.243
[client(9050)]CRS-2112:The OLR service started on node ip-172-31-1-113.

2019/03/27 08:28:17 CLSRSC-318: Failed to start Oracle OHASD service
Died at /opt/app/oracle/product/18.0.0/grid/crs/install/crsinstall.pm line 3226.

It appears during the reconfiguration, which is executed as root user the folders inside $ORACLE_BASE/diag/crs/`hostname -s`/crs gets created with root ownership but are written to as oracle (or grid if role separation is used) user.
cd /opt/app/oracle/diag/crs/ip-172-31-1-113/crs
[root@ip-172-31-1-113 crs]# ls -l

drwxrwx---. 2 root root  4096 Mar 27 08:24 alert
drwxrwx---. 2 root root  4096 Mar 27 08:24 cdump
drwxrwx---. 2 root root  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 root root  4096 Mar 27 08:24 lck
drwxrwx---. 4 root root  4096 Mar 27 08:29 log
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 root root  4096 Mar 27 08:24 stage
drwxrwx---. 2 root root  4096 Mar 27 08:24 sweep
drwxrwx---. 2 root root 12288 Mar 27 08:33 trace


This issue only happens during reconfiguration. On a new installation the ownership is set to the user installing the grid software, either oracle or grid. Changing the ownership to oracle (or grid) user in these directories resolved the issue.
chown -R oracle:oinstall *

[root@ip-172-31-1-113 crs]# ls -l
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 alert
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 cdump
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incident
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 lck
drwxrwx---. 4 oracle oinstall  4096 Mar 27 08:29 log
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 stage
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 sweep
drwxrwx---. 2 oracle oinstall 20480 Mar 27 08:41 trace
Re-run the Oracle restart configuration again.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-38-56AM.log
2019/03/27 08:38:59 CLSRSC-363: User ignored prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:39:28 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113'
CRS-2673: Attempting to stop 'ora.evmd' on 'ip-172-31-1-113'
CRS-2677: Stop of 'ora.evmd' on 'ip-172-31-1-113' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.

ip-172-31-1-113     2019/03/27 08:42:19     /opt/app/oracle/product/18.0.0/grid/cdata/ip-172-31-1-113/backup_20190327_084219.olr     2532936542
2019/03/27 08:42:53 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

Related Post
Changing Hostname in a Standalone DB Configuration with ASM
↧
Search

Creating Gold Images on 18c

April 22, 2019, 12:00 am
$
0
0
Both gridSetup.sh in grid infrastructure and runInstaller in Oracle database software has the option to create gold images from an installation. This facilitate standardization and uniformity. Moreover each time a patch is applied a new gold image could be created which will include the patched binaries. So any new home created from the gold image will also include the patch. This post shows steps for creating gold image for grid infrastructure and oracle database.

Creating Gold Image of Grid Infrastructure
Gold image from a grid infrastructure setup is created using gridSetup.sh with the createGoldImage option. By default this expect X-windows to be available. If not the execution will fail.
./gridSetup.sh -createGoldImage -destinationLocation /media/giGold
ERROR: Unable to verify the graphical display setup. This application requires X display. Make sure that xdpyinfo exist under PATH variable.

Secondly the destination location must be at least 23GB, even though the final zip file generated could be 5-6 GB (depending on excluding certain files). Execution will fail if there isn't sufficient space
./gridSetup.sh -createGoldImage -destinationLocation /media/giGold -silent
Launching Oracle Grid Infrastructure Setup Wizard...

[FATAL] [INS-32707] The specified destination location (/media/giGold) does not have enough free space.
   ACTION: Provide a destination location path with at least (22,694) MB of free space.
If there's no risk of patches being rolled back, then the .patch_storage folder could be excluded from the gold image. This will reduce the size of the final zip file created. Other candidates for exclusions are files that have the server hostname in the name.
 ./gridSetup.sh -createGoldImage -destinationLocation /media/giGold -silent -exclFiles $GI_HOME/.patch_storage
Launching Oracle Database Setup Wizard...

Successfully Setup Software.
Gold Image location: /media/giGold/grid_home_2019-03-26_12-20-38PM.zip
Any new grid infrastructure installation done using the gold image will have the patches.
$ /opt/app/oracle/product/18.x.0/grid/OPatch/opatch lspatches
28864607;ACFS RELEASE UPDATE 18.5.0.0.0 (28864607)
28864593;OCW RELEASE UPDATE 18.5.0.0.0 (28864593)
28822489;Database Release Update : 18.5.0.0.190115 (28822489)
28547619;TOMCAT RELEASE UPDATE 18.0.0.0.0 (28547619)
28435192;DBWLM RELEASE UPDATE 18.0.0.0.0 (28435192)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
The gold image doesn't zip all the files in the current installation. Some directories such as log directories are excluded automatically. In previous versions when installing from a gold image, these missing folders caused issue. But this appear to be fixed on 18c and no issues were encountered when a new installation was done using the gold image.




Creating Gold Image of Oracle Database
The gold image for Oracle database software is created using runInstaller. 
./runInstaller -createGoldImage -destinationLocation /media/ohGold -silent -exclFiles $ORACLE_HOME/.patch_storage
Launching Oracle Database Setup Wizard...

Successfully Setup Software.
Gold Image location: /media/ohGold/db_home_2019-03-26_03-35-48PM.zip
As mentioned earlier patch storage could be excluded from the gold image. Any new database installation created using the gold image will have the patches already applied.
/opt/app/oracle/product/18.x.0/dbhome_1/OPatch/opatch lspatches
28864593;OCW RELEASE UPDATE 18.5.0.0.0 (28864593)
28822489;Database Release Update : 18.5.0.0.190115 (28822489)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
Any new database created will also have the post patch scripts applied.
SQL> select PATCH_ID,PATCH_TYPE,ACTION,STATUS,SOURCE_VERSION,TARGET_VERSION,ACTION_TIME from dba_registry_sqlpatch;

  PATCH_ID PATCH_TYPE ACTION          STATUS                    SOURCE_VERSION  TARGET_VERSION  ACTION_TIME
---------- ---------- --------------- ------------------------- --------------- --------------- ------------------------------
  28822489 RU         APPLY           SUCCESS                   18.1.0.0.0      18.5.0.0.0      27-MAR-19 12.10.44.167872 PM
  27923415 INTERIM    APPLY           SUCCESS                   18.1.0.0.0      18.1.0.0.0      27-MAR-19 12.10.44.171197 PM
↧
↧

Add or Remove Database Options with chopt

April 28, 2019, 12:00 am
$
0
0
chopt tool allows easier way of enabling or disabling certain database options after the database software installation. chopt does the binary relinking as part of enabling or disabling. Following shows an oracle database installation where real application testing is enabled (output generated using script here). 
RAC             [    INSTALLED     ]
RAT             [    INSTALLED     ]
OLS             [  NOT INSTALLED   ]
DV              [  NOT INSTALLED   ]
ASM             [  NOT INSTALLED   ]
OLAP            [    INSTALLED     ]
PART            [    INSTALLED     ]
CTX             [    INSTALLED     ]
chopt is used to disable RAT. Shutdown the database first and then use chopt.
srvctl stop database -db en18c

chopt disable rat

Writing to /opt/app/oracle/product/18.x.0/dbhome_1/install/disable_rat_2019-04-09_15-36-37pm.log...
/usr/bin/make -f /opt/app/oracle/product/18.x.0/dbhome_1/rdbms/lib/ins_rdbms.mk rat_off ORACLE_HOME=/opt/app/oracle/product/18.x.0/dbhome_1
/usr/bin/make -f /opt/app/oracle/product/18.x.0/dbhome_1/rdbms/lib/ins_rdbms.mk ioracle ORACLE_HOME=/opt/app/oracle/product/18.x.0/dbhome_1

srvctl start database -db en18c


Checking the output for enabled components after the change shows RAT is not installed any more.
RAC             [    INSTALLED     ]
RAT             [  NOT INSTALLED   ]
OLS             [  NOT INSTALLED   ]
DV              [  NOT INSTALLED   ]
ASM             [  NOT INSTALLED   ]
OLAP            [    INSTALLED     ]
PART            [    INSTALLED     ]
CTX             [    INSTALLED     ]
Currently chopt could be used to enable or disable Oracle Advanced Analytics, Oracle OLAP, Oracle Partitioning and Oracle Real Application Testing options in 12.1 and above. For 11.2 the list include Oracle Data Mining RDBMS Files, Oracle Database Vault option, Oracle Label Security, Oracle OLAP, Oracle Partitioning and Oracle Real Application Testing.
↧

Deleting a Node From 18c RAC

May 1, 2019, 12:00 am
$
0
0
Deleting a node from a 18c RAC is similar deleting a node from 12cR1 RAC. Deletion has three distinct phases, that is removing of the database instance, removing of Oracle database software and finally the clusterware.
The RAC setup in this case is a 2 node RAC and node named rhel72 will be removed from the cluster. The database is a CDB which has a single PDB.
SQL> select instance_number,instance_name,host_name from gv$instance;

INSTANCE_NUMBER INSTANCE_NAME    HOST_NAME
--------------- ---------------- --------------------
              1 ent18c1          rhel71.domain.net
              2 ent18c2          rhel72.domain.net #<-- node to be removed

SQL>  select inst_id,con_id,name from gv$pdbs order by 1;

   INST_ID     CON_ID NAME
---------- ---------- --------------
         1          2 PDB$SEED
         1          3 PDB18C
         2          2 PDB$SEED
         2          3 PDB18C
1. First phase include removing the database instance from the node to be deleted. For this run the DBCA on any node except on the node that has the instance being deleted. In this case DBCA is run from node rhel71. Follow the instance management option to remove the instance.
Select RAC database instance management and delete instance options.
Following message is shown with regard to the service associated with the PDB running on the instance being deleted. DBCA will modify the service to include only the remaining instances.

2. At the end of the DBCA run the database instance is removed from the node to be deleted
SQL>  select instance_number,instance_name,host_name from gv$instance;

INSTANCE_NUMBER INSTANCE_NAME    HOST_NAME
--------------- ---------------- -------------------
              1 ent18c1          rhel71.domain.net

SQL>  select inst_id,con_id,name from gv$pdbs order by 1;

   INST_ID     CON_ID NAME
---------- ---------- --------------
         1          2 PDB$SEED
         1          3 PDB18C
Oracle RAC configuration is updated to reflect the change instances along with the database service.
srvctl config database -db ent18c
Database unique name: ent18c
Database name: ent18c
Oracle home: /opt/app/oracle/product/18.x.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/ENT18C/PARAMETERFILE/spfile.272.1003495623
Password file: +DATA/ENT18C/PASSWORD/pwdent18c.256.1003490291
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,FRA
Mount point paths:
Services: pdbsrv
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: ent18c1
Configured nodes: rhel71
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed

srvctl config service -db ent18c -service pdbsrv
Service name: pdbsrv
Server pool:
Cardinality: 1
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
Failover retries:
Failover delay:
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name: pdb18c
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: ent18c1
Available instances:
CSS critical: no
3. Check if redo log threads for the deleted instance is removed from the database. If not remove database thread associated with the deleted instance.
SQL> select inst_id,group#,thread# from gv$log;

   INST_ID     GROUP#    THREAD#
---------- ---------- ----------
         1          1          1
         1          2          1
4. Once the instance removal is complete next step is to remove the database software. On 18c there's a difference to removing a node compared 12cR1. Updating the inventory file is not needed to remove the database software. Simply execute the deinstall with the local option on the node to be deleted. It's important to include the local option, without it the oracle database software on all nodes will get uninstalled.
cd /opt/app/oracle/product/18.x.0/dbhome_1/deinstall/

./deinstall -local
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /opt/app/oraInventory/logs/

############ ORACLE DECONFIG TOOL START ############


######################### DECONFIG CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /opt/app/oracle/product/18.x.0/dbhome_1
Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database
Oracle Base selected for deinstall is: /opt/app/oracle
Checking for existence of central inventory location /opt/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /opt/app/18.x.0/grid
The following nodes are part of this cluster: rhel72,rhel71
Checking for sufficient temp space availability on node(s) : 'rhel72'

## [END] Install check configuration ##


Network Configuration check config START

Network de-configuration trace file location: /opt/app/oraInventory/logs/netdc_check2019-04-04_02-52-54PM.log

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /opt/app/oraInventory/logs/databasedc_check2019-04-04_02-52-54PM.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured locally on this node for this Oracle home. Local configurations of the discovered databases will be removed [ent18c2]:
Database Check Configuration END

######################### DECONFIG CHECK OPERATION END #########################


####################### DECONFIG CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /opt/app/18.x.0/grid
The following nodes are part of this cluster: rhel72,rhel71
The cluster node(s) on which the Oracle home deinstallation will be performed are:rhel72
Oracle Home selected for deinstall is: /opt/app/oracle/product/18.x.0/dbhome_1
Inventory Location where the Oracle home registered is: /opt/app/oraInventory
The option -local will not modify any database configuration for this Oracle home.
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/opt/app/oraInventory/logs/deinstall_deconfig2019-04-04_02-51-47-PM.out'
Any error messages from this session will be written to: '/opt/app/oraInventory/logs/deinstall_deconfig2019-04-04_02-51-47-PM.err'

######################## DECONFIG CLEAN OPERATION START ########################
Database de-configuration trace file location: /opt/app/oraInventory/logs/databasedc_clean2019-04-04_02-52-54PM.log

Network Configuration clean config START

Network de-configuration trace file location: /opt/app/oraInventory/logs/netdc_clean2019-04-04_02-52-54PM.log

Network Configuration clean config END


######################### DECONFIG CLEAN OPERATION END #########################


####################### DECONFIG CLEAN OPERATION SUMMARY #######################
#######################################################################


############# ORACLE DECONFIG TOOL END #############

Using properties file /tmp/deinstall2019-04-04_02-50-37PM/response/deinstall_2019-04-04_02-51-47-PM.rsp
Location of logs /opt/app/oraInventory/logs/

############ ORACLE DEINSTALL TOOL START ############





####################### DEINSTALL CHECK OPERATION SUMMARY #######################
A log of this session will be written to: '/opt/app/oraInventory/logs/deinstall_deconfig2019-04-04_02-51-47-PM.out'
Any error messages from this session will be written to: '/opt/app/oraInventory/logs/deinstall_deconfig2019-04-04_02-51-47-PM.err'

######################## DEINSTALL CLEAN OPERATION START ########################
## [START] Preparing for Deinstall ##
Setting LOCAL_NODE to rhel72
Setting CLUSTER_NODES to rhel72
Setting CRS_HOME to false
Setting oracle.installer.invPtrLoc to /tmp/deinstall2019-04-04_02-50-37PM/oraInst.loc
Setting oracle.installer.local to true

## [END] Preparing for Deinstall ##

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/opt/app/oracle/product/18.x.0/dbhome_1' from the central inventory on the local node : Done

Delete directory '/opt/app/oracle/product/18.x.0/dbhome_1' on the local node : Done

The Oracle Base directory '/opt/app/oracle' will not be removed on local node. The directory is in use by Oracle Home '/opt/app/18.x.0/grid'.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##


## [END] Oracle install clean ##


######################### DEINSTALL CLEAN OPERATION END #########################


####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home '/opt/app/oracle/product/18.x.0/dbhome_1' from the central inventory on the local node.
Successfully deleted directory '/opt/app/oracle/product/18.x.0/dbhome_1' on the local node.
Oracle Universal Installer cleanup was successful.

Review the permissions and contents of '/opt/app/oracle' on nodes(s) 'rhel72'.
If there are no Oracle home(s) associated with '/opt/app/oracle', manually delete '/opt/app/oracle' and its contents.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL TOOL END #############



5. The last step is to remove the clusterware from the node. Check if the node to be deleted is active and unpinned. If the node is pinned then unpin it with crsctl unpin command. Following could be run as either grid user or root.
olsnodes -s -t
rhel71  Active  Unpinned
rhel72  Active  Unpinned
6. Similar to database software removal, there's no inventory update to remove the clusterware. Run deinstall with local option on the node to be deleted.
cd $GI_HOME/deinstall
./deinstall -local
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2019-04-04_03-00-47PM/logs/

############ ORACLE DECONFIG TOOL START ############


######################### DECONFIG CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /opt/app/18.x.0/grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster
Oracle Base selected for deinstall is: /opt/app/oracle
Checking for existence of central inventory location /opt/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /opt/app/18.x.0/grid
The following nodes are part of this cluster: rhel72,rhel71
Checking for sufficient temp space availability on node(s) : 'rhel72'

## [END] Install check configuration ##

Traces log file: /tmp/deinstall2019-04-04_03-00-47PM/logs//crsdc_2019-04-04_03-02-05-PM.log

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2019-04-04_03-00-47PM/logs/netdc_check2019-04-04_03-02-11PM.log

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2019-04-04_03-00-47PM/logs/asmcadc_check2019-04-04_03-02-11PM.log

Database Check Configuration START

Database de-configuration trace file location: /tmp/deinstall2019-04-04_03-00-47PM/logs/databasedc_check2019-04-04_03-02-11PM.log

Oracle Grid Management database was found in this Grid Infrastructure home

Database Check Configuration END

######################### DECONFIG CHECK OPERATION END #########################


####################### DECONFIG CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /opt/app/18.x.0/grid
The following nodes are part of this cluster: rhel72,rhel71
The cluster node(s) on which the Oracle home deinstallation will be performed are:rhel72
Oracle Home selected for deinstall is: /opt/app/18.x.0/grid
Inventory Location where the Oracle home registered is: /opt/app/oraInventory
Option -local will not modify any ASM configuration.
Oracle Grid Management database was found in this Grid Infrastructure home
Oracle Grid Management database will be relocated to another node during deconfiguration of local node
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2019-04-04_03-00-47PM/logs/deinstall_deconfig2019-04-04_03-01-58-PM.out'
Any error messages from this session will be written to: '/tmp/deinstall2019-04-04_03-00-47PM/logs/deinstall_deconfig2019-04-04_03-01-58-PM.err'

######################## DECONFIG CLEAN OPERATION START ########################
Database de-configuration trace file location: /tmp/deinstall2019-04-04_03-00-47PM/logs/databasedc_clean2019-04-04_03-02-11PM.log
ASM de-configuration trace file location: /tmp/deinstall2019-04-04_03-00-47PM/logs/asmcadc_clean2019-04-04_03-02-11PM.log
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2019-04-04_03-00-47PM/logs/netdc_clean2019-04-04_03-02-11PM.log

Network Configuration clean config END


Run the following command as the root user or the administrator on node "rhel72".

/opt/app/18.x.0/grid/crs/install/rootcrs.sh -force  -deconfig -paramfile "/tmp/deinstall2019-04-04_03-00-47PM/response/deinstall_OraGI18Home1.rsp"

Press Enter after you finish running the above commands

<----------------------------------------
Output from the command to be run as root.
# /opt/app/18.x.0/grid/crs/install/rootcrs.sh -force  -deconfig -paramfile "/tmp/deinstall2019-04-04_03-00-47PM/response/deinstall_OraGI18Home1.rsp"
Using configuration parameter file: /tmp/deinstall2019-04-04_03-00-47PM/response/deinstall_OraGI18Home1.rsp
The log of current session can be found at:
  /tmp/deinstall2019-04-04_03-00-47PM/logs/crsdeconfig_rhel72_2019-04-04_03-04-5                                                                                                              7PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel72'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel72'
CRS-2673: Attempting to stop 'ora.chad' on 'rhel72'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel72'
CRS-2673: Attempting to stop 'ora.OCRDG.dg' on 'rhel72'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel72'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.OCRDG.dg' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel72' succeeded
CRS-2673: Attempting to stop 'ora.GIMRDG.dg' on 'rhel72'
CRS-2677: Stop of 'ora.GIMRDG.dg' on 'rhel72' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel72'
CRS-2677: Stop of 'ora.asm' on 'rhel72' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72'
CRS-2677: Stop of 'ora.chad' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel72' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel72' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel72'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel72'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel72'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel72'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel72' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel72'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel72' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel72'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel72'
CRS-2677: Stop of 'ora.evmd' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel72' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel72'
CRS-2677: Stop of 'ora.cssd' on 'rhel72' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel72'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel72'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rhel72' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/04/04 15:08:05 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2019/04/04 15:08:43 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2019/04/04 15:09:11 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node

7. From the node remaining in the cluster run the node deletion command as root
# crsctl delete node -n rhel72
CRS-4661: Node rhel72 successfully deleted.
8. Use cluvfy to verify node deletion.
cluvfy stage -post nodedel -n rhel72

Verifying Node Removal ...
  Verifying CRS Integrity ...PASSED
  Verifying Clusterware Version Consistency ...PASSED
Verifying Node Removal ...PASSED

Post-check for node removal was successful.

CVU operation performed:      stage -post nodedel
Date:                         04-Apr-2019 15:12:44
CVU home:                     /opt/app/18.x.0/grid/
User:                         grid
This conclude the node deletion on 18c RAC.

Related Post
Deleting a Node From 12cR1 RAC
Deleting a Node From 11gR2 RAC
Deleting a 11gR1 RAC Node
↧

Upgrading Oracle Restart from 18c (18.6) to 19c (19.3)

May 3, 2019, 12:00 am
$
0
0
This post lists the steps for upgrading an Oracle restart environment (Single instance non-CDB on ASM) from 18.6 to 19.3. The 18c setup is on RHEL 7. The OS and kernel versions are as follows
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)

uname -r
3.10.0-957.el7.x86_64
The current versions of the GI are
crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [18.0.0.0.0]
crsctl query has softwareversion
Oracle High Availability Services version on the local node is [18.0.0.0.0]
The Oracle restart environment used is not a role separated setup, both Oracle and GI homes are installed using oracle user.
Download and run orachk -u -o pre (1268927.2) on the 18c installation to identify any required patches before the upgrade. With the above 18.6 RU applied no other patches were required be applied prior to the upgrade.
19c grid software instillation is also similar to 18c and is based on a image file. Before unzipping the GI image file create the 19c GI home directory and then unzip the GI image file into it.
mkdir -p /opt/app/oracle/product/19.x.0/grid
unzip LINUX.X64_193000_grid_home.zip -d /opt/app/oracle/product/19.x.0/grid
Use cluvfy with hacfg to verify pre-reqs. The SWAP size failure shown is ignorable on a test system.
cd /opt/app/oracle/product/19.x.0/grid/

./runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: ip-172-31-2-77:/usr,ip-172-31-2-77:/var,ip-172-31-2-77:/etc,ip-172-31-2-77:/sbin,ip-172-31-2-77:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.


Failures were encountered during execution of CVU verification request "stage -pre hacfg".

Verifying Swap Size ...FAILED
ip-172-31-2-77: PRVF-7573 : Sufficient swap size is not available on node
"ip-172-31-2-77" [Required = 16GB (1.6777216E7KB) ; Found = 0.0
                bytes]
ASM will be upgraded as part of the upgrade process.
Therefore stop the database before starting the GI upgrade.
srvctl stop database -d gold
Start the GI upgrade by running the gridSetup.sh from the grid home.
cd /opt/app/oracle/product/19.x.0/grid
./gridSetup.sh
Select upgrade GI option.
Location of the GI home cannot be changed.
Upgrade summary page.
Run root upgrade script when prompted.
Following shows the output from running the rootugprade script.
# /opt/app/oracle/product/19.x.0/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/oracle/product/19.x.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-2-77/crsconfig/roothas_2019-05-02_11-06-31AM.log
2019/05/02 11:06:34 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2019/05/02 11:06:37 CLSRSC-363: User ignored prerequisites during installation
2019/05/02 11:06:37 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.
2019/05/02 11:06:41 CLSRSC-595: Executing upgrade step 3 of 12: 'GenSiteGUIDs'.
2019/05/02 11:06:41 CLSRSC-595: Executing upgrade step 4 of 12: 'SetupOSD'.
2019/05/02 11:06:41 CLSRSC-595: Executing upgrade step 5 of 12: 'PreUpgrade'.

ASM has been upgraded and started successfully.

2019/05/02 11:07:21 CLSRSC-595: Executing upgrade step 6 of 12: 'UpgradeAFD'.
2019/05/02 11:07:22 CLSRSC-595: Executing upgrade step 7 of 12: 'UpgradeOLR'.
clscfg: EXISTING configuration version 0 detected.
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
2019/05/02 11:07:26 CLSRSC-595: Executing upgrade step 8 of 12: 'UpgradeOCR'.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-2-77 successfully pinned.
2019/05/02 11:07:29 CLSRSC-595: Executing upgrade step 9 of 12: 'CreateOHASD'.
2019/05/02 11:07:30 CLSRSC-595: Executing upgrade step 10 of 12: 'ConfigOHASD'.
2019/05/02 11:07:30 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2019/05/02 11:07:50 CLSRSC-595: Executing upgrade step 11 of 12: 'UpgradeSIHA'.

ip-172-31-2-77     2019/05/02 07:08:28     /opt/app/oracle/crsdata/ip-172-31-2-77/olr/backup_20190502_070828.olr     724960844

ip-172-31-2-77     2019/05/02 05:23:49     /opt/app/oracle/product/18.x.0/grid/cdata/ip-172-31-2-77/backup_20190502_052349.olr     2532936542
2019/05/02 11:08:28 CLSRSC-595: Executing upgrade step 12 of 12: 'InstallACFS'.
2019/05/02 11:10:08 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
Click OK on the execution configuration to proceed with the rest of the upgrade steps. Following shows the end of the upgrade page.
The HAS software is now upgraded to 19c.

crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]

crsctl query has softwareversion
Oracle High Availability Services version on the local node is [19.0.0.0.0]
If the ASM spfile was moved to $GI_HOME/dbs (described in step 12 in an earlier post) then this need to be moved to 19c GI_HOME. 
ASMCMD> spget
/opt/app/oracle/product/18.x.0/grid/dbs/spfile+ASM.ora
ASMCMD> spcopy /opt/app/oracle/product/18.x.0/grid/dbs/spfile+ASM.ora  /opt/app/oracle/product/19.x.0/grid/dbs/spfile+ASM.ora
ASMCMD> spset /opt/app/oracle/product/19.x.0/grid/dbs/spfile+ASM.ora
ASMCMD> spget
/opt/app/oracle/product/19.x.0/grid/dbs/spfile+ASM.ora
Use cluvfy post check validate as follows
cluvfy stage -post hacfg

Verifying Oracle Restart Integrity ...PASSED
Verifying OLR Integrity ...PASSED

Post-check for Oracle Restart configuration was successful.

CVU operation performed:      stage -post hacfg
Date:                         May 2, 2019 11:43:30 AM
CVU home:                     /opt/app/oracle/product/19.x.0/grid/
User:                         oracle


Next step is to upgrade the database software. Similar to GI home instillation, this too is based on image. Create the 19c oracle home location and unzip the DB software image file into it.
mkdir -p /opt/app/oracle/product/19.x.0/dbhome_1
unzip LINUX.X64_193000_db_home.zip -d /opt/app/oracle/product/19.x.0/dbhome_1
To begin the installation execute runInstaller from Oracle home location.
cd /opt/app/oracle/product/19.x.0/dbhome_1
./runInstaller
Select software only setup for the instillation option.
Select single database installation as this is an Oracle restart setup.
Select appropriate edition based on the licensing.
What is interesting in the above is that SE2 no longer mentions RAC usages where as in 18c it did. With 19c, RAC is not an option available under SE2.
Oracle home location is fixed and the location is where the DB software was unzipped.
There are no new OS groups introduced in 19c. So all the OS groups used 18c are used here as well.
DB software installation summary.
When prompted run the root script. Unlike 18c there's no prompt for installing TFA. A standalone TFA is setup as part of 19c.
# /opt/app/oracle/product/19.x.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/oracle/product/19.x.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Oracle Trace File Analyzer (TFA - Standalone Mode) is available at :
    /opt/app/oracle/product/19.x.0/dbhome_1/bin/tfactl

Note :
1. tfactl will use TFA Service if that service is running and user has been granted access
2. tfactl will configure TFA Standalone Mode only if user has no access to TFA Service or TFA is not installed


After the database software is installed next step is upgrade of the database. 19c has the option of autoupgrade (which is also backported to 12.2 and 18c) which automates the upgrade process. However, in this post the DB will be upgraded using the DBUA. Run preupgrade.jar available in the 19c home (19c_home/rdbms/admin) to check the upgrade readiness of the database (for more refer 2421552.1)
 $ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
19.0.0.0.0 Build: 1 on 2019-05-02T12:16:42

Upgrade-To version: 19.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  GOLD
     Container Name:  gold
       Container ID:  0
            Version:  18.0.0.0.0
     DB Patch Level:  Database Release Update : 18.6.0.0.190416 (29301631)
         Compatible:  18.0.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  31
  Database log mode:  NOARCHIVELOG
           Readonly:  FALSE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Real Application Clusters              [to be upgraded]  OPTION OFF
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  1.  (AUTOFIXUP) Gather stale data dictionary statistics prior to database
      upgrade in off-peak time using:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Dictionary statistics do not exist or are stale (not up-to-date).

      Dictionary statistics help the Oracle optimizer find efficient SQL
      execution plans and are essential for proper upgrade timing. Oracle
      recommends gathering dictionary statistics in the last 24 hours before
      database upgrade.

      For information on managing optimizer statistics, refer to the 18.0.0.0
      Oracle Database Upgrade Guide.

  2.  (AUTOFIXUP) Gather statistics on fixed objects prior the upgrade.

      None of the fixed object tables have had stats collected.

      Gathering statistics on fixed objects, if none have been gathered yet, is
      recommended prior to upgrading.

      For information on managing optimizer statistics, refer to the 18.0.0.0
      Oracle Database Upgrade Guide.

  INFORMATION ONLY
  ================
  3.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      TEMP                                20 MB       150 MB
      UNDOTBS1                           365 MB       412 MB

      Minimum tablespace sizes for upgrade are estimates.

  4.  Check the Oracle Backup and Recovery User's Guide for information on how
      to manage an RMAN recovery catalog schema.

      If you are using a version of the recovery catalog schema that is older
      than that required by the RMAN client version, then you must upgrade the
      catalog schema.

      It is good practice to have the catalog schema the same or higher version
      than the RMAN client version you are using.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database GOLD
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following

    SQL>@/opt/app/oracle/cfgtoollogs/gold/preupgrade/preupgrade_fixups.sql

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  5.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 31 and the target 19 release
      ships with time zone file version 32.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 19 Oracle Database
      Globalization Support Guide.

  6.  (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  7.  Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 18.0.0.0
      Oracle Database Upgrade Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database GOLD
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following

    SQL>@/opt/app/oracle/cfgtoollogs/gold/preupgrade/postupgrade_fixups.sql


==================
PREUPGRADE SUMMARY
==================
  /opt/app/oracle/cfgtoollogs/gold/preupgrade/preupgrade.log
  /opt/app/oracle/cfgtoollogs/gold/preupgrade/preupgrade_fixups.sql
  /opt/app/oracle/cfgtoollogs/gold/preupgrade/postupgrade_fixups.sql

Execute fixup scripts as indicated below:

Before upgrade:

Log into the database and execute the preupgrade fixups
@/opt/app/oracle/cfgtoollogs/gold/preupgrade/preupgrade_fixups.sql

After the upgrade:

Log into the database and execute the postupgrade fixups
@/opt/app/oracle/cfgtoollogs/gold/preupgrade/postupgrade_fixups.sql

Preupgrade complete: 2019-05-02T12:16:42
Run the pre-upgrade fixup script and then DBUA from the 19c home. Select the database to upgrade.
Check and resolve any validation concerns.
Parallel upgrade will speed up the upgrade process. This step also allows to specify whether timezone is also upgraded at the same time as the database.
The upgrade summary page.
Upgrade progress.
Upgrade result page.
Once the upgrade has finished execute the post-upgrade fixup script (mentioned in the output when preupgrade.jar was run). The DB components versions and status after the upgrade is as follows.
SQL> select comp_name,status,version,version_full from cdb_registry order by 1,2;

COMP_NAME                                STATUS          VERSION         VERSION_FULL
---------------------------------------- --------------- --------------- ---------------
Oracle Database Catalog Views            VALID           19.0.0.0.0      19.3.0.0.0
Oracle Database Packages and Types       VALID           19.0.0.0.0      19.3.0.0.0
Oracle Real Application Clusters         OPTION OFF      19.0.0.0.0      19.3.0.0.0
Oracle Text                              VALID           19.0.0.0.0      19.3.0.0.0
Oracle Workspace Manager                 VALID           19.0.0.0.0      19.3.0.0.0
Oracle XML Database                      VALID           19.0.0.0.0      19.3.0.0.0

6 rows selected.
The timezone file has been upgraded to 32.
SQL> select * from v$timezone_file;

FILENAME                     VERSION     CON_ID
-------------------- --------------- ----------
timezlrg_32.dat                   32          0
If satisfied with the upgrade and application testing then change the compatibility parameter on the DB and ASM disk groups.
SQL> alter system set compatible='19.0.0.0.0' scope=spfile;
shutdown immediate;
Login as grid user and then login to ASM instance as sysasm to change asm compatibility parameters.
SQL> alter diskgroup FRA SET attribute 'compatible.asm'='19.0.0.0.0';
SQL> alter diskgroup DATA  SET attribute 'compatible.asm'='19.0.0.0.0';
SQL> alter diskgroup fra set attribute 'compatible.rdbms'='19.0.0.0.0';
SQL> alter diskgroup data set attribute 'compatible.rdbms'='19.0.0.0.0';

SQL> select g.name,a.name,a.value from v$asm_diskgroup g, v$asm_attribute a where g.group_number=a.group_number and a.name like '%compat%';

NAME                 NAME                 VALUE
-------------------- -------------------- --------------------
DATA                 compatible.asm       19.0.0.0.0
DATA                 compatible.rdbms     19.0.0.0.0
FRA                  compatible.asm       19.0.0.0.0
FRA                  compatible.rdbms     19.0.0.0.0
After ASM compatibility parameters are updated start the database.
Finally run orachk -u -o post check the post upgrade state of the oracle restart setup.

Related Posts
Upgrading Oracle Restart from 12.2.0.1 to 18c
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 12.2.0.1
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 12.1.0.2 to 12.2.0.1
Upgrading Single Instance on ASM from 11.2.0.3 to 11.2.0.4
Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2
↧

Adding a Node to 18c RAC

May 17, 2019, 12:00 am
$
0
0
This post list the steps for adding a node to 18c RAC which is similar to that of adding a node to 12cR1 RAC. Node addition is done in three phases. Phase one is to add the clusterware to the new node. Second phase will add the database software and final phase will extend the database to the new node by creating a new instance for it.
The current cluster setup consists of a single node named rhel71 and a new node named rhel72 will be added to the cluster.
1. It is assumed that physical connections (shared storage connections, network) are made to the new node being added. The cluvfy tool could be used to check the pre-reqs for node addition are successful. In this case the failures relate to memory and shm mount point could be ignored as this is a test setup. 
cluvfy stage -pre nodeadd -n rhel72

Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/18.x.0/grid,rhel72:/sbin,rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhel71:/tmp ...FAILED (PRVF-7501)
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying User Existence: grid ...
  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying User Existence: root ...
  Verifying Users With Same UID: 0 ...PASSED
Verifying User Existence: root ...PASSED
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall ...PASSED
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Node Addition ...
  Verifying CRS Integrity ...PASSED
  Verifying Clusterware Version Consistency ...PASSED
  Verifying '/opt/app/18.x.0/grid' ...PASSED
Verifying Node Addition ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
  Verifying ASM device sharedness check ...
    Verifying Shared Storage Accessibility:/dev/oracleasm/data1,/dev/oracleasm/gimr,/dev/oracleasm/fra1,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3 ...PASSED
  Verifying ASM device sharedness check ...PASSED
  Verifying Access Control List check ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying Database home availability ...PASSED
Verifying OCR Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Network Time Protocol (NTP) ...
  Verifying '/etc/ntp.conf' ...PASSED
  Verifying '/etc/chrony.conf' ...PASSED
  Verifying '/var/run/ntpd.pid' ...PASSED
  Verifying '/var/run/chronyd.pid' ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying /boot mount ...PASSED
Verifying zeroconf check ...PASSED

Pre-check for node addition was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre nodeadd".

Verifying Physical Memory ...FAILED
rhel72: PRVF-7530 : Sufficient physical memory is not available on node
"rhel72" [Required physical memory = 8GB (8388608.0KB)]

rhel71: PRVF-7530 : Sufficient physical memory is not available on node
"rhel71" [Required physical memory = 8GB (8388608.0KB)]

Verifying Swap Size ...FAILED
rhel72: PRVF-7573 : Sufficient swap size is not available on node "rhel72"
        [Required = 4.686GB (4913676.0KB) ; Found = 3.7246GB (3905532.0KB)]

rhel71: PRVF-7573 : Sufficient swap size is not available on node "rhel71"
        [Required = 4.686GB (4913676.0KB) ; Found = 3.7246GB (3905532.0KB)]

Verifying Free Space:
rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhe
l71:/tmp ...FAILED
PRVF-7501 : Sufficient space is not available at location
"/opt/app/18.x.0/grid" on node "rhel71" [Required space = 6.9GB ; available
space = 6.0439GB ]
PRVF-7501 : Sufficient space is not available at location "/" on node "rhel71"
[Required space = [25MB (/usr) + 5MB (/var) + 25MB (/etc) + 6.9GB
(/opt/app/18.x.0/grid) + 10MB (/sbin) + 1GB (/tmp) = 7.9635GB ]; available
space = 6.0439GB ]

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm


CVU operation performed:      stage -pre nodeadd
Date:                         04-Apr-2019 16:15:51
CVU home:                     /opt/app/18.x.0/grid/
User:                         grid

2. To extend the cluster by installing clusterware on the new node run the gridSetup.sh from an existing node and select add more nodes to cluster option.

3. Execute the root script when prompted.
# /opt/app/18.x.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/18.x.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-04-08_11-08-58AM.log
2019/04/08 11:09:11 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/04/08 11:09:11 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/04/08 11:09:57 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/04/08 11:09:57 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/04/08 11:10:10 CLSRSC-363: User ignored prerequisites during installation
2019/04/08 11:10:10 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/04/08 11:10:11 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/04/08 11:10:17 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/04/08 11:10:23 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/04/08 11:10:23 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/04/08 11:10:32 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/04/08 11:10:35 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/04/08 11:10:35 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/04/08 11:10:45 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/04/08 11:10:46 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/04/08 11:10:48 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/04/08 11:10:49 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/04/08 11:12:25 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/04/08 11:12:29 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/04/08 11:13:52 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/04/08 11:13:55 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel72'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel72' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/04/08 11:14:10 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel72'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel72'
CRS-2676: Start of 'ora.mdnsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel72'
CRS-2676: Start of 'ora.gpnpd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel72'
CRS-2676: Start of 'ora.gipcd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel72'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel72'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel72'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel72'
CRS-2676: Start of 'ora.diskmon' on 'rhel72' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel72'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel72'
CRS-2676: Start of 'ora.ctssd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel72'
CRS-2676: Start of 'ora.storage' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel72'
CRS-2676: Start of 'ora.crsd' on 'rhel72' succeeded
CRS-6017: Processing resource auto-start for servers: rhel72
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel71'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72'
CRS-2672: Attempting to start 'ora.ons' on 'rhel72'
CRS-2672: Attempting to start 'ora.chad' on 'rhel72'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rhel72'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel71'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel72'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.ons' on 'rhel72' succeeded
CRS-2676: Start of 'ora.chad' on 'rhel72' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel72'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel72' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.chad' on 'rhel71'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rhel72'
CRS-2676: Start of 'ora.DATA.dg' on 'rhel72' succeeded
CRS-6016: Resource auto-start has completed for server rhel72
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/04/08 11:16:43 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/04/08 11:16:43 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2019/04/08 11:17:43 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.
2019/04/08 11:17:57 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded



4. Use cluvfy to verify node addition
cluvfy stage -post nodeadd -n rhel72

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Cluster Integrity ...PASSED
Verifying Node Addition ...
  Verifying CRS Integrity ...PASSED
  Verifying Clusterware Version Consistency ...PASSED
  Verifying '/opt/app/18.x.0/grid' ...PASSED
Verifying Node Addition ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Clock Synchronization ...PASSED

Post-check for node addition was successful.

CVU operation performed:      stage -post nodeadd
Date:                         08-Apr-2019 11:21:17
CVU home:                     /opt/app/18.x.0/grid/
User:                         grid
5. The next step is to extend the database software to new node. To add database software run addNode.sh from the $ORACLE_HOME/addnode directory as oracle user from an existing node.
cd $ORACLE_HOME/addnode
./addnode.sh "CLUSTER_NEW_NODES={rhel72}"
When the OUI starts the new node comes selected by default. When prompted run root.sh to complete the database software installation on the new node.

6. Before extending the database by adding a new instance change the permission on the admin folder on the new node.
cd $ORACLE_BASE
chmod 775 admin
7. Run DBCA and select RAC instance management and add an instance options.
Select the database to be extended to new node.
Specify new instance's details
6. Verify the database has been extended to include a new instance.
srvctl config database -d ent18c
Database unique name: ent18c
Database name: ent18c
Oracle home: /opt/app/oracle/product/18.x.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/ENT18C/PARAMETERFILE/spfile.272.1003495623
Password file: +DATA/ENT18C/PASSWORD/pwdent18c.256.1003490291
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,FRA
Mount point paths:
Services: pdbsrv
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: ent18c1,ent18c2
Configured nodes: rhel71,rhel72
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed

SQL> select inst_id,instance_name,host_name from gv$instance;

   INST_ID INSTANCE_NAME    HOST_NAME
---------- ---------------- ----------------------
         1 ent18c1          rhel71.domain.net
         2 ent18c2          rhel72.domain.net

SQL>  select inst_id,con_id,name from gv$pdbs order by 1;

   INST_ID     CON_ID NAME
---------- ---------- --------------
         1          2 PDB$SEED
         1          3 PDB18C
         2          2 PDB$SEED
         2          3 PDB18C
7. Modify the service to include the new database instance.
srvctl modify service -db ent18c -pdb pdb18c -s pdbsrv -modifyconfig -preferred "ent18c1,ent18c2"
srvctl config service -db ent18c -service pdbsrv
Service name: pdbsrv
Server pool:
Cardinality: 2
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
Failover retries:
Failover delay:
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name: pdb18c
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: ent18c1,ent18c2
Available instances:
CSS critical: no
This concludes the addition of a new node to 12cR1 RAC.

Related Posts
Adding a Node to 12cR1 RAC
Adding a Node to 11gR2 RAC
Adding a Node to 11gR1 RAC
↧
↧

ORA-03186: Cannot start Oracle ADG recovery on a non-Oracle Cloud database on a server that is not a primary server

June 14, 2019, 12:00 am
$
0
0
ORA-03186 occurs on a data guard configuration if the standby (or one of the standbys in a multiple standby configuration) was open for read only when the primary is restarted (or switchover happens between primary and another standby in a multiple standby configuration).
The setup is same setup used in postupgrading from 12.2 to 18c. The exact version of 18c is 18.6. All the servers are hosted on AWS. The current setup is as follows.
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  colombo - Primary database
    london  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 51 seconds ago)
When the primary is restarted
[oracle@ip-172-31-20-117 trace]$ srvctl stop database -d colombo
[oracle@ip-172-31-20-117 trace]$ srvctl start database -d colombo
redo shipping to standby stops.
Following message could be seen on the alert log of the standby instance (instance open in read only mode, london).
2019-05-29T05:38:20.414031-04:00
 rfs (PID:9340): Client is running on host ip-172-31-20-117.eu-west-1.compute.internal, not the current host ip-172-31-15-199.eu-west-1.compute.internal
2019-05-29T05:38:21.974469-04:00
 rfs (PID:9348): Client is running on host ip-172-31-20-117.eu-west-1.compute.internal, not the current host ip-172-31-15-199.eu-west-1.compute.internal
On the alert log of the primary (colombo) following could be seen.
 2019-05-29T05:38:22.283282-04:00
Errors in file /opt/app/oracle/diag/rdbms/colombo/colombo/trace/colombo_tt00_10673.trc:
ORA-03186: Cannot start Oracle ADG recovery on a non-Oracle Cloud database on a server that is not a primary server.
If the error message shown is
ORA-03816: Message 3816 not found;  product=RDBMS; facility=ORA
then apply patch 27539475 (on 18c) to get the above error message.
Inside the trace file following lines shows that attempts were made to ship redo to the instance open in read only mode.
*** 2019-05-29T05:38:22.282064-04:00
krsu_upi_status: Error 3186 attaching RFS server to standby instance at host 'londontns'
krsi_verify_network: Error 3186 attaching to LOG_ARCHIVE_DEST_2 standby host londontns
 at 0x7fffa4dc23c8 placed krsg.c@4998
ORA-03186: Cannot start Oracle ADG recovery on a non-Oracle Cloud database on a server that is not a primary server.
Data guard configuration shows error status
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  colombo - Primary database
    Error: ORA-16810: multiple errors or warnings detected for the member

    london  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR   (status updated 10 seconds ago)
Primary database configuration and status is shown below.
DGMGRL> show database verbose colombo

Database - colombo

  Role:               PRIMARY
  Intended State:     TRANSPORT-ON
  Instance(s):
    colombo
      Error: ORA-16737: the redo transport service for member "london" has an error

  Database Warning(s):
    ORA-16629: database reports a different protection level from the protection mode

  Properties:
    DGConnectIdentifier             = 'colombotns'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'SYNC'
    RedoRoutes                      = ''
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyLagThreshold               = '30'
    TransportLagThreshold           = '30'
    TransportDisconnectedThreshold  = '30'
    ApplyParallel                   = 'AUTO'
    ApplyInstances                  = '0'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '10'
    LogArchiveMinSucceedDest        = '1'
    DataGuardSyncLatency            = '0'
    DbFileNameConvert               = '/london/, /colombo/'
    LogFileNameConvert              = '/london/, /colombo/'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    PreferredObserverHosts          = ''
    HostName                        = 'ip-172-31-20-117.eu-west-1.compute.internal'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ip-172-31-20-117.eu-west-1.compute.internal)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=colombo_DGMGRL.codegen.net)(INSTANCE_NAME=colombo)(SERVER=DEDICATED)))'
    OnlineArchiveLocation           = ''
    OnlineAlternateLocation         = ''
    StandbyArchiveLocation          = ''
    StandbyAlternateLocation        = ''
    LogArchiveTrace                 = '2049'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'
    SidName                         = '(monitor)'

  Log file locations:
    Alert log               : /opt/app/oracle/diag/rdbms/colombo/colombo/trace/alert_colombo.log
    Data Guard Broker log   : /opt/app/oracle/diag/rdbms/colombo/colombo/trace/drccolombo.log

Database Status:
ERROR
Standby database configuration
DGMGRL> show database verbose london

Database - london

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 203 seconds ago)
  Apply Lag:          0 seconds (computed 203 seconds ago)
  Average Apply Rate: 62.00 KByte/s
  Active Apply Rate:  0 Byte/s
  Maximum Apply Rate: 0 Byte/s
  Real Time Query:    ON
  Instance(s):
    london

  Database Warning(s):
    ORA-16857: member disconnected from redo source for longer than specified threshold

  Properties:
    DGConnectIdentifier             = 'londontns'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'SYNC'
    RedoRoutes                      = ''
    DelayMins                       = '0'
    Binding                         = 'OPTIONAL'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyLagThreshold               = '30'
    TransportLagThreshold           = '30'
    TransportDisconnectedThreshold  = '30'
    ApplyParallel                   = 'AUTO'
    ApplyInstances                  = '0'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '10'
    LogArchiveMinSucceedDest        = '1'
    DataGuardSyncLatency            = '0'
    DbFileNameConvert               = '/colombo/, /london/'
    LogFileNameConvert              = '/colombo/, /london/'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    PreferredObserverHosts          = ''
    HostName                        = 'ip-172-31-15-199.eu-west-1.compute.internal'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ip-172-31-15-199.eu-west-1.compute.internal)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=london_DGMGRL.codegen.net)(INSTANCE_NAME=london)(SERVER=DEDICATED)))'
    OnlineArchiveLocation           = ''
    OnlineAlternateLocation         = ''
    StandbyArchiveLocation          = ''
    StandbyAlternateLocation        = ''
    LogArchiveTrace                 = '2049'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'
    SidName                         = '(monitor)'

  Log file locations:
    Alert log               : /opt/app/oracle/diag/rdbms/london/london/trace/alert_london.log
    Data Guard Broker log   : /opt/app/oracle/diag/rdbms/london/london/trace/drclondon.log

Database Status:
WARNING




There are no inconsistent properties in any of the databases.
DGMGRL> show database colombo 'InconsistentProperties';
INCONSISTENT PROPERTIES
   INSTANCE_NAME        PROPERTY_NAME         MEMORY_VALUE         SPFILE_VALUE         BROKER_VALUE

DGMGRL>  show database london  'InconsistentProperties';
INCONSISTENT PROPERTIES
   INSTANCE_NAME        PROPERTY_NAME         MEMORY_VALUE         SPFILE_VALUE         BROKER_VALUE
The archive dest status on the primary shows the same error as alert log.
SQL> select dest_id id,dest_name name, status, database_mode db_mode,recovery_mode,         protection_mode,standby_logfile_count "SRLs",        standby_logfile_active ACTIVE,         archived_seq#,error  from v$archive_dest_status where dest_id=2;

        ID NAME                 STATUS    DB_MODE         RECOVERY_MODE                      PROTECTION_MODE            SRLs     ACTIVE ARCHIVED_SEQ# ERROR
---------- -------------------- --------- --------------- ---------------------------------- -------------------- ---------- ---------- ------------- ----------------------------------------
         2 LOG_ARCHIVE_DEST_2   ERROR     UNKNOWN         IDLE                               RESYNCHRONIZATION             0          0             0 ORA-03186: Cannot start Oracle ADG
                                                                                                                                                      recovery on a non-Oracle Cloud database
                                                                                                                                                      on a server that is not a primary
                                                                                                                                                      server.

To resolve the redo shipping issue, restart the instance open in read only mode.
[oracle@ip-172-31-15-199 trace]$ srvctl stop database -d london
[oracle@ip-172-31-15-199 trace]$ srvctl start database -d london
Once it is started the redo shipping begins and log archive dest error clears.
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  colombo - Primary database
    london  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 11 seconds ago)

The other workaround is to put the read only instance into mount mode before the primary is stopped or switchover happens and open it in read only mode afterwards. Either method would result in clients being disconnected to from the read only instance.
The same issue was observed on a 19c multiple data guard configuration hosted on AWS.
DGMGRL> show configuration
Configuration - fcdg
Protection Mode: MaxPerformance
Members:
gold - Primary database
silver - Physical standby database
bronze - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS.
The silver instance is in mount only mode and bronze is open read only mode. When there was a switchover to between gold and silver, redo transport to bronze stopped from the new primary (silver) with the same error as in 18c. Only way to resolve the issue was to restart the instance open in read only mode.
However, 11.2.0.4 data guard configuration hosted on AWS did not have this issue. The redo shipping was working fine across primary restarts when the standby was open in read only mode. Seems whatever issue or removal of capabilities on non-oracle clouds happened in later versions.
↧

Adding a Node to 18c RAC Using Cloning

June 21, 2019, 12:00 am
$
0
0
This post shows steps of adding a node by way of cloning on 18c. On 19c, extending RAC by way of cloning is depreciated. However, the steps are still listed on 19c clusterware.
The post assumes all other pre-reqs related to adding a node have been completed. The new node is called rhel72.
1. As the first step create an image from an existing node. Steps in creating a gold image could be used for this.
2. Unzip gold image in the new node on the same directory structure as existing nodes.
3. Removing following files from the new node.
rm -rf $GI_HOME/network/admin/*.ora
rm -rf $GI_HOMEroot.sh*
4. Run gridsetup and select software only option.
Following is promoted about missing root.sh files. Ignore and click yes to continue.
Summary of software only setup.
When prompted execute the root scripts.
Root script output is shown below.
# /opt/app/18.x.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/18.x.0/grid

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Cluster or Grid Infrastructure for a Stand-Alone Server execute the following command as grid user:
/opt/app/18.x.0/grid/gridSetup.sh
This command launches the Grid Infrastructure Setup Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.
Complete the GI software only install.

5. From an existing node run gridSetup with noCopy option and select add node.
on exisitng node run
gridsetup.sh -noCopy
The noCopy option will prevent GI binaries being copied to remote node being added, since it's already has software installed. Rest of the steps for adding the node is same as previous add node post.Run the root.sh on the new node when prompted. Output from root.sh on the new node is shown below.
# /opt/app/18.x.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/18.x.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-04-09_11-10-41AM.log
2019/04/09 11:10:51 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/04/09 11:10:51 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/04/09 11:11:32 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/04/09 11:11:32 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/04/09 11:11:43 CLSRSC-363: User ignored prerequisites during installation
2019/04/09 11:11:44 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/04/09 11:11:45 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/04/09 11:11:50 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/04/09 11:11:57 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/04/09 11:11:57 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/04/09 11:12:34 CLSRSC-17: Invalid GPnP setup
2019/04/09 11:12:43 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/04/09 11:12:45 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/04/09 11:12:45 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/04/09 11:12:56 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/04/09 11:12:56 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/04/09 11:12:59 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/04/09 11:12:59 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/04/09 11:14:46 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/04/09 11:14:50 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/04/09 11:16:15 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/04/09 11:16:17 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel72'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel72' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/04/09 11:16:31 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel72'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel72'
CRS-2676: Start of 'ora.mdnsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel72'
CRS-2676: Start of 'ora.gpnpd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel72'
CRS-2676: Start of 'ora.gipcd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel72'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel72'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel72'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel72'
CRS-2676: Start of 'ora.diskmon' on 'rhel72' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel72'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel72'
CRS-2676: Start of 'ora.ctssd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel72'
CRS-2676: Start of 'ora.storage' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel72'
CRS-2676: Start of 'ora.crsd' on 'rhel72' succeeded
CRS-6017: Processing resource auto-start for servers: rhel72
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel71'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72'
CRS-2672: Attempting to start 'ora.ons' on 'rhel72'
CRS-2672: Attempting to start 'ora.chad' on 'rhel72'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.chad' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel71'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel72'
CRS-2676: Start of 'ora.ons' on 'rhel72' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel72'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel72' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rhel72'
CRS-2676: Start of 'ora.DATA.dg' on 'rhel72' succeeded
CRS-6016: Resource auto-start has completed for server rhel72
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/04/09 11:18:29 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/04/09 11:18:29 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2019/04/09 11:19:15 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.
2019/04/09 11:19:28 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded




6. Once the GI is cloned and node added, next is to clone the Oracle database software. For this too a gold image of the OH could be used.
7. Unzip the OH gold image on the new node following the same directory structure as existing nodes.
8. Run the clone script to configure and OH. On oracle documentation -noConfig is option is listed as part of the clone command. But on 18c this was not supported.
$ORACLE_HOME/perl/bin/perl clone.pl -silent -O 'CLUSTER_NODES={rhel71,rhel72}' -O LOCAL_NODE=rhel72 ORACLE_BASE=$ORACLE_BASE ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB18Home1 -O -noConfig
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 2595 MB    Passed
Checking swap space: must be greater than 500 MB.   Actual 3006 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-04-09_02-42-13PM. Please wait ...
[INS-04009] The argument [-noconfig] passed is not supported for the current context Clone
Run the clone command without the noConfig and cloning of OH completes without issue.
perl $ORACLE_HOME/clone/bin/clone.pl -silent ORACLE_HOME="/opt/app/oracle/product/18.x.0/dbhome_1"  ORACLE_HOME_NAME="OraDB18Home1" ORACLE_BASE="/opt/app/oracle""CLUSTER_NODES={rhel71,rhel72}" LOCAL_NODE=rhel72
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 3918 MB    Passed
Checking swap space: must be greater than 500 MB.   Actual 2974 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-04-09_03-26-55PM. Please wait ...You can find the log of this install session at:
 /opt/app/oraInventory/logs/cloneActions2019-04-09_03-26-55PM.log
..................................................   5% Done.
..................................................   10% Done.
..................................................   15% Done.
..................................................   20% Done.
..................................................   25% Done.
..................................................   30% Done.
..................................................   35% Done.
..................................................   40% Done.
..................................................   45% Done.
..................................................   50% Done.
..................................................   55% Done.
..................................................   60% Done.
..................................................   65% Done.
..................................................   70% Done.
..................................................   75% Done.
..................................................   80% Done.
..................................................   85% Done.
..........
Copy files in progress.

Copy files successful.

Link binaries in progress.
..........
Link binaries successful.

Setup files in progress.
..........
Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.
..........
Finish Setup successful.
The cloning of OraDB18Home1 was successful.
Please check '/opt/app/oraInventory/logs/cloneActions2019-04-09_03-26-55PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   95% Done.

As a root user, execute the following script(s):
        1. /opt/app/oracle/product/18.x.0/dbhome_1/root.sh

Execute /opt/app/oracle/product/18.x.0/dbhome_1/root.sh on the following nodes:
[rhel72]


..................................................   100% Done.

9. Run DBCA from an exiting node and select add instance option.
↧
Search

Root.sh Fails with CLSRSC-119: Start of the exclusive mode cluster failed

June 28, 2019, 12:00 am
$
0
0
On a two node cluster setup on 19c, running root.sh on the first node fails as below.
Using configuration parameter file: /opt/app/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/opt/app/oracle/crsdata/rhel71/crsconfig/rootcrs_rhel71_2019-06-03_05-55-22AM.log
2019/06/03 05:55:34 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2019/06/03 05:55:34 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2019/06/03 05:55:34 CLSRSC-363: User ignored prerequisites during installation
2019/06/03 05:55:34 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2019/06/03 05:55:34 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/06/03 05:55:38 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2019/06/03 05:55:39 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2019/06/03 05:55:39 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2019/06/03 05:55:40 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2019/06/03 05:55:42 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2019/06/03 05:55:48 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2019/06/03 05:55:49 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2019/06/03 05:56:22 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2019/06/03 05:56:24 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2019/06/03 05:56:24 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/06/03 05:56:47 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2019/06/03 05:56:49 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2019/06/03 05:56:52 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2019/06/03 05:56:57 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel71'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel71'
CRS-2676: Start of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel71'
CRS-2676: Start of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel71'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel71'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2674: Start of 'ora.gipcd' on 'rhel71' failed
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rhel71'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel71'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel71'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel71'
CRS-2677: Stop of 'ora.evmd' on 'rhel71' succeeded
CRS-4000: Command Start failed, or completed with errors.
2019/06/03 05:58:40 CLSRSC-119: Start of the exclusive mode cluster failed
Died at /opt/app/19.x.0/grid/crs/install/crsinstall.pm line 2439.
There are multiple MOS notes on root.sh failures as there are many different reasons for root.sh failures. But in this case MOS nor any of the logs generated during the installation revealed the real reason, hence this post.



Reason for this failure was the length of the cluster name. It seems the cluster name, which should be between 1 - 15 characters in length is never checked during the response file parsing (field oracle.install.crs.config.clusterName) or when installing via OUI to see if it is over the 15 character limit. Therefore it's possible to begin an installation with a cluster name longer than 15 characters as shown below (where the cluster name is 17 char long).

However, as shown earlier the installation will fail at root.sh execution time. Therefore, if root.sh fails on the first node check the length of the cluster name conform to the limit set by Oracle.

Related Post
Changing The Cluster Name
↧

4K Sector, compatible.rdbms and Redo Log File Block Size

July 1, 2019, 12:00 am
$
0
0
Disks these days comes in 512 or 4K sector sizes. Oracle DB is aware of the 4K sector size and supports it(including on ASM). The 4K sector size mainly affect the redo logs, standby redo log and archive logs. On DBs that are on Linux redo logs have a block size of 512 and could be affected negatively running on a 4K sector disk unless some additional work is done. 4K sector disks come in two types, native and emulated. In emulated mode (denoted as 512e), a 4K physical sector is presented as eight logical 512 sectors. On the native mode a one physical 4K sector is presented as a one logical 4k sector. The MOS notes mentioned at the end of the post has more information on these.
This post show how setting the compatible.rdbms could affect a the default block size of the newly created redo logs. The database and GI (for ASM) used for this was 19.3. OS is OEL 7.4 The setup was created on Azure.
For this setup, two separate disks were attached to the Azure VM to be used in two ASM groups. These two ASM groups will be used to host each member of a two member redo log group. Azure premium SSD disks are in emulation mode. This could be verified by looking at the logical and physical sector information.
# fdisk -l /dev/sde

Disk /dev/sde: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

# fdisk -l /dev/sdg

Disk /dev/sdg: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Another way to get the same information is to query the virtual file system.
# cat  /sys/block/sde/queue/physical_block_size
4096
# cat  /sys/block/sde/queue/logical_block_size
512

# cat  /sys/block/sdg/queue/physical_block_size
4096
# cat  /sys/block/sdg/queue/logical_block_size
512
A single partition was created on each disk taking in the whole disk. When using 4K sectors with ASM it's important to align the partition to avoid any additional IO due to misalignment. Follow MOS 1523947.1 for more on disk alignment. The start of the partition is aligned with the start of the sector by specifying the start to occur 1MB from the beginning of the disk.
parted /dev/sde
GNU Parted 3.1
Using /dev/sde
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mktable gpt
(parted) u
Unit?  [compact]? mib
(parted) p
Model: Msft Virtual Disk (scsi)
Disk /dev/sde: 4096MiB <====
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start  End  Size  File system  Name  Flags

(parted) mkpart redo1 1 4095 (above value - 1)

(parted) p
Model: Msft Virtual Disk (scsi)
Disk /dev/sde: 4096MiB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start    End      Size     File system  Name   Flags
 1      1.00MiB  4095MiB  4094MiB               redo1
Same partition viewed in sectors shows
(parted) u
Unit?  [MiB]? s
(parted) p
Model: Msft Virtual Disk (scsi)
Disk /dev/sde: 8388608s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start  End       Size      File system  Name   Flags
 1      2048s  8386559s  8384512s               redo1
Similarly the other disk is also partitioned
(parted) p
Model: Msft Virtual Disk (scsi)
Disk /dev/sdg: 8388608s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start  End       Size      File system  Name   Flags
 1      2048s  8386559s  8384512s               redo2
No ASMLib was used for creating the ASM disks. Instead udev rules were created for the partitions. Following that the ASM diskgroups were created with default options (no explicit sector sizes were set).
create diskgroup redo1 external redundancy disk '/dev/oracleasm/cgredo1' attribute 'au_size'='4m';
create diskgroup redo2 external redundancy disk '/dev/oracleasm/cgredo2' attribute 'au_size'='4m';
Each ASM disk group is then assigned online log destination 1 and 2.
SQL> alter system set db_create_online_log_dest_1='+redo1' scope=both;
SQL> alter system set db_create_online_log_dest_2='+redo2' scope=both;
The compatible and sector size attributes for each of the disk groups is shown below. What's interesting here is that the sector size is set to 512. The actual physical sector of 4K is not detected.
SQL>  select (select name from v$asm_diskgroup dg where dg.group_number=a.group_number) as dg_name,name,value from v$asm_attribute a where  group_number in (3,4) and (name like '%sector%' or name like '%compat%');

DG_NAME                        NAME                           VALUE
------------------------------ ------------------------------ --------------
REDO1                          sector_size                    512
REDO1                          compatible.asm                 11.2.0.2.0
REDO1                          compatible.rdbms               10.1.0.0.0
REDO1                          logical_sector_size            512
REDO2                          sector_size                    512
REDO2                          compatible.asm                 11.2.0.2.0
REDO2                          compatible.rdbms               10.1.0.0.0
REDO2                          logical_sector_size            512
At this stage when redo group is created the block size of the redo logs is 512. 
GROUP# BLOCKSIZE 
---------- ----------
         1 512
Changing the compatible.asm to 19.
alter diskgroup redo1 set attribute 'compatible.asm'='19.0.0.0.0';
alter diskgroup redo2 set attribute 'compatible.asm'='19.0.0.0.0';

DG_NAME                        NAME                           VALUE
------------------------------ ------------------------------ --------------
REDO1                          sector_size                    512
REDO1                          compatible.asm                 19.0.0.0.0
REDO1                          compatible.rdbms               10.1.0.0.0
REDO1                          logical_sector_size            512
REDO2                          sector_size                    512
REDO2                          compatible.asm                 19.0.0.0.0
REDO2                          compatible.rdbms               10.1.0.0.0
REDO2                          logical_sector_size            512
New redo logs created would still have 512 as block size. There's no warning with regard to log switches of existing groups.

Next the sector size is set to 4096, which is the physical sector size. Also the logical sector size is set explicitly to 512.
alter diskgroup redo1 set attribute 'sector_size'='4096';
alter diskgroup redo2 set attribute 'sector_size'='4096';

alter diskgroup redo1 set attribute 'logical_sector_size'='512';
alter diskgroup redo2 set attribute 'logical_sector_size'='512';

DG_NAME                        NAME                           VALUE
------------------------------ ------------------------------ -------------
REDO1                          sector_size                    4096
REDO1                          compatible.asm                 19.0.0.0.0
REDO1                          compatible.rdbms               10.1.0.0.0
REDO1                          logical_sector_size            512
REDO2                          sector_size                    4096
REDO2                          compatible.asm                 19.0.0.0.0
REDO2                          compatible.rdbms               10.1.0.0.0
REDO2                          logical_sector_size            512
Similar to previous times, new redo logs created would still have 512 as block size. There's no warning with regard to log switches of existing groups.



Finally the compatible.rdbms is set to 19.
alter diskgroup redo1 set attribute 'compatible.rdbms'='19.0.0.0.0';
alter diskgroup redo2 set attribute 'compatible.rdbms'='19.0.0.0.0';

DG_NAME                        NAME                           VALUE
------------------------------ ------------------------------ ---------------
REDO1                          sector_size                    4096
REDO1                          compatible.asm                 19.0.0.0.0
REDO1                          compatible.rdbms               19.0.0.0.0
REDO1                          logical_sector_size            512
REDO2                          sector_size                    4096
REDO2                          compatible.asm                 19.0.0.0.0
REDO2                          compatible.rdbms               19.0.0.0.0
REDO2                          logical_sector_size            512
At this stage when a new redo log group is added the block size defaults to 4096.
SQL> alter database add logfile group 3;

SQL> select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME     CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
         1          1        102  104857600        512          1 NO  CURRENT                 902579 30-MAY-19   1.8447E+19                    0
         2          1        101  104857600        512          1 NO  INACTIVE                902576 30-MAY-19       902579 30-MAY-19          0
         3          1          0  104857600       4096          1 YES UNUSED                       0                      0                    0
If a database was created after changing the ASM attribute (compat.rbdms to 19) then the newly created database's redo logs will ahve a block size of 4096.
Since the disks are in emulated mode, it's still possible to create redo groups with 512 block size.
alter database add logfile group 4 blocksize 512;

Database altered.
However, this will result in following warning shown on the alert log when log switching happens.
2019-05-30T14:59:59.922757+00:00
WARNING! THE SPECIFIED LOGFILE BLOCKSIZE (512) IS LESS THAN THE MEDIA SECTOR SIZE ON DISK (4096). LOGFILE WRITES ARE NOT SECTOR ALIGNED. THIS MAY LEAD TO DEGRADED PERFORMANCE DUE TO READ-MODIFY-WRITE OPERATIONS.
2019-05-30T15:00:04.977357+00:00
WARNING! THE SPECIFIED LOGFILE BLOCKSIZE (512) IS LESS THAN THE MEDIA SECTOR SIZE ON DISK (4096). LOGFILE WRITES ARE NOT SECTOR ALIGNED. THIS MAY LEAD TO DEGRADED PERFORMANCE DUE TO READ-MODIFY-WRITE OPERATIONS.
2019-05-30T15:00:10.072368+00:00
WARNING! THE LOGICAL BLOCKSIZE (512) OF LOGFILE MEMBER +REDO1/FCTEST/ONLINELOG/group_4.257.1009637999 IS LESS THAN THE PHYSICAL SECTOR SIZE (4096). LOGFILE WRITES ARE NOT SECTOR ALIGNED. THIS MAY LEAD TO DEGRADED PERFORMANCE DUE TO THE READ-MODIFY-WRITE OPERATIONS.
WARNING! THE LOGICAL BLOCKSIZE (512) OF LOGFILE MEMBER +REDO2/FCTEST/ONLINELOG/group_4.257.1009638005 IS LESS THAN THE PHYSICAL SECTOR SIZE (4096). LOGFILE WRITES ARE NOT SECTOR ALIGNED. THIS MAY LEAD TO DEGRADED PERFORMANCE DUE TO THE READ-MODIFY-WRITE OPERATIONS.


An interesting behaviour was observed during this changing of the compat.rdbms attribute. If only one of the ASM disk group has this attribute changed, then the outcome changes based on if it is the first or the second online destination. For example if it's only the first online destination's ASM diskgroup that has compat.rdbms changed as below
alter diskgroup redo1 set attribute 'compatible.rdbms'='19.0.0.0.0';

DG_NAME                        NAME                           VALUE
------------------------------ ------------------------------ --------------
REDO1                          sector_size                    4096
REDO1                          compatible.asm                 19.0.0.0.0
REDO1                          compatible.rdbms               19.0.0.0.0
REDO1                          logical_sector_size            512
REDO2                          sector_size                    4096
REDO2                          compatible.asm                 19.0.0.0.0
REDO2                          compatible.rdbms               10.1.0.0.0
REDO2                          logical_sector_size            512
then any attempt to create online redo log groups would fail as below (no issue for the existing redo logs).
SQL> alter database add logfile group 3;
alter database add logfile group 3
*
ERROR at line 1:
ORA-01378: The logical block size (4096) of file +redo2 is not compatible with the disk sector size (media sector size is 512 and host sector size is 512)
On the other hand if it's only the second online log destination that was changed (instead of both) as shown below
alter diskgroup redo2 set attribute 'compatible.rdbms'='19.0.0.0.0';

DG_NAME                        NAME                           VALUE
------------------------------ ------------------------------ ----------------
REDO1                          sector_size                    4096
REDO1                          compatible.asm                 19.0.0.0.0
REDO1                          compatible.rdbms               10.1.0.0.0
REDO1                          logical_sector_size            512
REDO2                          sector_size                    4096
REDO2                          compatible.asm                 19.0.0.0.0
REDO2                          compatible.rdbms               19.0.0.0.0
REDO2                          logical_sector_size            512
then newly created redo logs will have block size of 512. But during log switching warning is shown for the log file members who are in redo2 group.
SQL> alter database add logfile group 3;

Database altered.

SQL> select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME     CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
         1          1        156  104857600        512          2 NO  CURRENT                 903992 30-MAY-19   1.8447E+19                    0
         2          1          0  104857600        512          2 YES UNUSED                       0                      0                    0
         3          1          0  104857600        512          2 YES UNUSED                       0                      0                    0


2019-05-30T14:56:21.282852+00:00
WARNING! THE LOGICAL BLOCKSIZE (512) OF LOGFILE MEMBER +REDO2/FCTEST/ONLINELOG/group_3.256.1009637741 IS LESS THAN THE PHYSICAL SECTOR SIZE (4096). LOGFILE WRITES ARE NOT SECTOR ALIGNED. THIS MAY LEAD TO DEGRADED PERFORMANCE DUE TO THE   READ-MODIFY-WRITE OPERATIONS.
WARNING! THE LOGICAL BLOCKSIZE (512) OF LOGFILE MEMBER +REDO2/FCTEST/ONLINELOG/group_3.256.1009637741 IS LESS THAN THE PHYSICAL SECTOR SIZE (4096). LOGFILE WRITES ARE NOT SECTOR ALIGNED. THIS MAY LEAD TO DEGRADED PERFORMANCE DUE TO THE   READ-MODIFY-WRITE OPERATIONS.


Useful metalink notes
Supporting ASM on 4K/4096 Sector Size (SECTOR_SIZE) Disks [ID 1630790.1]
Alert: After SAN Firmware Upgrade, ASM Diskgroups ( Using ASMLIB) Cannot Be Mounted Due To ORA-15085: ASM disk "" has inconsistent sector size. [ID 1500460.1]
4096 Physical Block Size Support in Oracle Linux [ID 1582530.1]
Supporting 4K Sector Disks [ID 1133713.1]
ASM Diskgroups Cannot Be Created On RedHat6 Using 4K(4096) SECTOR_SIZE Non-ASMLIB Disks [ID 1961171.1]
Using 4k Redo Logs on Flash, 4k-Disk and SSD-based Storage [ID 1681266.1]
↧
↧

Changing OCI PaaS DB Shape

July 9, 2019, 12:00 am
$
0
0
Currently the OCI PaaS DB doesn't have a way of changing its shape once created. For example if the DB is doing more load than initially anticipated and require more CPU power to be added then there's no simple way to increase the CPU count, which may be available on other cloud platforms. However, Oracle does provide the ability to change the shape of the host sever indirectly. This is done by creating a new DB using an existing backup.
The details of the existing DB is shown below. This post shows the steps for changing the DB host shape from VM2.1 to VM2.2.
It is possible to create a database using the backups created by the automatic backup service.
However, this would mean any new data written to the database since the backup was taken won't be included in the newly created database. To avoid that stop any write activity to the database (this would mean down time for the application) and create a manual backup. The process of creating a backup nor database from backup doesn't require the existing database to be shutdown. But to keep the data consistent between the two DB instances it would require a stop to all writes by the application. Once the user initiated backup completes, write click on the backup and select create database.
If there's an existing bare metal DB system then it could be used to create the new database (from backup). But if it is a virtual machine then only option is to create a new DB system.
Another thing to look out for is that, if the new DB system is created on the same subnet then it cannot have the same hostname as the original DB's hostname. This would mean either creating the new DB system on a different subnet or with a different hostname in the same subnet. For this post the later approach was chosen. It is also possible to create the database with same name or different name (in the post the new database has a different name). Depending on which option chosen may require some reconfiguration work on the application such as changing the DB connection string.



When the new DB system option is selected it allows to specify a new shape, which in this case was set as VM2.2, whereas the original DB was on a shape of VM2.1.
As mentioned earlier it also allows to create the new DB system on a different subnet and specify a new database name as well.
The final outcome is two database system with same data but different shapes.
If this was done as a scale up activity then once data consistency is verified the original database system could be terminated.

Related Posts
Duplicate Database Without Target Connection or Catalog Connection
↧

Installing Enterprise Manager Cloud Control 13c (13.3)

August 29, 2019, 4:50 am
$
0
0
This post gives the highlights of installing EM Cloud Control 13c (13.3). There are earlier post of installing grid control 11g and upgrading grid control to 12c.
Database repository templates are available from Oracle site for EM 13.3 but these templates were for 12.1 DB. For this setup the repository database used was 12.2 as such no database templates were used. Certification information on MOS shows 12.2 could be used as a repository database. A custom database was created with following database components.
COMP_NAME                                STATUS
---------------------------------------- ----------
Oracle Database Catalog Views            VALID
Oracle Database Packages and Types       VALID
Oracle XML Database                      VALID
Oracle Workspace Manager                 VALID
Oracle Text                              VALID
There were no additional patches applied to the database. The OS is RHEL 7.4.
Once the database is created set the following parameters and creates other items as per specification (i.e. redo log size etc). These settings must be passed by the installer before it could proceed with the installation.
Once the above settings are done proceed with the installation. Installation could be registered with MOS and checked for updates, or simply skipped and proceed with the installation with the downloaded installer files.
Pre-req check would flag local port range values, set as part of the DB installation as insufficient. Increase is as per recommendation.
Advance option selection gives more option and customising capabilities during the instillation.
Set location for EM and agent software installation.
Select the plugins based on the monitoring targets.
Set credentials for weblogic domain and node manager.
Specify the DB repository credentials. In this setup both DB repository and Cloud control both reside on the same server. Based on the number of targets choose the deployment size. Based on this setting number of database initialization parameter values will be set. Refer Cloud Control documentation for more details.
Set the sysman and agent registration passwords.
It is possible to disable BI publisher and install it later. But in this setup BI publisher was installed and later the BI services were shutdown until it's used.
It's possible to customize the ports used by each service.
Installation summary.
When prompted run the root script. 

# /opt/app/software/em/middleware/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /opt/app/software/em/middleware/root.sh ......
/etc exist
/opt/app/software/em/middleware
Finished product-specific root actions.
/etc exist
Finished execution of  /opt/app/software/em/middleware/root.sh ......


Starting to execute /opt/app/software/em/agent/agent_13.3.0.0.0/root.sh ......
Finished product-specific root actions.
/etc exist
Finished execution of  /opt/app/software/em/agent/agent_13.3.0.0.0/root.sh ......
At the end of the installation the setup info is shown.
Login to the Cloud control using the sysman password of the repository DB. Other users could be created subsequently.
The starting and stopping of OMS could done following way.
To start all components
cd /opt/app/software/em/middleware/bin
./emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up
To check status
./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server is Up
To stop just the BI publisher
 ./emctl stop oms  -bip_only
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation.  All rights reserved.
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
BI Publisher Server is Down
To check the status of BI publisher
./emctl status oms -bip_only
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation.  All rights reserved.
BI Publisher Server is Down
To stop all components
 ./emctl stop oms  -all
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
The next step is to add remote monitoring targets.

Related Posts
Installing Grid Control 11gR1 and Deploying Agents
Upgrading Grid Control 11g to 12c - 1
↧

Installing 19c (19.3) RAC on RHEL 7 Using Response File

September 7, 2019, 12:00 am
$
0
0
This post shows the steps for installing 19c RAC using response file (silent installation). This post is not a step by step guide to installing 19c RAC. The majority of the pre-reqs are same as that of 18c RAC installation. For detail pre-reqs refer the Oracle 19c clusterware installation documentations.
There few notable differences on 19c RAC compared to previous versions. One of the main ones is RAC is no longer supported on standard edition 2. With 19c release RAC becomes effectively a enterprise edition only.
In terms of installation two of the notable differences are setting up GIMR has become optional. It is possible to install RAC without GIMR. Secondly the support for storing OCR and vote disks on shared file system is back. This was de-supported in 12.2.
This setup keeps in with the same configurations as previous RAC installation. That is, OCR and vote disks are stored in ASM and RAC is installed as a standard cluster.
Assuming the pre-reqs are completed, check the clusterware instillation readiness with cluvfy.
./runcluvfy.sh stage -pre crsinst -n rhel71,rhel72 -r 19 -osdba asmdba -osoper asmoper -asm -asmgrp asmadmin -presence local -asmdev /dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3,/dev/oracleasm/gimr,/dev/oracleasm/data1,/dev/oracleasm/fra1 -crshome /opt/app/19.x.0/grid -fixup  -orainv oinstall -method root

Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/19.x.0/grid,rhel72:/sbin,rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/19.x.0/grid,rhel71:/sbin,rhel71:/tmp ...PASSED
Verifying User Existence: grid ...
  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...PASSED
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying Device Checks for ASM ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
  Verifying ASM device sharedness check ...
    Verifying Shared Storage Accessibility:/dev/sdd1,/dev/sde1,/dev/sdf1,/dev/sdg1,/dev/sdb1,/dev/sdc1 ...PASSED
  Verifying ASM device sharedness check ...PASSED
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Time offset between nodes ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Domain Sockets ...PASSED
Verifying /boot mount ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying Grid Infrastructure home path: /opt/app/19.x.0/grid ...
  Verifying '/opt/app/19.x.0/grid' ...PASSED
Verifying Grid Infrastructure home path: /opt/app/19.x.0/grid ...PASSED
Verifying User Equivalence ...PASSED
Verifying RPM Package Manager database ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying File system mount options for path /var ...PASSED
Verifying DefaultTasksMax parameter ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Physical Memory ...FAILED
rhel72: PRVF-7530 : Sufficient physical memory is not available on node
"rhel72" [Required physical memory = 8GB (8388608.0KB)]

rhel71: PRVF-7530 : Sufficient physical memory is not available on node
"rhel71" [Required physical memory = 8GB (8388608.0KB)]

Verifying Swap Size ...FAILED
rhel72: PRVF-7573 : Sufficient swap size is not available on node "rhel72"
        [Required = 4.6848GB (4912356.0KB) ; Found = 3.7246GB (3905532.0KB)]

rhel71: PRVF-7573 : Sufficient swap size is not available on node "rhel71"
        [Required = 4.6848GB (4912368.0KB) ; Found = 3.7246GB (3905532.0KB)]

Verifying RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm


CVU operation performed:      stage -pre crsinst
Date:                         Jun 13, 2019 8:08:49 AM
CVU home:                     /opt/app/19.x.0/grid/
User:                         grid

The memory related failures are ignored (as this is a test system). To fix /dev/shm related failure refer 2065603.1.
GI installation provide several resposne (*.rsp) files that could be used to install grid. Though content of these files are the same they are located in different folders and named differently. Copy one of the following files to temporary location to be modified for the installation.
$GI_HOME/install/response/gridsetup.rsp
or
$GI_HOME/inventory/response/grid_install.rsp
Open the file copied to the temporary location and fill out section A-G "To register and configure 'Grid Infrastructure for Cluster'". Below is the content of the response file used for this setup.
###############################################################################
## Copyright(c) Oracle Corporation 1998,2019. All rights reserved.           ##
##                                                                           ##
## Specify values for the variables listed below to customize                ##
## your installation.                                                        ##
##                                                                           ##
## Each variable is associated with a comment. The comment                   ##
## can help to populate the variables with the appropriate                   ##
## values.                                                                   ##
##                                                                           ##
## IMPORTANT NOTE: This file contains plain text passwords and               ##
## should be secured to have read permission only by oracle user             ##
## or db administrator who owns this installation.                           ##
##                                                                           ##
###############################################################################

###############################################################################
##                                                                           ##
## Instructions to fill this response file                                   ##
## To register and configure 'Grid Infrastructure for Cluster'               ##
##  - Fill out sections A,B,C,D,E,F and G                                    ##
##  - Fill out section G if OCR and voting disk should be placed on ASM      ##
##                                                                           ##
## To register and configure 'Grid Infrastructure for Standalone server'     ##
##  - Fill out sections A,B and G                                            ##
##                                                                           ##
## To register software for 'Grid Infrastructure'                            ##
##  - Fill out sections A,B and D                                            ##
##  - Provide the cluster nodes in section D when choosing CRS_SWONLY as     ##
##    installation option in section A                                       ##
##                                                                           ##
## To upgrade clusterware and/or Automatic storage management of earlier     ##
## releases                                                                  ##
##  - Fill out sections A,B,C,D and H                                        ##
##                                                                           ##
## To add more nodes to the cluster                                          ##
##  - Fill out sections A and D                                              ##
##  - Provide the cluster nodes in section D when choosing CRS_ADDNODE as    ##
##    installation option in section A                                       ##
##                                                                           ##
###############################################################################

#------------------------------------------------------------------------------
# Do not change the following system generated value. 
#------------------------------------------------------------------------------
oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v19.0.0

###############################################################################
#                                                                             #
#                          SECTION A - BASIC                                  #
#                                                                             #
###############################################################################


#-------------------------------------------------------------------------------
# Specify the location which holds the inventory files.
# This is an optional parameter if installing on  
# Windows based Operating System.
#-------------------------------------------------------------------------------
INVENTORY_LOCATION=/opt/app/oraInventory

#-------------------------------------------------------------------------------
# Specify the installation option.
# Allowed values: CRS_CONFIG or HA_CONFIG or UPGRADE or CRS_SWONLY or HA_SWONLY
#   - CRS_CONFIG  : To register home and configure Grid Infrastructure for cluster
#   - HA_CONFIG   : To register home and configure Grid Infrastructure for stand alone server
#   - UPGRADE     : To register home and upgrade clusterware software of earlier release
#   - CRS_SWONLY  : To register Grid Infrastructure Software home (can be configured for cluster 
#                   or stand alone server later)
#   - HA_SWONLY   : To register Grid Infrastructure Software home (can be configured for stand 
#                   alone server later. This is only supported on Windows.)
#   - CRS_ADDNODE : To add more nodes to the cluster
#   - CRS_DELETE_NODE : To delete nodes to the cluster
#-------------------------------------------------------------------------------
oracle.install.option=CRS_CONFIG

#-------------------------------------------------------------------------------
# Specify the complete path of the Oracle Base.
#-------------------------------------------------------------------------------
ORACLE_BASE=/opt/app/oracle

################################################################################
#                                                                              #
#                              SECTION B - GROUPS                              #
#                                                                              #
#   The following three groups need to be assigned for all GI installations.   #
#   OSDBA and OSOPER can be the same or different.  OSASM must be different    #
#   than the other two.                                                        #
#   The value to be specified for OSDBA, OSOPER and OSASM group is only for    #
#   Unix based Operating System.                                               #
#   These groups are not required for upgrades, as they will be determined     #
#   from the Oracle home to upgrade.                                           #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
# The OSDBA_GROUP is the OS group which is to be granted SYSDBA privileges.
#-------------------------------------------------------------------------------
oracle.install.asm.OSDBA=asmdba

#-------------------------------------------------------------------------------
# The OSOPER_GROUP is the OS group which is to be granted SYSOPER privileges.
# The value to be specified for OSOPER group is optional.
# Value should not be provided if configuring Client Cluster - i.e. storageOption=CLIENT_ASM_STORAGE.
#-------------------------------------------------------------------------------
oracle.install.asm.OSOPER=asmoper

#-------------------------------------------------------------------------------
# The OSASM_GROUP is the OS group which is to be granted SYSASM privileges. This
# must be different than the previous two.
#-------------------------------------------------------------------------------
oracle.install.asm.OSASM=asmadmin

################################################################################
#                                                                              #
#                           SECTION C - SCAN                                   #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
# Specify the type of SCAN configuration for the cluster
# Allowed values : LOCAL_SCAN and SHARED_SCAN
#-------------------------------------------------------------------------------
oracle.install.crs.config.scanType=LOCAL_SCAN

#-------------------------------------------------------------------------------
# Applicable only if SHARED_SCAN is being configured for cluster
# Specify the path to the SCAN client data file
#-------------------------------------------------------------------------------
oracle.install.crs.config.SCANClientDataFile=

#-------------------------------------------------------------------------------
# Specify a name for SCAN
# Applicable if LOCAL_SCAN is being configured for the cluster
# If you choose to configure the cluster with GNS with Auto assigned Node VIPs(DHCP),then the scanName should be specified in the format of 'SCAN name.Cluster name.GNS sub-domain'
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.scanName=rac-scan.domain.net

#-------------------------------------------------------------------------------
# Specify a unused port number for SCAN service
#-------------------------------------------------------------------------------

oracle.install.crs.config.gpnp.scanPort=1521

################################################################################
#                                                                              #
#                           SECTION D - CLUSTER & GNS                         #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
# Specify the required cluster configuration
# Allowed values: STANDALONE, DOMAIN, MEMBERDB, MEMBERAPP
#-------------------------------------------------------------------------------
oracle.install.crs.config.ClusterConfiguration=STANDALONE

#-------------------------------------------------------------------------------
# Specify 'true' if you would like to configure the cluster as Extended, else
# specify 'false'
#
# Applicable only for STANDALONE and DOMAIN cluster configuration
#-------------------------------------------------------------------------------
oracle.install.crs.config.configureAsExtendedCluster=false


#-------------------------------------------------------------------------------
# Specify the Member Cluster Manifest file
#
# Applicable only for MEMBERDB and MEMBERAPP cluster configuration
#-------------------------------------------------------------------------------
oracle.install.crs.config.memberClusterManifestFile=

#-------------------------------------------------------------------------------
# Specify a name for the Cluster you are creating.
#
# The maximum length allowed for clustername is 63 characters. The name can be 
# any combination of lower and uppercase alphabets (A - Z), (0 - 9) and hyphens (-).
#
# Applicable only for STANDALONE and DOMAIN cluster configuration
#-------------------------------------------------------------------------------
oracle.install.crs.config.clusterName=rhel7-19c-clust

#-------------------------------------------------------------------------------
# Applicable only for STANDALONE, DOMAIN, MEMBERDB cluster configuration.
# Specify 'true' if you would like to configure Grid Naming Service(GNS), else
# specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.configureGNS=false

#-------------------------------------------------------------------------------
# Applicable only for STANDALONE and DOMAIN cluster configuration if you choose to configure GNS.
# Specify 'true' if you would like to assign SCAN name VIP and Node VIPs by DHCP
# , else specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.config.autoConfigureClusterNodeVIP=false

#-------------------------------------------------------------------------------
# Applicable only if you choose to configure GNS.
# Specify the type of GNS configuration for cluster
# Allowed values are: CREATE_NEW_GNS and USE_SHARED_GNS
# Only USE_SHARED_GNS value is allowed for MEMBERDB cluster configuration.
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.gnsOption=

#-------------------------------------------------------------------------------
# Applicable only if SHARED_GNS is being configured for cluster
# Specify the path to the GNS client data file
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.gnsClientDataFile=

#-------------------------------------------------------------------------------
# Applicable only for STANDALONE and DOMAIN cluster configuration if you choose to 
# configure GNS for this cluster oracle.install.crs.config.gpnp.gnsOption=CREATE_NEW_GNS
# Specify the GNS subdomain and an unused virtual hostname for GNS service
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.gnsSubDomain=
oracle.install.crs.config.gpnp.gnsVIPAddress=

#-------------------------------------------------------------------------------
# Specify the list of sites - only if configuring an Extended Cluster
#-------------------------------------------------------------------------------
oracle.install.crs.config.sites=

#-------------------------------------------------------------------------------
# Specify the list of nodes that have to be configured to be part of the cluster.
#
# The list should a comma-separated list of tuples.  Each tuple should be a
# colon-separated string that contains
# - 1 field if you have chosen CRS_SWONLY as installation option, or
# - 1 field if configuring an Application Cluster, or
# - 3 fields if configuring a Flex Cluster
# - 3 fields if adding more nodes to the configured cluster, or
# - 4 fields if configuring an Extended Cluster
# 
# The fields should be ordered as follows:
# 1. The first field should be the public node name.
# 2. The second field should be the virtual host name
#    (Should be specified as AUTO if you have chosen 'auto configure for VIP'
#     i.e. autoConfigureClusterNodeVIP=true)
# 3. The third field indicates the site designation for the node. To be specified only if configuring an Extended Cluster.
# Only the 1st field is applicable if you have chosen CRS_SWONLY as installation option
# Only the 1st field is applicable if configuring an Application Cluster
#
# Examples
# For registering GI for a cluster software: oracle.install.crs.config.clusterNodes=node1,node2
# For adding more nodes to the configured cluster: oracle.install.crs.config.clusterNodes=node1:node1-vip,node2:node2-vip
# For configuring Application Cluster: oracle.install.crs.config.clusterNodes=node1,node2
# For configuring Flex Cluster: oracle.install.crs.config.clusterNodes=node1:node1-vip,node2:node2-vip
# For configuring Extended Cluster: oracle.install.crs.config.clusterNodes=node1:node1-vip:site1,node2:node2-vip:site2
# You can specify a range of nodes in the tuple using colon separated fields of format
# hostnameprefix:lowerbound-upperbound:hostnamesuffix:vipsuffix:role of node
#
#-------------------------------------------------------------------------------
oracle.install.crs.config.clusterNodes=rhel71.domain.net:rhel71-vip,rhel72.domain.net:rhel72-vip

#-------------------------------------------------------------------------------
# The value should be a comma separated strings where each string is as shown below
# InterfaceName:SubnetAddress:InterfaceType
# where InterfaceType can be either "1", "2", "3", "4", or "5"
# InterfaceType stand for the following values
#   - 1 : PUBLIC
#   - 2 : PRIVATE
#   - 3 : DO NOT USE
#   - 4 : ASM
#   - 5 : ASM & PRIVATE
#
# For example: eth0:140.87.24.0:1,eth1:10.2.1.0:2,eth2:140.87.52.0:3
#
#-------------------------------------------------------------------------------
oracle.install.crs.config.networkInterfaceList=enp0s3:192.168.0.0:1,enp0s8:192.168.1.0:5

#------------------------------------------------------------------------------
# Specify 'true' if you would like to configure Grid Infrastructure Management
# Repository (GIMR), else specify 'false'.
# This option is only applicable when CRS_CONFIG is chosen as install option,
# and STANDALONE is chosen as cluster configuration.
#------------------------------------------------------------------------------
oracle.install.crs.configureGIMR=true

#------------------------------------------------------------------------------
# Create a separate ASM DiskGroup to store GIMR data.
# Specify 'true' if you would like to separate GIMR data with clusterware data, 
# else specify 'false'
# Value should be 'true' for DOMAIN cluster configurations
# Value can be true/false for STANDALONE cluster configurations.
#------------------------------------------------------------------------------
oracle.install.asm.configureGIMRDataDG=true

################################################################################
#                                                                              #
#                              SECTION E - STORAGE                             #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------
# Specify the type of storage to use for Oracle Cluster Registry(OCR) and Voting
# Disks files. Only applicable for Standalone and MemberDB cluster.
#   - FLEX_ASM_STORAGE
#   - CLIENT_ASM_STORAGE
#   - FILE_SYSTEM_STORAGE
#
# Option FILE_SYSTEM_STORAGE is only for STANDALONE cluster configuration.
#-------------------------------------------------------------------------------
oracle.install.crs.config.storageOption=FLEX_ASM_STORAGE

#-------------------------------------------------------------------------------
# These properties are applicable only if FILE_SYSTEM_STORAGE is chosen for 
# storing OCR and voting disk
# Specify the location(s) for OCR and voting disks
# Three(3) or one(1) location(s) should be specified for OCR and voting disk, 
# separated by commas.
# Example:
#     For Unix based Operating System:
#     oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=/oradbocfs/storage/vdsk1,/oradbocfs/storage/vdsk2,/oradbocfs/storage/vdsk3
#     oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=/oradbocfs/storage/ocr1,/oradbocfs/storage/ocr2,/oradbocfs/storage/ocr3
#     For Windows based Operating System OCR/VDSK on shared storage is not supported.
#-------------------------------------------------------------------------------
oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=
oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=                
################################################################################
#                                                                              #
#                               SECTION F - IPMI                               #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------
# Specify 'true' if you would like to configure Intelligent Power Management interface
# (IPMI), else specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.config.useIPMI=false

#-------------------------------------------------------------------------------
# Applicable only if you choose to configure IPMI
# i.e. oracle.install.crs.config.useIPMI=true
# Specify the username and password for using IPMI service
#-------------------------------------------------------------------------------
oracle.install.crs.config.ipmi.bmcUsername=
oracle.install.crs.config.ipmi.bmcPassword=
################################################################################
#                                                                              #
#                                SECTION G - ASM                               #
#                                                                              #
################################################################################


#-------------------------------------------------------------------------------
# Password for SYS user of Oracle ASM
#-------------------------------------------------------------------------------
oracle.install.asm.SYSASMPassword=test1234

#-------------------------------------------------------------------------------
# The ASM DiskGroup
#
# Example: oracle.install.asm.diskGroup.name=data
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.name=OCRDG

#-------------------------------------------------------------------------------
# Redundancy level to be used by ASM.
# It can be one of the following  
#   - NORMAL
#   - HIGH
#   - EXTERNAL
#   - FLEX#   - EXTENDED (required if oracle.install.crs.config.ClusterConfiguration=EXTENDED)
# Example: oracle.install.asm.diskGroup.redundancy=NORMAL
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.redundancy=NORMAL

#-------------------------------------------------------------------------------
# Allocation unit size to be used by ASM.
# It can be one of the following values
#   - 1
#   - 2
#   - 4
#   - 8
#   - 16
# Example: oracle.install.asm.diskGroup.AUSize=4
# size unit is MB
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.AUSize=4

#-------------------------------------------------------------------------------
# Failure Groups for the disk group
# If configuring for Extended cluster specify as list of "failure group name:site"
# tuples.
# Else just specify as list of failure group names
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.FailureGroups=

#-------------------------------------------------------------------------------
# List of disks and their failure groups to create a ASM DiskGroup
# (Use this if each of the disks have an associated failure group)
# Failure Groups are not required if oracle.install.asm.diskGroup.redundancy=EXTERNAL
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.diskGroup.disksWithFailureGroupNames=/oracle/asm/disk1,FGName,/oracle/asm/disk2,FGName
#     For Windows based Operating System:
#     oracle.install.asm.diskGroup.disksWithFailureGroupNames=\\.\ORCLDISKDATA0,FGName,\\.\ORCLDISKDATA1,FGName
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.disksWithFailureGroupNames=/dev/oracleasm/ocr1,,/dev/oracleasm/ocr2,,/dev/oracleasm/ocr3,

#-------------------------------------------------------------------------------
# List of disks to create a ASM DiskGroup
# (Use this variable only if failure groups configuration is not required)
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.diskGroup.disks=/oracle/asm/disk1,/oracle/asm/disk2
#     For Windows based Operating System:
#     oracle.install.asm.diskGroup.disks=\\.\ORCLDISKDATA0,\\.\ORCLDISKDATA1
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.disks=/dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3

#-------------------------------------------------------------------------------
# List of failure groups to be marked as QUORUM.
# Quorum failure groups contain only voting disk data, no user data is stored
# Example:
# oracle.install.asm.diskGroup.quorumFailureGroupNames=FGName1,FGName2
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.quorumFailureGroupNames=
#-------------------------------------------------------------------------------
# The disk discovery string to be used to discover the disks used create a ASM DiskGroup
#
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.diskGroup.diskDiscoveryString=/oracle/asm/*
#     For Windows based Operating System:
#     oracle.install.asm.diskGroup.diskDiscoveryString=\\.\ORCLDISK*
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/oracleasm/*

#-------------------------------------------------------------------------------
# Password for ASMSNMP account
# ASMSNMP account is used by Oracle Enterprise Manager to monitor Oracle ASM instances
#-------------------------------------------------------------------------------
oracle.install.asm.monitorPassword=test1234

#-------------------------------------------------------------------------------
# GIMR Storage data ASM DiskGroup
# Applicable only when 
# oracle.install.asm.configureGIMRDataDG=true
# Example: oracle.install.asm.GIMRDG.name=MGMT
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.name=GIMRDG

#-------------------------------------------------------------------------------
# Redundancy level to be used by ASM.
# It can be one of the following  
#   - NORMAL
#   - HIGH
#   - EXTERNAL
#   - FLEX#   - EXTENDED (only if oracle.install.crs.config.ClusterConfiguration=EXTENDED)
# Example: oracle.install.asm.gimrDG.redundancy=NORMAL
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.redundancy=EXTERNAL

#-------------------------------------------------------------------------------
# Allocation unit size to be used by ASM.
# It can be one of the following values
#   - 1
#   - 2
#   - 4
#   - 8
#   - 16
# Example: oracle.install.asm.gimrDG.AUSize=4
# size unit is MB
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.AUSize=4

#-------------------------------------------------------------------------------
# Failure Groups for the GIMR storage data ASM disk group
# If configuring for Extended cluster specify as list of "failure group name:site"
# tuples.
# Else just specify as list of failure group names
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.FailureGroups=

#-------------------------------------------------------------------------------
# List of disks and their failure groups to create GIMR data ASM DiskGroup
# (Use this if each of the disks have an associated failure group)
# Failure Groups are not required if oracle.install.asm.gimrDG.redundancy=EXTERNAL
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.gimrDG.disksWithFailureGroupNames=/oracle/asm/disk1,FGName,/oracle/asm/disk2,FGName
#     For Windows based Operating System:
#     oracle.install.asm.gimrDG.disksWithFailureGroupNames=\\.\ORCLDISKDATA0,FGName,\\.\ORCLDISKDATA1,FGName
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.disksWithFailureGroupNames=/dev/oracleasm/gimr,

#-------------------------------------------------------------------------------
# List of disks to create GIMR data ASM DiskGroup
# (Use this variable only if failure groups configuration is not required)
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.gimrDG.disks=/oracle/asm/disk1,/oracle/asm/disk2
#     For Windows based Operating System:
#     oracle.install.asm.gimrDG.disks=\\.\ORCLDISKDATA0,\\.\ORCLDISKDATA1
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.disks=/dev/oracleasm/gimr

#-------------------------------------------------------------------------------
# List of failure groups to be marked as QUORUM.
# Quorum failure groups contain only voting disk data, no user data is stored
# Example:
# oracle.install.asm.gimrDG.quorumFailureGroupNames=FGName1,FGName2
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.quorumFailureGroupNames=

#-------------------------------------------------------------------------------
# Configure AFD - ASM Filter Driver
# Applicable only for FLEX_ASM_STORAGE option
# Specify 'true' if you want to configure AFD, else specify 'false'
#-------------------------------------------------------------------------------
oracle.install.asm.configureAFD=false
#-------------------------------------------------------------------------------
# Configure RHPS - Rapid Home Provisioning Service
# Applicable only for DOMAIN cluster configuration
# Specify 'true' if you want to configure RHP service, else specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.configureRHPS=false

################################################################################
#                                                                              #
#                             SECTION H - UPGRADE                              #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
# Specify whether to ignore down nodes during upgrade operation.
# Value should be 'true' to ignore down nodes otherwise specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.config.ignoreDownNodes=false                
################################################################################
#                                                                              #
#                               MANAGEMENT OPTIONS                             #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------
# Specify the management option to use for managing Oracle Grid Infrastructure
# Options are:
# 1. CLOUD_CONTROL - If you want to manage your Oracle Grid Infrastructure with Enterprise Manager Cloud Control.
# 2. NONE   -If you do not want to manage your Oracle Grid Infrastructure with Enterprise Manager Cloud Control.
#-------------------------------------------------------------------------------
oracle.install.config.managementOption=NONE

#-------------------------------------------------------------------------------
# Specify the OMS host to connect to Cloud Control.
# Applicable only when oracle.install.config.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.config.omsHost=

#-------------------------------------------------------------------------------
# Specify the OMS port to connect to Cloud Control.
# Applicable only when oracle.install.config.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.config.omsPort=0

#-------------------------------------------------------------------------------
# Specify the EM Admin user name to use to connect to Cloud Control.
# Applicable only when oracle.install.config.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.config.emAdminUser=

#-------------------------------------------------------------------------------
# Specify the EM Admin password to use to connect to Cloud Control.
# Applicable only when oracle.install.config.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.config.emAdminPassword=
################################################################################
#                                                                              #
#                      Root script execution configuration                     #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------------------------------
# Specify the root script execution mode.
#
#   - true  : To execute the root script automatically by using the appropriate configuration methods.
#   - false : To execute the root script manually.
#
# If this option is selected, password should be specified on the console.
#-------------------------------------------------------------------------------------------------------
oracle.install.crs.rootconfig.executeRootScript=false

#--------------------------------------------------------------------------------------
# Specify the configuration method to be used for automatic root script execution.
#
# Following are the possible choices:
#   - ROOT
#   - SUDO
#--------------------------------------------------------------------------------------
oracle.install.crs.rootconfig.configMethod=
#--------------------------------------------------------------------------------------
# Specify the absolute path of the sudo program.
#
# Applicable only when SUDO configuration method was chosen.
#--------------------------------------------------------------------------------------
oracle.install.crs.rootconfig.sudoPath=

#--------------------------------------------------------------------------------------
# Specify the name of the user who is in the sudoers list. 
# Applicable only when SUDO configuration method was chosen.
# Note:For Grid Infrastructure for Standalone server installations,the sudo user name must be the username of the user performing the installation.
#--------------------------------------------------------------------------------------
oracle.install.crs.rootconfig.sudoUserName=
#--------------------------------------------------------------------------------------
# Specify the nodes batch map.
#
# This should be a comma separated list of node:batch pairs.
# During upgrade, you can sequence the automatic execution of root scripts
# by pooling the nodes into batches. 
# A maximum of three batches can be specified. 
# Installer will execute the root scripts on all the nodes in one batch before
# proceeding to next batch.
# Root script execution on the local node must be in Batch 1.
#
# Examples: 
# 1. oracle.install.crs.config.batchinfo=Node1:1,Node2:2,Node3:2,Node4:3
# 2. oracle.install.crs.config.batchinfo=Node1:1,Node2:2,Node3:2,Node4:2
# 3. oracle.install.crs.config.batchinfo=Node1:1,Node2:1,Node3:2,Node4:3
#
# Applicable only for UPGRADE install option. 
#--------------------------------------------------------------------------------------
oracle.install.crs.config.batchinfo=
################################################################################
#                                                                              #
#                           APPLICATION CLUSTER OPTIONS                        #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------
# Specify the Virtual hostname to configure virtual access for your Application
# The value to be specified for Virtual hostname is optional.
#-------------------------------------------------------------------------------
oracle.install.crs.app.applicationAddress=
#################################################################################
#          #
#       DELETE NODE OPTIONS     #
#          #
#################################################################################

#--------------------------------------------------------------------------------
# Specify the node names to delete nodes from cluster. 
# Delete node will be performed only for the remote nodes from the cluster.
#--------------------------------------------------------------------------------
oracle.install.crs.deleteNode.nodes=

Start the grid installation processes by running gridSetup.sh and specifying the response file. The ignorePrereqFailure is used here due the failure of memory related pre-reqs.
./gridSetup.sh -silent -responseFile /media/grid19c.rsp -ignorePrereqFailure
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. gridSetupActions2019-06-13_08-14-04AM.log
   ACTION: Identify the list of failed prerequisite checks from the log: gridSetupActions2019-06-13_08-14-04AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /opt/app/19.x.0/grid/install/response/grid_2019-06-13_08-14-04AM.rsp

You can find the log of this install session at:
 /tmp/GridSetupActions2019-06-13_08-14-04AM/gridSetupActions2019-06-13_08-14-04AM.log

As a root user, execute the following script(s):
        1. /opt/app/oraInventory/orainstRoot.sh
        2. /opt/app/19.x.0/grid/root.sh

Execute /opt/app/oraInventory/orainstRoot.sh on the following nodes:
[rhel71, rhel72]
Execute /opt/app/19.x.0/grid/root.sh on the following nodes:
[rhel71, rhel72]

Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes.

Successfully Setup Software with warning(s).
As install user, execute the following command to complete the configuration.
        /opt/app/19.x.0/grid/gridSetup.sh -executeConfigTools -responseFile /media/grid19c.rsp [-silent]

When prompted run the root scripts. Unlike a GUI based installation, in a silent mode installation the root script output is written a logfile and not shown on the console. Root script outptu from first node.
cat /opt/app/19.x.0/grid/install/root_rhel71.domain.net_2019-06-13_08-25-41-362238834.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/19.x.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel71/crsconfig/rootcrs_rhel71_2019-06-13_08-26-09AM.log
2019/06/13 08:26:35 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2019/06/13 08:26:35 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2019/06/13 08:26:35 CLSRSC-363: User ignored prerequisites during installation
2019/06/13 08:26:36 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2019/06/13 08:26:40 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2019/06/13 08:26:42 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2019/06/13 08:26:42 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2019/06/13 08:26:44 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2019/06/13 08:27:39 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/06/13 08:27:42 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2019/06/13 08:27:52 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2019/06/13 08:28:19 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2019/06/13 08:28:19 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2019/06/13 08:28:31 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2019/06/13 08:28:32 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/06/13 08:29:48 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2019/06/13 08:30:03 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2019/06/13 08:31:31 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2019/06/13 08:31:43 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.

ASM has been created and started successfully.

[DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190613AM083230.log for details.

2019/06/13 08:33:51 CLSRSC-482: Running command: '/opt/app/19.x.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-4256: Updating the profile
Successful addition of voting disk a379bd1a1f614fcfbf2bc03dc4edd382.
Successful addition of voting disk e6caeaf494054f35bf6144b6ae79fd31.
Successful addition of voting disk cf400723672e4f0ebf65845059245bf1.
Successfully replaced voting disk group with +OCRDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a379bd1a1f614fcfbf2bc03dc4edd382 (/dev/oracleasm/ocr3) [OCRDG]
 2. ONLINE   e6caeaf494054f35bf6144b6ae79fd31 (/dev/oracleasm/ocr1) [OCRDG]
 3. ONLINE   cf400723672e4f0ebf65845059245bf1 (/dev/oracleasm/ocr2) [OCRDG]
Located 3 voting disk(s).
2019/06/13 08:36:34 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2019/06/13 08:38:05 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/06/13 08:38:05 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2019/06/13 08:41:34 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190613AM084151.log for details.


2019/06/13 08:44:51 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Root script outptu from last node.
cat /opt/app/19.x.0/grid/install/root_rhel72.domain.net_2019-06-13_08-45-22-859282472.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/19.x.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-06-13_08-45-55AM.log
2019/06/13 08:46:15 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2019/06/13 08:46:15 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2019/06/13 08:46:15 CLSRSC-363: User ignored prerequisites during installation
2019/06/13 08:46:16 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2019/06/13 08:46:19 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2019/06/13 08:46:19 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2019/06/13 08:46:19 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2019/06/13 08:46:22 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2019/06/13 08:46:24 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2019/06/13 08:46:24 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2019/06/13 08:46:38 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2019/06/13 08:46:39 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2019/06/13 08:46:42 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2019/06/13 08:46:42 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/06/13 08:47:15 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/06/13 08:47:48 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2019/06/13 08:47:52 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2019/06/13 08:49:10 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2019/06/13 08:49:13 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2019/06/13 08:49:31 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2019/06/13 08:50:36 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/06/13 08:50:36 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2019/06/13 08:51:11 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2019/06/13 08:51:31 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Finally run the configuration tools script.
/opt/app/19.x.0/grid/gridSetup.sh -executeConfigTools -responseFile /media/grid19c.rsp -silent
Launching Oracle Grid Infrastructure Setup Wizard...

You can find the logs of this session at:
/opt/app/oraInventory/logs/GridSetupActions2019-06-13_08-52-50AM

You can find the log of this install session at:
 /opt/app/oraInventory/logs/UpdateNodeList2019-06-13_08-52-50AM.log
Successfully Configured Software.
Check the clusterware post installation setup with cluvfy.
cluvfy stage -post crsinst -allnodes

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying User Mask ...PASSED
Verifying Cluster Integrity ...PASSED
Verifying OCR Integrity ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Voting Disk ...PASSED
Verifying ASM Integrity ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Access control attributes for /var/tmp/.oracle ...PASSED
Verifying Access control attributes for %OCRCONFIGDIR%/maps ...PASSED

Post-check for cluster services setup was successful.

CVU operation performed:      stage -post crsinst
Date:                         Jun 13, 2019 9:49:15 AM
CVU home:                     /opt/app/19.x.0/grid/
User:                         grid

In the 19c RAC setup, unlike in the 18c RAC installation the asm proxy instance, acfs volumes are by default in offline state.
Resource Name                                 Type                      Target             State              Host
-------------                                 ------                    -------            --------           ----------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)        ora.asm_listener.type     ONLINE             ONLINE             rhel71
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)        ora.asm_listener.type     ONLINE             ONLINE             rhel72
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)        ora.asm_listener.type     OFFLINE            OFFLINE
ora.DATA.dg(ora.asmgroup)                     ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.DATA.dg(ora.asmgroup)                     ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.DATA.dg(ora.asmgroup)                     ora.diskgroup.type        ONLINE             OFFLINE
ora.FRA.dg(ora.asmgroup)                      ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.FRA.dg(ora.asmgroup)                      ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.FRA.dg(ora.asmgroup)                      ora.diskgroup.type        ONLINE             OFFLINE
ora.GIMRDG.GHCHKPT.advm                       ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.GHCHKPT.advm                       ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.dg(ora.asmgroup)                   ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.GIMRDG.dg(ora.asmgroup)                   ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.GIMRDG.dg(ora.asmgroup)                   ora.diskgroup.type        OFFLINE            OFFLINE
ora.LISTENER.lsnr                             ora.listener.type         ONLINE             ONLINE             rhel71
ora.LISTENER.lsnr                             ora.listener.type         ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN1.lsnr                       ora.scan_listener.type    ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN2.lsnr                       ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.LISTENER_SCAN3.lsnr                       ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.MGMTLSNR                                  ora.mgmtlsnr.type         ONLINE             ONLINE             rhel71
ora.OCRDG.dg(ora.asmgroup)                    ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.OCRDG.dg(ora.asmgroup)                    ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.OCRDG.dg(ora.asmgroup)                    ora.diskgroup.type        OFFLINE            OFFLINE
ora.asm(ora.asmgroup)                         ora.asm.type              ONLINE             ONLINE             rhel71
ora.asm(ora.asmgroup)                         ora.asm.type              ONLINE             ONLINE             rhel72
ora.asm(ora.asmgroup)                         ora.asm.type              OFFLINE            OFFLINE
ora.asmnet1.asmnetwork(ora.asmgroup)          ora.asm_network.type      ONLINE             ONLINE             rhel71
ora.asmnet1.asmnetwork(ora.asmgroup)          ora.asm_network.type      ONLINE             ONLINE             rhel72
ora.asmnet1.asmnetwork(ora.asmgroup)          ora.asm_network.type      OFFLINE            OFFLINE
ora.chad                                      ora.chad.type             ONLINE             ONLINE             rhel71
ora.chad                                      ora.chad.type             ONLINE             ONLINE             rhel72
ora.cvu                                       ora.cvu.type              ONLINE             ONLINE             rhel71
ora.ent19c.db                                 ora.database.type         ONLINE             ONLINE             rhel71
ora.ent19c.db                                 ora.database.type         ONLINE             ONLINE             rhel72
ora.gimrdg.ghchkpt.acfs                       ora.acfs.type             OFFLINE            OFFLINE
ora.gimrdg.ghchkpt.acfs                       ora.acfs.type             OFFLINE            OFFLINE
ora.helper                                    ora.helper.type           OFFLINE            OFFLINE
ora.helper                                    ora.helper.type           OFFLINE            OFFLINE
ora.mgmtdb                                    ora.mgmtdb.type           ONLINE             ONLINE             rhel71
ora.net1.network                              ora.network.type          ONLINE             ONLINE             rhel71
ora.net1.network                              ora.network.type          ONLINE             ONLINE             rhel72
ora.ons                                       ora.ons.type              ONLINE             ONLINE             rhel71
ora.ons                                       ora.ons.type              ONLINE             ONLINE             rhel72
ora.proxy_advm                                ora.proxy_advm.type       OFFLINE            OFFLINE
ora.proxy_advm                                ora.proxy_advm.type       OFFLINE            OFFLINE
ora.qosmserver                                ora.qosmserver.type       ONLINE             ONLINE             rhel71
ora.rhel71.vip                                ora.cluster_vip_net1.type ONLINE             ONLINE             rhel71
ora.rhel72.vip                                ora.cluster_vip_net1.type ONLINE             ONLINE             rhel72
ora.rhpserver                                 ora.rhpserver.type        OFFLINE            OFFLINE
ora.scan1.vip                                 ora.scan_vip.type         ONLINE             ONLINE             rhel72
ora.scan2.vip                                 ora.scan_vip.type         ONLINE             ONLINE             rhel71
ora.scan3.vip                                 ora.scan_vip.type         ONLINE             ONLINE             rhel71

As the last step in GI installation, create other disk groups that would be used for the database. In this case two disk groups called +data and +fra are created.
asmca -silent -createDiskGroup -diskGroupName data -disk /dev/oracleasm/data1 -redundancy EXTERNAL -au_size 4 -compatible.asm 19.0.0.0.0 -compatible.rdbms 19.0.0.0.0
asmca -silent -createDiskGroup -diskGroupName fra -disk /dev/oracleasm/fra1 -redundancy EXTERNAL -au_size 4 -compatible.asm 19.0.0.0.0 -compatible.rdbms 19.0.0.0.0


The next phase is to install the database software. Check the datadbase software pre-reqs with cluvfy.
cluvfy stage -pre dbinst -allnodes -r 19

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 1001 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying User Mask ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying ASM storage privileges for the user: oracle ...
  Verifying Group Membership: asmdba ...PASSED
Verifying ASM storage privileges for the user: oracle ...PASSED
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying ACFS device special file ...PASSED
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying Maximum locked memory check ...FAILED (PRVE-0059)

Pre-check for database installation was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbinst".

Verifying Maximum locked memory check ...FAILED
rhel72: PRVE-0059 : no default entry or entry specific to user "oracle" was
        found in the configuration file "/etc/security/limits.conf" when
        checking the maximum locked memory "HARD" limit on node
"rhel72.domain.net"

rhel71: PRVE-0059 : no default entry or entry specific to user "oracle" was
        found in the configuration file "/etc/security/limits.conf" when
        checking the maximum locked memory "HARD" limit on node
"rhel71.domain.net"

The memlock related failure is ignorable at this stage. This could be set either before database creation or after database is created (in later case it would requrie a restart for SGA to use the large pages).
Similar to GI home, Oracle home also provides several resposne files that could be edited and used for installing the RAC DB software. Copy one of the following to temporary location and modify to reflect the cluster being setup.
$ORACLE_HOME/install/response/db_install.rsp
or
$ORACLE_HOME/inventory/response/db_install.rsp
The following text shows the response file used in this setup. Resposne file content is edited such that it will do a software only installation.
####################################################################
## Copyright(c) Oracle Corporation 1998,2019. All rights reserved.##
##                                                                ##
## Specify values for the variables listed below to customize     ##
## your installation.                                             ##
##                                                                ##
## Each variable is associated with a comment. The comment        ##
## can help to populate the variables with the appropriate        ##
## values.                                                        ##
##                                                                ##
## IMPORTANT NOTE: This file contains plain text passwords and    ##
## should be secured to have read permission only by oracle user  ##
## or db administrator who owns this installation.                ##
##                                                                ##
####################################################################


#------------------------------------------------------------------------------
# Do not change the following system generated value. 
#------------------------------------------------------------------------------
oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v19.0.0

#-------------------------------------------------------------------------------
# Specify the installation option.
# It can be one of the following:
#   - INSTALL_DB_SWONLY
#   - INSTALL_DB_AND_CONFIG
#-------------------------------------------------------------------------------
oracle.install.option=INSTALL_DB_SWONLY

#-------------------------------------------------------------------------------
# Specify the Unix group to be set for the inventory directory.  
#-------------------------------------------------------------------------------
UNIX_GROUP_NAME=oinstall

#-------------------------------------------------------------------------------
# Specify the location which holds the inventory files.
# This is an optional parameter if installing on
# Windows based Operating System.
#-------------------------------------------------------------------------------
INVENTORY_LOCATION=/opt/app/oraInventory

#-------------------------------------------------------------------------------
# Specify the complete path of the Oracle Base. 
#-------------------------------------------------------------------------------
ORACLE_BASE=/opt/app/oracle

#-------------------------------------------------------------------------------
# Specify the installation edition of the component.                     
#                                                             
# The value should contain only one of these choices.  

#   - EE     : Enterprise Edition 

#   - SE2     : Standard Edition 2


#-------------------------------------------------------------------------------

oracle.install.db.InstallEdition=EE
###############################################################################
#                                                                             #
# PRIVILEGED OPERATING SYSTEM GROUPS                                          #
# ------------------------------------------                                  #
# Provide values for the OS groups to which SYSDBA and SYSOPER privileges     #
# needs to be granted. If the install is being performed as a member of the   #
# group "dba", then that will be used unless specified otherwise below.       #
#                                                                             #
# The value to be specified for OSDBA and OSOPER group is only for UNIX based #
# Operating System.                                                           #
#                                                                             #
###############################################################################

#------------------------------------------------------------------------------
# The OSDBA_GROUP is the OS group which is to be granted SYSDBA privileges.
#-------------------------------------------------------------------------------
oracle.install.db.OSDBA_GROUP=dba

#------------------------------------------------------------------------------
# The OSOPER_GROUP is the OS group which is to be granted SYSOPER privileges.
# The value to be specified for OSOPER group is optional.
#------------------------------------------------------------------------------
oracle.install.db.OSOPER_GROUP=oper

#------------------------------------------------------------------------------
# The OSBACKUPDBA_GROUP is the OS group which is to be granted SYSBACKUP privileges.
#------------------------------------------------------------------------------
oracle.install.db.OSBACKUPDBA_GROUP=backupdba

#------------------------------------------------------------------------------
# The OSDGDBA_GROUP is the OS group which is to be granted SYSDG privileges.
#------------------------------------------------------------------------------
oracle.install.db.OSDGDBA_GROUP=dgdba

#------------------------------------------------------------------------------
# The OSKMDBA_GROUP is the OS group which is to be granted SYSKM privileges.
#------------------------------------------------------------------------------
oracle.install.db.OSKMDBA_GROUP=kmdba

#------------------------------------------------------------------------------
# The OSRACDBA_GROUP is the OS group which is to be granted SYSRAC privileges.
#------------------------------------------------------------------------------
oracle.install.db.OSRACDBA_GROUP=racdba
################################################################################
#                                                                              #
#                      Root script execution configuration                     #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------------------------------
# Specify the root script execution mode.
#
#   - true  : To execute the root script automatically by using the appropriate configuration methods.
#   - false : To execute the root script manually.
#
# If this option is selected, password should be specified on the console.
#-------------------------------------------------------------------------------------------------------
oracle.install.db.rootconfig.executeRootScript=false

#--------------------------------------------------------------------------------------
# Specify the configuration method to be used for automatic root script execution.
#
# Following are the possible choices:
#   - ROOT
#   - SUDO
#--------------------------------------------------------------------------------------
oracle.install.db.rootconfig.configMethod=
#--------------------------------------------------------------------------------------
# Specify the absolute path of the sudo program.
#
# Applicable only when SUDO configuration method was chosen.
#--------------------------------------------------------------------------------------
oracle.install.db.rootconfig.sudoPath=

#--------------------------------------------------------------------------------------
# Specify the name of the user who is in the sudoers list. 
# Applicable only when SUDO configuration method was chosen.
# Note:For Single Instance database installations,the sudo user name must be the username of the user installing the database.
#--------------------------------------------------------------------------------------
oracle.install.db.rootconfig.sudoUserName=

###############################################################################
#                                                                             #
#                               Grid Options                                  #
#                                                                             #
###############################################################################

#------------------------------------------------------------------------------
# Value is required only if the specified install option is INSTALL_DB_SWONLY
# 
# Specify the cluster node names selected during the installation.
# 
# Example : oracle.install.db.CLUSTER_NODES=node1,node2
#------------------------------------------------------------------------------
oracle.install.db.CLUSTER_NODES=rhel71,rhel72

###############################################################################
#                                                                             #
#                        Database Configuration Options                       #
#                                                                             #
###############################################################################

#-------------------------------------------------------------------------------
# Specify the type of database to create.
# It can be one of the following:
#   - GENERAL_PURPOSE                       
#   - DATA_WAREHOUSE 
# GENERAL_PURPOSE: A starter database designed for general purpose use or transaction-heavy applications.
# DATA_WAREHOUSE : A starter database optimized for data warehousing applications.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.type=GENERAL_PURPOSE

#-------------------------------------------------------------------------------
# Specify the Starter Database Global Database Name. 
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.globalDBName=

#-------------------------------------------------------------------------------
# Specify the Starter Database SID.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.SID=

#-------------------------------------------------------------------------------
# Specify whether the database should be configured as a Container database.
# The value can be either "true" or "false". If left blank it will be assumed
# to be "false".
#-------------------------------------------------------------------------------
oracle.install.db.ConfigureAsContainerDB=false

#-------------------------------------------------------------------------------
# Specify the  Pluggable Database name for the pluggable database in Container Database.
#-------------------------------------------------------------------------------
oracle.install.db.config.PDBName=

#-------------------------------------------------------------------------------
# Specify the Starter Database character set.
#                                               
#  One of the following
#  AL32UTF8, WE8ISO8859P15, WE8MSWIN1252, EE8ISO8859P2,
#  EE8MSWIN1250, NE8ISO8859P10, NEE8ISO8859P4, BLT8MSWIN1257,
#  BLT8ISO8859P13, CL8ISO8859P5, CL8MSWIN1251, AR8ISO8859P6,
#  AR8MSWIN1256, EL8ISO8859P7, EL8MSWIN1253, IW8ISO8859P8,
#  IW8MSWIN1255, JA16EUC, JA16EUCTILDE, JA16SJIS, JA16SJISTILDE,
#  KO16MSWIN949, ZHS16GBK, TH8TISASCII, ZHT32EUC, ZHT16MSWIN950,
#  ZHT16HKSCS, WE8ISO8859P9, TR8MSWIN1254, VN8MSWIN1258
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.characterSet=

#------------------------------------------------------------------------------
# This variable should be set to true if Automatic Memory Management 
# in Database is desired.
# If Automatic Memory Management is not desired, and memory allocation
# is to be done manually, then set it to false.
#------------------------------------------------------------------------------
oracle.install.db.config.starterdb.memoryOption=false

#-------------------------------------------------------------------------------
# Specify the total memory allocation for the database. Value(in MB) should be
# at least 256 MB, and should not exceed the total physical memory available 
# on the system.
# Example: oracle.install.db.config.starterdb.memoryLimit=512
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.memoryLimit=

#-------------------------------------------------------------------------------
# This variable controls whether to load Example Schemas onto
# the starter database or not.
# The value can be either "true" or "false". If left blank it will be assumed
# to be "false".
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.installExampleSchemas=false

###############################################################################
#                                                                             #
# Passwords can be supplied for the following four schemas in the       #
# starter database:                  #
#   SYS                                                                       #
#   SYSTEM                                                                    #
#   DBSNMP (used by Enterprise Manager)                                       #
#                                                                             #
# Same password can be used for all accounts (not recommended)         #
# or different passwords for each account can be provided (recommended)       #
#                                                                             #
###############################################################################

#------------------------------------------------------------------------------
# This variable holds the password that is to be used for all schemas in the
# starter database.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.ALL=

#-------------------------------------------------------------------------------
# Specify the SYS password for the starter database.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.SYS=

#-------------------------------------------------------------------------------
# Specify the SYSTEM password for the starter database.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.SYSTEM=

#-------------------------------------------------------------------------------
# Specify the DBSNMP password for the starter database.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.DBSNMP=

#-------------------------------------------------------------------------------
# Specify the PDBADMIN password required for creation of Pluggable Database in the Container Database.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.PDBADMIN=

#-------------------------------------------------------------------------------
# Specify the management option to use for managing the database.
# Options are:
# 1. CLOUD_CONTROL - If you want to manage your database with Enterprise Manager Cloud Control along with Database Express.
# 2. DEFAULT   -If you want to manage your database using the default Database Express option.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.managementOption=DEFAULT

#-------------------------------------------------------------------------------
# Specify the OMS host to connect to Cloud Control.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.omsHost=

#-------------------------------------------------------------------------------
# Specify the OMS port to connect to Cloud Control.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.omsPort=0

#-------------------------------------------------------------------------------
# Specify the EM Admin user name to use to connect to Cloud Control.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.emAdminUser=

#-------------------------------------------------------------------------------
# Specify the EM Admin password to use to connect to Cloud Control.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.emAdminPassword=

###############################################################################
#                                                                             #
# SPECIFY RECOVERY OPTIONS                                                #
# ------------------------------------                                #
# Recovery options for the database can be mentioned using the entries below  #
#                                                                             #
###############################################################################

#------------------------------------------------------------------------------
# This variable is to be set to false if database recovery is not required. Else 
# this can be set to true.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.enableRecovery=false

#-------------------------------------------------------------------------------
# Specify the type of storage to use for the database.
# It can be one of the following:
#   - FILE_SYSTEM_STORAGE
#   - ASM_STORAGE
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.storageType=

#-------------------------------------------------------------------------------
# Specify the database file location which is a directory for datafiles, control
# files, redo logs.         
#
# Applicable only when oracle.install.db.config.starterdb.storage=FILE_SYSTEM_STORAGE 
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.fileSystemStorage.dataLocation=

#-------------------------------------------------------------------------------
# Specify the recovery location.
#
# Applicable only when oracle.install.db.config.starterdb.storage=FILE_SYSTEM_STORAGE 
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.fileSystemStorage.recoveryLocation=

#-------------------------------------------------------------------------------
# Specify the existing ASM disk groups to be used for storage.
#
# Applicable only when oracle.install.db.config.starterdb.storageType=ASM_STORAGE
#-------------------------------------------------------------------------------
oracle.install.db.config.asm.diskGroup=

#-------------------------------------------------------------------------------
# Specify the password for ASMSNMP user of the ASM instance.                 
#
# Applicable only when oracle.install.db.config.starterdb.storage=ASM_STORAGE 
#-------------------------------------------------------------------------------
oracle.install.db.config.asm.ASMSNMPPassword=

Execute the installer specifying the response file. The ignorePrereqFailure is used here due to pre-req failures being ignored as this is a test system.
$ORACLE_HOME/runInstaller -silent -responseFile /media/db_install.rsp -ignorePrereqFailure
Launching Oracle Database Setup Wizard...

[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
   CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /opt/app/oraInventory/logs/InstallActions2019-06-13_10-22-10AM/installActions2019-06-13_10-22-10AM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /opt/app/oraInventory/logs/InstallActions2019-06-13_10-22-10AM/installActions2019-06-13_10-22-10AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /opt/app/oracle/product/19.x.0/dbhome_1/install/response/db_2019-06-13_10-22-10AM.rsp

You can find the log of this install session at:
 /opt/app/oraInventory/logs/InstallActions2019-06-13_10-22-10AM/installActions2019-06-13_10-22-10AM.log

As a root user, execute the following script(s):
        1. /opt/app/oracle/product/19.x.0/dbhome_1/root.sh

Execute /opt/app/oracle/product/19.x.0/dbhome_1/root.sh on the following nodes:
[rhel71, rhel72]


Successfully Setup Software with warning(s).

The last phase is the creation of the RAC database. In a role separated setup change the permission following folders.
cd $ORACLE_BASE
chmod 770 audit admin

cd $ORACLE_BASE/cfgtoollogs
chmod 770 dbca sqlpatch
Check database configuraiton pre-reqs with cluvfy.
cluvfy stage -pre dbcfg -allnodes -d $ORACLE_HOME

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 1001 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dgdba ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: racdba ...PASSED
Verifying Group Existence: backupdba ...PASSED
Verifying Group Existence: oper ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: backupdba ...PASSED
Verifying Group Membership: racdba ...PASSED
Verifying Group Membership: dgdba ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Group Membership: oper ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Current Group ID ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying Maximum locked memory check ...FAILED (PRVE-0059)
Verifying File system mount options for path ORACLE_HOME ...FAILED (PRVE-0007)
Verifying /dev/shm mounted as temporary file system ...PASSED

Pre-check for database configuration was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbcfg".

Verifying Maximum locked memory check ...FAILED
rhel72: PRVE-0059 : no default entry or entry specific to user "oracle" was
        found in the configuration file "/etc/security/limits.conf" when
        checking the maximum locked memory "HARD" limit on node
"rhel72.domain.net"

rhel71: PRVE-0059 : no default entry or entry specific to user "oracle" was
        found in the configuration file "/etc/security/limits.conf" when
        checking the maximum locked memory "HARD" limit on node
"rhel71.domain.net"

Verifying File system mount options for path ORACLE_HOME ...FAILED
rhel72: PRVE-0007 : Could not find executable
"/tmp/CVU_19.0.0.0.0_oracle/checkFSMountOptions.sh"

rhel71: PRVE-0007 : Could not find executable
"/tmp/CVU_19.0.0.0.0_oracle/checkFSMountOptions.sh"

The pre-reqs failurs are ignored. The databae creation could be done using dbca in silent mode and with the use of a custom template. One way to create a custom template is by modifying the $ORACLE_HOME/assistants/dbca/templates/New_Database.dbt file. Following shows the content of the custom template created by modifying the aforementioned file.
 cat testdb.dbt
<DatabaseTemplate name="Test Database" description="" version="19.0.0.0.0">
<CommonAttributes>
<option name="OMS" value="false" includeInPDBs="false"/>
<option name="JSERVER" value="false" includeInPDBs="false"/>
<option name="SPATIAL" value="false" includeInPDBs="false"/>
<option name="IMEDIA" value="false" includeInPDBs="false"/>
<option name="ORACLE_TEXT" value="true" includeInPDBs="true">
<tablespace id="SYSAUX"/>
</option>
<option name="CWMLITE" value="false" includeInPDBs="false">
<tablespace id="SYSAUX"/>
</option>
<option name="SAMPLE_SCHEMA" value="false" includeInPDBs="false"/>
<option name="APEX" value="false" includeInPDBs="false"/>
<option name="DV" value="false" includeInPDBs="false"/>
</CommonAttributes>
<Variables/>
<CustomScripts Execute="false"/>
<InitParamAttributes>
<InitParams>
<initParam name="db_block_size" value="8" unit="KB"/>
<initParam name="open_cursors" value="300"/>
<initParam name="undo_tablespace" value="UNDOTBS1"/>
<initParam name="control_files" value="("{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/control01.ctl", "{ORACLE_BASE}/fast_recovery_area/{DB_UNIQUE_NAME}/control02.ctl")"/>

<initParam name="compatible" value="19.0.0"/>
<initParam name="audit_file_dest" value="{ORACLE_BASE}/admin/{DB_UNIQUE_NAME}/adump"/>
<initParam name="audit_trail" value="OS"/>
<initParam name="diagnostic_dest" value="{ORACLE_BASE}"/>
<initParam name="remote_login_passwordfile" value="EXCLUSIVE"/>
<initParam name="dispatchers" value="(PROTOCOL=TCP) (SERVICE={SID}XDB)"/>
</InitParams>
<MiscParams>
<percentageMemTOSGA>40</percentageMemTOSGA>
<archiveLogMode>false</archiveLogMode>
<initParamFileName>{ORACLE_BASE}/admin/{DB_UNIQUE_NAME}/pfile/init.ora</initParamFileName>
</MiscParams>
<SPfile useSPFile="true">{ORACLE_HOME}/dbs/spfile{SID}.ora</SPfile>
</InitParamAttributes>
<StorageAttributes>
<ControlfileAttributes id="Controlfile">
<maxDatafiles>100</maxDatafiles>
<maxLogfiles>16</maxLogfiles>
<maxLogMembers>3</maxLogMembers>
<maxLogHistory>1</maxLogHistory>
<maxInstances>8</maxInstances>
<image name="control01.ctl" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
<image name="control02.ctl" filepath="{ORACLE_BASE}/fast_recovery_area/{DB_UNIQUE_NAME}/"/>

</ControlfileAttributes>
<DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/sysaux01.dbf">
<tablespace>SYSAUX</tablespace>
<temporary>false</temporary>
<online>true</online>
<status>0</status>
<size unit="MB">550</size>
<reuse>true</reuse>
<autoExtend>true</autoExtend>
<increment unit="KB">10240</increment>
<maxSize unit="MB">-1</maxSize>
</DatafileAttributes>
<DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/users01.dbf">
<tablespace>USERS</tablespace>
<temporary>false</temporary>
<online>true</online>
<status>0</status>
<size unit="MB">5</size>
<reuse>true</reuse>
<autoExtend>true</autoExtend>
<increment unit="KB">1280</increment>
<maxSize unit="MB">-1</maxSize>
</DatafileAttributes>
<DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/system01.dbf">
<tablespace>SYSTEM</tablespace>
<temporary>false</temporary>
<online>true</online>
<status>0</status>
<size unit="MB">700</size>
<reuse>true</reuse>
<autoExtend>true</autoExtend>
<increment unit="KB">10240</increment>
<maxSize unit="MB">-1</maxSize>
</DatafileAttributes>
<DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/temp01.dbf">
<tablespace>TEMP</tablespace>
<temporary>false</temporary>
<online>true</online>
<status>0</status>
<size unit="MB">20</size>
<reuse>true</reuse>
<autoExtend>true</autoExtend>
<increment unit="KB">640</increment>
<maxSize unit="MB">-1</maxSize>
</DatafileAttributes>
<DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/undotbs01.dbf">
<tablespace>UNDOTBS1</tablespace>
<temporary>false</temporary>
<online>true</online>
<status>0</status>
<size unit="MB">200</size>
<reuse>true</reuse>
<autoExtend>true</autoExtend>
<increment unit="KB">5120</increment>
<maxSize unit="MB">-1</maxSize>
</DatafileAttributes>
<TablespaceAttributes id="SYSAUX">
<online>true</online>
<offlineMode>1</offlineMode>
<readOnly>false</readOnly>
<temporary>false</temporary>
<defaultTemp>false</defaultTemp>
<undo>false</undo>
<local>true</local>
<blockSize>-1</blockSize>
<allocation>1</allocation>
<uniAllocSize unit="KB">-1</uniAllocSize>
<initSize unit="KB">64</initSize>
<increment unit="KB">64</increment>
<incrementPercent>50</incrementPercent>
<minExtends>1</minExtends>
<maxExtends>4096</maxExtends>
<minExtendsSize unit="KB">64</minExtendsSize>
<logging>true</logging>
<recoverable>false</recoverable>
<maxFreeSpace>0</maxFreeSpace>
<bigfile>false</bigfile>
<datafilesList>
<TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/sysaux01.dbf"/>
</datafilesList>
</TablespaceAttributes>
<TablespaceAttributes id="USERS">
<online>true</online>
<offlineMode>1</offlineMode>
<readOnly>false</readOnly>
<temporary>false</temporary>
<defaultTemp>false</defaultTemp>
<undo>false</undo>
<local>true</local>
<blockSize>-1</blockSize>
<allocation>1</allocation>
<uniAllocSize unit="KB">-1</uniAllocSize>
<initSize unit="KB">128</initSize>
<increment unit="KB">128</increment>
<incrementPercent>0</incrementPercent>
<minExtends>1</minExtends>
<maxExtends>4096</maxExtends>
<minExtendsSize unit="KB">128</minExtendsSize>
<logging>true</logging>
<recoverable>false</recoverable>
<maxFreeSpace>0</maxFreeSpace>
<bigfile>false</bigfile>
<datafilesList>
<TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/users01.dbf"/>
</datafilesList>
</TablespaceAttributes>
<TablespaceAttributes id="SYSTEM">
<online>true</online>
<offlineMode>1</offlineMode>
<readOnly>false</readOnly>
<temporary>false</temporary>
<defaultTemp>false</defaultTemp>
<undo>false</undo>
<local>true</local>
<blockSize>-1</blockSize>
<allocation>3</allocation>
<uniAllocSize unit="KB">-1</uniAllocSize>
<initSize unit="KB">64</initSize>
<increment unit="KB">64</increment>
<incrementPercent>50</incrementPercent>
<minExtends>1</minExtends>
<maxExtends>-1</maxExtends>
<minExtendsSize unit="KB">64</minExtendsSize>
<logging>true</logging>
<recoverable>false</recoverable>
<maxFreeSpace>0</maxFreeSpace>
<bigfile>false</bigfile>
<datafilesList>
<TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/system01.dbf"/>
</datafilesList>
</TablespaceAttributes>
<TablespaceAttributes id="TEMP">
<online>true</online>
<offlineMode>1</offlineMode>
<readOnly>false</readOnly>
<temporary>true</temporary>
<defaultTemp>true</defaultTemp>
<undo>false</undo>
<local>true</local>
<blockSize>-1</blockSize>
<allocation>1</allocation>
<uniAllocSize unit="KB">-1</uniAllocSize>
<initSize unit="KB">64</initSize>
<increment unit="KB">64</increment>
<incrementPercent>0</incrementPercent>
<minExtends>1</minExtends>
<maxExtends>0</maxExtends>
<minExtendsSize unit="KB">64</minExtendsSize>
<logging>true</logging>
<recoverable>false</recoverable>
<maxFreeSpace>0</maxFreeSpace>
<bigfile>false</bigfile>
<datafilesList>
<TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/temp01.dbf"/>
</datafilesList>
</TablespaceAttributes>
<TablespaceAttributes id="UNDOTBS1">
<online>true</online>
<offlineMode>1</offlineMode>
<readOnly>false</readOnly>
<temporary>false</temporary>
<defaultTemp>false</defaultTemp>
<undo>true</undo>
<local>true</local>
<blockSize>-1</blockSize>
<allocation>1</allocation>
<uniAllocSize unit="KB">-1</uniAllocSize>
<initSize unit="KB">512</initSize>
<increment unit="KB">512</increment>
<incrementPercent>50</incrementPercent>
<minExtends>8</minExtends>
<maxExtends>4096</maxExtends>
<minExtendsSize unit="KB">512</minExtendsSize>
<logging>true</logging>
<recoverable>false</recoverable>
<maxFreeSpace>0</maxFreeSpace>
<bigfile>false</bigfile>
<datafilesList>
<TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/undotbs01.dbf"/>
</datafilesList>
</TablespaceAttributes>
<RedoLogGroupAttributes id="1">
<reuse>false</reuse>
<fileSize unit="KB">204800</fileSize>
<Thread>1</Thread>
<member ordinal="0" memberName="redo01.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
</RedoLogGroupAttributes>
<RedoLogGroupAttributes id="2">
<reuse>false</reuse>
<fileSize unit="KB">204800</fileSize>
<Thread>1</Thread>
<member ordinal="0" memberName="redo02.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
</RedoLogGroupAttributes>
<RedoLogGroupAttributes id="3">
<reuse>false</reuse>
<fileSize unit="KB">204800</fileSize>
<Thread>1</Thread>
<member ordinal="0" memberName="redo03.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
</RedoLogGroupAttributes>
</StorageAttributes>
</DatabaseTemplate>

Run the dbca specifying the template and other parameters as command options.
dbca -createDatabase -gdbName racext -templateName $ORACLE_HOME/assistants/dbca/templates/testdb.dbt -characterSet AL32UTF8 -emConfiguration DBEXPRESS 
-storageType ASM -asmsnmpPassword testASM1234 
-diskGroupName DATA -recoveryGroupName FRA 
-nodelist rhel71,rhel72 -sysPassword racextDB1234 
-systemPassword racextDB1234 
-createAsContainerDatabase false 
-memoryMgmtType AUTO_SGA 
-enableArchive false -useOMF true 
-adminManaged -nationalCharacterSet AL16UTF16 
-databaseConfigType RAC -silent
[WARNING] [DBT-09102] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details.
   ACTION: Find the appropriate configuration from the log file or from the installation guide to meet the prerequisites and fix this manually.
Prepare for db operation
8% complete
Creating and starting Oracle instance
10% complete
11% complete
15% complete
Creating database files
16% complete
17% complete
23% complete
Creating data dictionary views
25% complete
30% complete
31% complete
32% complete
37% complete
39% complete
42% complete
46% complete
Oracle Text
47% complete
48% complete
52% complete
54% complete
Creating cluster database views
55% complete
69% complete
Completing Database Creation
73% complete
76% complete
77% complete
Executing Post Configuration Actions
100% complete
Database creation complete. For details check the logfiles at:
 /opt/app/oracle/cfgtoollogs/dbca/racext.
Database Information:
Global Database Name:racext
System Identifier(SID) Prefix:racext
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/racext/racext0.log" for further details.

Check the databae status at the end
srvctl status database -db racext
Instance racext1 is running on node rhel71
Instance racext2 is running on node rhel72
This concludes the installing 19c RAC.
Related Posts
Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Clusterware
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Following screenshots shows the same done using the OUI. OUI also gives the option of saving the setup inputs in a repsonse which could be later used in silent instalations.
In step the private interface use defautls only for "private". In previous versions this would default to "ASM & private". This is due to the fact now that OCR and vote disks could be located in shared file system and GIMR is optional. Therefore it is possible to have a RAC without ASM (if data and recovery area is also located in a NFS).
If ASM is used for OCR and GIMR change the use to "ASM & private".

Database software setup (not all steps are shown)

Database Setup (not all steps shown)
↧
Viewing all 315 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>