Unlike other cloud vendors, in OCI it is possible to create a 2 node RAC without any third party software. The minimum shape needed for creating a 2 node RAC for virtual machine type is VM.Standard(x).2. This will automatically enables the node count spinner allowing it to be changed to 2 nodes (currently the max number of nodes). Additionally cluster name could be specified. Rest of the steps are similar to creating a single instance DB. For this post 18.3 RAC DB was created in private subnet. The Database specifications are shown below. The database details are shown below. An app server on a public subnet on the same VCN is able to resolve the SCAN without any other user configurations.
The RAC configuration is role separated. Following roles exists for oracle and grid user. There are no separate groups for backupdba, kmdba, dgdba or racdba.
The cluster does not use GNS. Below is the full list of resources at the end of the cluster creation.
The ntpd file is missing the -x option which prevents setting the clock backwards.
Similar to single instance DB, RAC DB can also be associated with a data guard configuration. The London region where the initial RAC DB was created had an issue creating a data guard association. Oracle support also confirmed that there are intermittent issues when creating a data guard for RAC DB. However, it was possible to create a RAC DB with data guard association in the Frankfurt region. Following screenshot is of this new RAC DB (as such some names are different). The DG protection mode is maximum performance.
Related Posts
Data Guard on OCI PaaS DB
nslookup rac-scan.londbsubnet.lonvcn.oraclevcn.comThe hostnames will be suffixed with node number when the cluster is created. The subnet CIDR block is treated as the public network and second 192.168.* network is created for private interconnect.
Server: 169.254.169.254
Address: 169.254.169.254#53
Non-authoritative answer:
Name: rac-scan.londbsubnet.lonvcn.oraclevcn.com
Address: 172.21.0.7
Name: rac-scan.londbsubnet.lonvcn.oraclevcn.com
Address: 172.21.0.8
Name: rac-scan.londbsubnet.lonvcn.oraclevcn.com
Address: 172.21.0.6
oifcfg getifThe /etc/oratab files on all nodes are empty of any ASM or DB instance informations. These need to be added manually if needed.
eth0 172.21.0.0 global public
eth1 192.168.16.0 global cluster_interconnect,asm
The RAC configuration is role separated. Following roles exists for oracle and grid user. There are no separate groups for backupdba, kmdba, dgdba or racdba.
[opc@rac1 ~]$ id oracleThere's no user equivalence between nodes for oracle or grid user.
uid=101(oracle) gid=1001(oinstall) groups=1001(oinstall),1006(asmdba),1003(dba),1002(dbaoper)
[opc@rac1 ~]$ id grid
uid=102(grid) gid=1001(oinstall) groups=1001(oinstall),1006(asmdba),1005(asmoper),1004(asmadmin),1002(dbaoper)
cluvfy stage -post crsinst -n allOracle documentation mentions (this documentation may not be relevant any more) that user equivalence exists for opc user. But that too didn't exists.
Verifying User Equivalence ...FAILED (PRVG-2019, PRKC-1191)
PRVF-4009 : User equivalence is not set for nodes: rac2
[oracle@rac1 ~]$ ssh rac2
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[oracle@rac2 ~]$ ssh rac1
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[grid@rac1 ~]$ ssh rac2
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[grid@rac2 ~]$ ssh rac1
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[opc@rac1 ~]$ ssh rac2
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[opc@rac2 ~]$ ssh rac1
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
The cluster does not use GNS. Below is the full list of resources at the end of the cluster creation.
Resource Name Type Target State HostSimilar to single instance DB RAC DB too consists of two ASM disk groups. OCR and vote is stored in the DATA disk group. The backup location for OCR is also the DATA disk group.
------------- ------ ------- -------- ----------
ora.ASMNET1LSNR_ASM.lsnr ora.asm_listener.type ONLINE ONLINE rac1
ora.ASMNET1LSNR_ASM.lsnr ora.asm_listener.type ONLINE ONLINE rac2
ora.DATA.COMMONSTORE.advm ora.volume.type ONLINE ONLINE rac1
ora.DATA.COMMONSTORE.advm ora.volume.type ONLINE ONLINE rac2
ora.DATA.dg ora.diskgroup.type ONLINE ONLINE rac1
ora.DATA.dg ora.diskgroup.type ONLINE ONLINE rac2
ora.LISTENER.lsnr ora.listener.type ONLINE ONLINE rac1
ora.LISTENER.lsnr ora.listener.type ONLINE ONLINE rac2
ora.LISTENER_SCAN1.lsnr ora.scan_listener.type ONLINE ONLINE rac1
ora.LISTENER_SCAN2.lsnr ora.scan_listener.type ONLINE ONLINE rac2
ora.LISTENER_SCAN3.lsnr ora.scan_listener.type ONLINE ONLINE rac2
ora.RECO.dg ora.diskgroup.type ONLINE ONLINE rac1
ora.RECO.dg ora.diskgroup.type ONLINE ONLINE rac2
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac2
ora.cdbrac_lhr1fq.db ora.database.type ONLINE ONLINE rac1
ora.cdbrac_lhr1fq.db ora.database.type ONLINE ONLINE rac2
ora.cvu ora.cvu.type ONLINE ONLINE rac2
ora.data.commonstore.acfs ora.acfs.type ONLINE ONLINE rac1
ora.data.commonstore.acfs ora.acfs.type ONLINE ONLINE rac2
ora.net1.network ora.network.type ONLINE ONLINE rac1
ora.net1.network ora.network.type ONLINE ONLINE rac2
ora.ons ora.ons.type ONLINE ONLINE rac1
ora.ons ora.ons.type ONLINE ONLINE rac2
ora.proxy_advm ora.proxy_advm.type ONLINE ONLINE rac1
ora.proxy_advm ora.proxy_advm.type ONLINE ONLINE rac2
ora.qosmserver ora.qosmserver.type OFFLINE OFFLINE
ora.rac1.vip ora.cluster_vip_net1.type ONLINE ONLINE rac1
ora.rac2.vip ora.cluster_vip_net1.type ONLINE ONLINE rac2
ora.scan1.vip ora.scan_vip.type ONLINE ONLINE rac1
ora.scan2.vip ora.scan_vip.type ONLINE ONLINE rac2
ora.scan3.vip ora.scan_vip.type ONLINE ONLINE rac2
SQL> select name,total_mb,free_Mb from v$asm_diskgroup;ASM disks are SCSI attached and permissions are set with udev rules.
NAME TOTAL_MB FREE_MB
------------------------------ ---------- ----------
DATA 262144 253408
RECO 262144 257380
cat /etc/oracle/ocr.loc
#Device/file +DATA getting replaced by device +DATA/dbSyslw4gbgsa/OCRFILE/registry.255.1002113311
ocrconfig_loc=+DATA/dbSyslw4gbgsa/OCRFILE/registry.255.1002113311
[grid@rac1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 1e4fd3b7b9364fdfbf4dbf41ffde36d9 (/dev/DATADISK3) [DATA]
ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+DATA].
SQL> select name,path from v$asm_disk;Manual intervention is needed to create the time synchronization between the two nodes. Although NTP service is running, it is not synchronized with an external time source.
NAME PATH
------------------------------ ------------------------------
RECODISK4 /dev/RECODISK4
RECODISK3 /dev/RECODISK3
RECODISK2 /dev/RECODISK2
RECODISK1 /dev/RECODISK1
DATA_0003 /dev/DATADISK4
DATA_0000 /dev/DATADISK3
DATA_0001 /dev/DATADISK2
DATA_0002 /dev/DATADISK1
[grid@rac1 admin]$ ls -l /dev/DATA*
lrwxrwxrwx 1 root root 3 Mar 5 14:58 /dev/DATADISK1 -> sdb
lrwxrwxrwx 1 root root 3 Mar 5 14:58 /dev/DATADISK2 -> sdc
lrwxrwxrwx 1 root root 3 Mar 5 14:58 /dev/DATADISK3 -> sdd
lrwxrwxrwx 1 root root 3 Mar 5 14:58 /dev/DATADISK4 -> sde
[grid@rac1 admin]$ ls -l /dev/disk/by-id/ | egrep "sdb|sdc|sdd|sde"
lrwxrwxrwx 1 root root 9 Mar 5 14:59 scsi-36012159faf2d4794be5ab57d8a5b7498 -> ../../sdc
lrwxrwxrwx 1 root root 10 Mar 5 12:32 scsi-36012159faf2d4794be5ab57d8a5b7498-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 Mar 5 14:59 scsi-36022a3f8dffa46b4a57049af3a2dd275 -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 5 12:32 scsi-36022a3f8dffa46b4a57049af3a2dd275-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Mar 5 14:59 scsi-360748b62328c4ff6bb8fe3e9e4f04826 -> ../../sdd
lrwxrwxrwx 1 root root 10 Mar 5 12:32 scsi-360748b62328c4ff6bb8fe3e9e4f04826-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 Mar 5 14:59 scsi-360923cd17c134686829d791e2b4ff82f -> ../../sde
lrwxrwxrwx 1 root root 10 Mar 5 12:32 scsi-360923cd17c134686829d791e2b4ff82f-part1 -> ../../sde1
cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36022a3f8dffa46b4a57049af3a2dd275", SYMLINK+="DATADISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36012159faf2d4794be5ab57d8a5b7498", SYMLINK+="DATADISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360748b62328c4ff6bb8fe3e9e4f04826", SYMLINK+="DATADISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360923cd17c134686829d791e2b4ff82f", SYMLINK+="DATADISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360bc2c20ddc145ceafa98bc1bb1532d3", SYMLINK+="RECODISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606feb49caf045598f66042c2831ad38", SYMLINK+="RECODISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360ad6aca673e4d0a8d2d52aded3ee466", SYMLINK+="RECODISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360f2aa11901e4736b65d54f96f000dcf", SYMLINK+="RECODISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e328f14b14ddf9042d0eca9038593", SYMLINK+="localdisk", OWNER="grid", GROUP="asmadmin", MODE="0660"
Verifying Clock Synchronization ...FAILEDAs the ntp.conf file is present, the ctss runs in observe mode.
Verifying Network Time Protocol (NTP) ...FAILED
Verifying NTP daemon is synchronized with at least one external time source
...FAILED
rac1: PRVG-13602 : NTP daemon is not synchronized with any external time
source on node "rac1".
[grid@rac1 ~]$ crsctl check ctssThis also mentioned in oracle documentation where it is recommended "that you configure NTP on both nodes in a 2-node RAC DB System to synchronize time across the nodes. If you do not configure NTP, then Oracle Clusterware configures and uses the Cluster Time Synchronization Service (CTSS), and the cluster time might be out-of-sync with applications that use NTP for time synchronization".
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.
The ntpd file is missing the -x option which prevents setting the clock backwards.
cat /etc/sysconfig/ntpdOnce the time synchronization is established it could be verified with
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"
cluvfy comp clocksyncThough the cluster was 18c, the oracle home is a read/write home. Not a read only home which was introduced in 18c.
echo $ORACLE_HOMEThe RAC DB created has two log groups per thread.
/u01/app/oracle/product/18.0.0.0/dbhome_1
[oracle@rac2 bin]$ ./orabasehome
/u01/app/oracle/product/18.0.0.0/dbhome_1
SQL> select * from v$log;The RAC DB is a administrator managed DB.
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 3 1073741824 512 1 YES INACTIVE 1262776 05-MAR-19 1264208 05-MAR-19 0
2 1 4 1073741824 512 1 NO CURRENT 1264244 05-MAR-19 1.8447E+19 05-MAR-19 0
3 2 5 1073741824 512 1 NO CURRENT 1262779 05-MAR-19 1.8447E+19 0
4 2 4 1073741824 512 1 YES INACTIVE 1242884 05-MAR-19 1262779 05-MAR-19 0
srvctl config database -db cdbrac_lhr1fq -aBoth DB System patching and DB patching happens in a rolling fashion. Other DB versions where OJVM patch is needed (non-rolling patch) may require total outage. But at least in this case with 18c the patch applying happened rolling fashion.
Database unique name: cdbrac_lhr1fq
Database name: cdbrac
Oracle home: /u01/app/oracle/product/18.0.0.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/CDBRAC_LHR1FQ/PARAMETERFILE/spfile.269.1002115587
Password file: +DATA/CDBRAC_LHR1FQ/PASSWORD/pwdcdbrac_lhr1fq.259.1002114845
Domain: londbsubnet.lonvcn.oraclevcn.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: RECO,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
Database is enabled
Database is individually enabled on nodes:
Database is individually disabled on nodes:
OSDBA group: dba
OSOPER group: dbaoper
Database instances: cdbrac1,cdbrac2
Configured nodes: rac1,rac2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
Similar to single instance DB, RAC DB can also be associated with a data guard configuration. The London region where the initial RAC DB was created had an issue creating a data guard association. Oracle support also confirmed that there are intermittent issues when creating a data guard for RAC DB. However, it was possible to create a RAC DB with data guard association in the Frankfurt region. Following screenshot is of this new RAC DB (as such some names are different). The DG protection mode is maximum performance.
DGMGRL> show configurationPrimary Database
Configuration - raccdb_fra2bg_raccdb_fra359
Protection Mode: MaxPerformance
Members:
raccdb_fra2bg - Primary database
raccdb_fra359 - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS (status updated 19 seconds ago)
DGMGRL> show database raccdb_fra2bgStandby Database has both instance up and running, with one being the apply instance.
Database - raccdb_fra2bg
Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
raccdb1
raccdb2
Database Status:
SUCCESS
DGMGRL> show database raccdb_fra359Switchover could be performed from the console.
Database - raccdb_fra359
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 1.00 KByte/s
Real Time Query: ON
Instance(s):
raccdb1 (apply instance)
raccdb2
Database Status:
SUCCESS
DGMGRL> show configuration
Configuration - raccdb_fra2bg_raccdb_fra359
Protection Mode: MaxPerformance
Members:
raccdb_fra359 - Primary database
raccdb_fra2bg - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS (status updated 58 seconds ago)
Related Posts
Data Guard on OCI PaaS DB