Convert Single Instance DB to RAC DB - Manual Method

There are many ways to convert a single database to RAC. Using database template, rconfig or the manual method. This post shows the steps for manual method of converting a single instance database to RAC (tested for both 11.2.0.4 and 12.1.0.2). The single database is running out of a non-rac oracle home. Backup of this would be restored using a rac enabled oracle home and then converted to a RAC database.
The database is called "asanga" and will retain the same name when converted to RAC.
1. Create a pfile of the single instance database. The pfile entries are shown below

 more pfile.ora
*.audit_file_dest='/opt/app/oracle/admin/asanga/adump'
*.audit_trail='NONE'
*.compatible='11.2.0.4.0'
*.db_block_size=8192
*.db_create_file_dest='+DATA'
*.db_domain=''
*.db_name='ASANGA'
*.db_recovery_file_dest_size=5218762752
*.db_recovery_file_dest='+FLASH'
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=asangaXDB)'
*.open_cursors=300
*.pga_aggregate_target=524288000
*.processes=3000
*.remote_login_passwordfile='EXCLUSIVE'
*.sessions=4524
*.sga_target=2147483648
*.undo_tablespace='UNDOTBS1'

2. Create the adump directories in all nodes of the cluster (in this case it is a two node cluster)

mkdir -p /opt/app/oracle/admin/asanga/adump

3. Copy the pfile created earlier to one of the RAC nodes and restore the datababase.

SQL> startup nomount pfile='/home/oracle/backup/pfile.ora';

SQL> create spfile from pfile='/home/oracle/backup/pfile.ora';
SQL> startup force nomount;

RMAN> restore controlfile from '/home/oracle/backup/ctlbkp04qfl69p_1_1.ctl';

Starting restore at 28-AUG-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=2275 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
output file name=+DATA/asanga/controlfile/current.270.888927491
output file name=+FLASH/asanga/controlfile/current.447.888927493
Finished restore at 28-AUG-15

RMAN> alter database mount;

RMAN> catalog start with '/home/oracle/backup/';

RMAN> run {
2> restore database;
3> recover database;
4> }

Starting restore at 28-AUG-15
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to +DATA/asanga/datafile/system.309.888838053
channel ORA_DISK_1: restoring datafile 00002 to +DATA/asanga/datafile/sysaux.308.888838053
channel ORA_DISK_1: restoring datafile 00003 to +DATA/asanga/datafile/undotbs1.307.888838053
channel ORA_DISK_1: restoring datafile 00004 to +DATA/asanga/datafile/users.310.888838053
channel ORA_DISK_1: reading from backup piece /home/oracle/backup/full_bkp_ASANGA_20150827_02qfl697_1_1
channel ORA_DISK_1: piece handle=/home/oracle/backup/full_bkp_ASANGA_20150827_02qfl697_1_1 tag=FULL_BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:25
Finished restore at 28-AUG-15

Starting recover at 28-AUG-15
using channel ORA_DISK_1

starting media recovery

channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2
channel ORA_DISK_1: reading from backup piece /home/oracle/backup/full_arc_ASANGA_20150827_03qfl69n_1_1
channel ORA_DISK_1: piece handle=/home/oracle/backup/full_arc_ASANGA_20150827_03qfl69n_1_1 tag=FULL_ARC_BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
archived log file name=+FLASH/asanga/archivelog/2015_08_28/thread_1_seq_2.444.888927577 thread=1 sequence=2
channel default: deleting archived log(s)
archived log file name=+FLASH/asanga/archivelog/2015_08_28/thread_1_seq_2.444.888927577 RECID=3 STAMP=888927577
unable to find archived log
archived log thread=1 sequence=3
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 08/28/2015 12:19:38
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 3 and starting SCN of 1483282

RMAN> alter database open resetlogs;

database opened

4. Shutdown and restart the database to see if it opens without any issues.

5. The single instance will have only one redo thread. Add anther redo thread (or more if RAC has more nodes) and enable it.

SQL> select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ ---------
         1          1          1   52428800        512          2 NO  CURRENT                1483283 28-AUG-15   2.8147E+14
         2          1          0   52428800        512          2 YES UNUSED                       0                      0
         3          1          0   52428800        512          2 YES UNUSED                       0                      0


alter database add logfile thread 2 group 4 ('+DATA','+FLASH') size 50m ;
alter database add logfile thread 2 group 5 ('+DATA','+FLASH') size 50m ;
alter database add logfile thread 2 group 6 ('+DATA','+FLASH') size 50m ;

SQL> alter database enable public thread 2;

Database altered.

SQL>  select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ ---------
         1          1          1   52428800        512          2 NO  CURRENT                1483283 28-AUG-15   2.8147E+14
         2          1          0   52428800        512          2 YES UNUSED                       0                      0
         3          1          0   52428800        512          2 YES UNUSED                       0                      0
         4          2          1   52428800        512          2 YES INACTIVE               1484502 28-AUG-15      1484512 28-AUG-15
         5          2          0   52428800        512          2 YES UNUSED                       0                      0
         6          2          0   52428800        512          2 YES UNUSED                       0                      0

6 rows selected.

6. The single instance would have had one undo tablespace. Create undo tablespace for each additional instance

create undo tablespace UNDOTBS2 datafile '+DATA(datafile)' SIZE 600M AUTOEXTEND ON NEXT 64M MAXSIZE UNLIMITED;

7. Create a pfile from the spfile and edit it by removing the *.undo_tablespace='UNDOTBS1' and add instance specific entries

*.cluster_database_instances=2
*.cluster_database=true
asanga1.instance_number=1
asanga2.instance_number=2
asanga1.thread=1
asanga2.thread=2
asanga1.undo_tablespace='UNDOTBS1'
asanga2.undo_tablespace='UNDOTBS2'

Also make sure log archive format has thread number

log_archive_format                   string      %t_%s_%r.dbf

8. Shutdown the database and copy the new pfile to all nodes and start each instance using the pfile.

[oracle@rhel6m1 ~]$ export ORACLE_SID=asanga1

[oracle@rhel6m1 backup]$ sqlplus  / as sysdba

SQL> startup pfile='/home/oracle/backup/initasanga.ora';
ORACLE instance started.

Total System Global Area 2137886720 bytes
Fixed Size                  2254952 bytes
Variable Size            1811941272 bytes
Database Buffers          318767104 bytes
Redo Buffers                4923392 bytes
Database mounted.
Database opened.

SQL> select instance_name,status from v$instance;

INSTANCE_NAME    STATUS
---------------- ------------
asanga1          OPEN

[oracle@rhel6m2 ~]$ export ORACLE_SID=asanga2
[oracle@rhel6m2 ~]$ sqlplus  / as sysdba

SQL> startup pfile='/home/oracle/initasanga.ora';
ORACLE instance started.

Total System Global Area 2137886720 bytes
Fixed Size                  2254952 bytes
Variable Size            1811941272 bytes
Database Buffers          318767104 bytes
Redo Buffers                4923392 bytes
Database mounted.
Database opened.

SQL> select instance_name,status from v$instance;

INSTANCE_NAME    STATUS
---------------- ------------
asanga2          OPEN

SQL> select HOST_NAME,INSTANCE_NUMBER,INSTANCE_NAME,STATUS,THREAD# from gv$instance;

HOST_NAME                 INSTANCE_NUMBER INSTANCE_NAME    STATUS          THREAD#
------------------------- --------------- ---------------- ------------ ----------
rhel6m2.domain.net                     2 asanga2          OPEN                  2
rhel6m1.domain.net                     1 asanga1          OPEN                  1

9. Once confirmed that all instances are opening without any issue, create a spfile in a shared location.

SQL> create spfile='+data' from pfile='/home/oracle/backup/initasanga.ora';

Make an alias in ASM for the spfile

ASMCMD> mkalias spfile.283.888929113 spfileasanga.ora
ASMCMD> ls -l
Type           Redund  Striped  Time             Sys  Name
PARAMETERFILE  UNPROT  COARSE   AUG 28 12:00:00  Y    spfile.283.888929113
                                                 N    spfileasanga.ora => +DATA/ASANGA/PARAMETERFILE/spfile.283.888929113

Create instance specific pfile with entries to the spfile alias. Also remove the spfile created earlier from the local node

cat initasanga1.ora
spfile='+DATA/ASANGA/PARAMETERFILE/spfileasanga.ora'

scp initasanga1.ora rhel6m2:`pwd`/initasanga2.ora

10. Create oracle password files for each node

11. Run the following script to create cluster related views

@?/rdbms/admin/catclust.sql

After running this scrip the database registry will show RAC component entry

COMP_ID    COMP_NAME                           STATUS     VERSION
---------- ----------------------------------- ---------- ----------
RAC        Oracle Real Application Clusters    VALID      11.2.0.4.0

12.Add the database to the cluster

srvctl add database -d asanga -o $ORACLE_HOME -p "+DATA/ASANGA/PARAMETERFILE/spfileasanga.ora"
srvctl add instance -d asanga -i asanga1 -n rhel6m1
srvctl add instance -d asanga -i asanga2 -n rhel6m2

[oracle@rhel6m1 dbs]$ srvctl config database -d asanga -a
Database unique name: asanga
Database name: rhel6m1
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_4
Oracle user: oracle
Spfile: +DATA/ASANGA/PARAMETERFILE/spfileasanga.ora
Domain: local
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: asanga
Database instances: asanga1,asanga2
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed

13. Shutdown both instances and start the database using srvctl.

srvctl start database -d asanga
srvctl status database -d asanga
Instance asanga1 is running on node rhel6m1
Instance asanga2 is running on node rhel6m2

srvctl config database -d asanga -a
Database unique name: asanga
Database name: rhel6m1
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_4
Oracle user: oracle
Spfile: +DATA/ASANGA/PARAMETERFILE/spfileasanga.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: asanga
Database instances: asanga1,asanga2
Disk Groups: DATA,FLASH
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed

14. Verify the database is also listed as a cluster resource

crsctl stat res ora.asanga.db
NAME=ora.asanga.db
TYPE=ora.database.type
TARGET=ONLINE           , ONLINE
STATE=ONLINE on rhel6m1, ONLINE on rhel6m2