Quantcast
Channel: A! Help
Viewing all 315 articles
Browse latest View live

12c Compression Comparison

$
0
0
OLTP Table compression introduced during 11g is called Advanced Row Compression in 12c. However the old syntax still works on 12c as well. Advance row compression maintains compression during all types of data manipulation operations, including conventional DML such as INSERT and UPDATE. Oracle's claim is that this minimizes the overhead for write operation of and making it suitable for OLTP and Data Warehouses. Advanced Row Compression requires purchasing Oracle Advanced Compression option while basic table compression is a feature of Oracle Database 12c Enterprise Edition (EE).
This post compares the reduction of storage usage, overhead of CPU and redo generated for advance vs basic vs no compression options. It is not an extensive test, how useful the compression depends on data being compressed.
As said earlier compress for OLTP creates the table with advance row compression.
create table x (a number, b varchar2(100)) compress for oltp;
select table_name,COMPRESSION,COMPRESS_FOR from user_tables where table_name='X';

TABLE_NAME COMPRESS COMPRESS_FOR
---------- -------- ------------------------------
X ENABLED ADVANCED

create table x (a number, b varchar2(100)) ROW STORE COMPRESS ADVANCED;
select table_name,COMPRESSION,COMPRESS_FOR from user_tables where table_name='X';

TABLE_NAME COMPRESS COMPRESS_FOR
---------- -------- ------------------------------
X ENABLED ADVANCED
If no option is specified this would create basic compression or it could be explicitly specified.
create table x (a number, b varchar2(100)) compress;
SQL> select table_name,COMPRESSION,COMPRESS_FOR from user_tables where table_name='X';

TABLE_NAME COMPRESS COMPRESS_FOR
---------- -------- ------------------------------
X ENABLED BASIC

create table x (a number, b varchar2(100)) ROW STORE COMPRESS basic;
select table_name,COMPRESSION,COMPRESS_FOR from user_tables where table_name='X';

TABLE_NAME COMPRESS COMPRESS_FOR
---------- -------- ------------------------------
X ENABLED BASIC
For the test case three tables were created with no compression, advance and basic compression. The database is 12.1.0.2.
create table NOCOMPRESSTABLE(a number, b varchar2(100));
create table ADVCOMPRESSTABLE(a number, b varchar2(100)) ROW STORE COMPRESS ADVANCED;
create table BASICVCOMPRESSTABLE(a number, b varchar2(100)) ROW STORE COMPRESS basic;

SQL> select table_name,COMPRESSION,COMPRESS_FOR from user_tables where table_name like '%COMPRESSTABLE';

TABLE_NAME COMPRESS COMPRESS_FOR
-------------------- -------- -------------
NOCOMPRESSTABLE DISABLED
BASICVCOMPRESSTABLE ENABLED BASIC
ADVCOMPRESSTABLE ENABLED ADVANCED
Each table consists of two columns and was populated using the following anonymous PL/SQL block. It inserts a unique value to first column while second column get the same value inserted with each row.
begin
for i in 1 .. 1000000
loop
insert into NOCOMPRESSTABLE values (i,'asdfghjkllqwertyuiopzxcvbnm134567890');
insert into ADVCOMPRESSTABLE values(i,'asdfghjkllqwertyuiopzxcvbnm134567890');
insert into BASICVCOMPRESSTABLE values(i,'asdfghjkllqwertyuiopzxcvbnm134567890');
commit;
end loop;
end;
/
At the end of the insert the segment size of each table was measured
select segment_name,bytes/1024/1024 as MB from user_segments where segment_name like '%COMPRESS%';

SEGMENT_NAME MB
--------------------- ---
ADVCOMPRESSTABLE 14
BASICVCOMPRESSTABLE 47
NOCOMPRESSTABLE 52
Results are no surprise as the segment of the table created with advance compression is the smallest and table with no compression is the largest. However this test is the most optimistic of cases where one column consists of only a single value.





Therefore the test was rerun after recreating the tables and inserts were modified as below where some of the values inserted to second column are duplicates but not all of them
begin
for i in 1 .. 1000000
loop
-- insert into NOCOMPRESSTABLE values (i,'asdfghjkllqwertyuiopzxcvbnm134567890'||mod(i,500));
insert into ADVCOMPRESSTABLE values(i,'asdfghjkllqwertyuiopzxcvbnm134567890'||mod(i,500));
-- insert into BASICVCOMPRESSTABLE values(i,'asdfghjkllqwertyuiopzxcvbnm134567890'||mod(i,500));
commit;
end loop;
end;
/
Only one table was inserted at a time and CPU used by session and redo size were measured for each test, shown on the graphs below.
Strangely the inserts done with compress options used less CPU than when inserted without any compression (test were repeated 3 times and same pattern was observed). However the difference wasn't huge. On the other hand use of advance compression option resulted in more redo being created than basic compression or no compression.
Comparing the segment size revealed the following.
select segment_name,bytes/1024/1024 as MB from user_segments where segment_name like '%COMPRESS%';

SEGMENT_NAME MB
--------------------- ---
ADVCOMPRESSTABLE 55
BASICVCOMPRESSTABLE 50
NOCOMPRESSTABLE 55
It seems basic compression is slightly better than no compression option but advance compression didn't yield any benefit at all. Reason could be that advance compression didn't encounter enough duplicate values inside a block to make any significant reduction. It's important to remember that compression happens at block level.
Test was rerun this time however the tables were created on a tablespace with a block size of 32k. The segment size at the end of the test are as below
select segment_name,bytes/1024/1024 as MB from user_segments where segment_name like '%COMPRESS%';

SEGMENT_NAME MB
--------------------- ---
ADVCOMPRESSTABLE 49
BASICVCOMPRESSTABLE 49
NOCOMPRESSTABLE 54
This time there's difference between segment size whose tables were created with a compression options and table created without any compression. However the reduction in segment size is not same as when all the values were duplicates.
From this simple test case it seems, to benefit from a compression strategy, the nature of the data is more important than the compression option being used. The reduction in space is dictated by how many duplicate values are found inside a block.

Convert Single Instance DB to RAC DB - Manual Method

$
0
0
There are many ways to convert a single database to RAC. Using database template, rconfig or the manual method. This post shows the steps for manual method of converting a single instance database to RAC (tested for both 11.2.0.4 and 12.1.0.2). The single database is running out of a non-rac oracle home. Backup of this would be restored using a rac enabled oracle home and then converted to a RAC database.
The database is called "asanga" and will retain the same name when converted to RAC.
1. Create a pfile of the single instance database. The pfile entries are shown below
 more pfile.ora
*.audit_file_dest='/opt/app/oracle/admin/asanga/adump'
*.audit_trail='NONE'
*.compatible='11.2.0.4.0'
*.db_block_size=8192
*.db_create_file_dest='+DATA'
*.db_domain=''
*.db_name='ASANGA'
*.db_recovery_file_dest_size=5218762752
*.db_recovery_file_dest='+FLASH'
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=asangaXDB)'
*.open_cursors=300
*.pga_aggregate_target=524288000
*.processes=3000
*.remote_login_passwordfile='EXCLUSIVE'
*.sessions=4524
*.sga_target=2147483648
*.undo_tablespace='UNDOTBS1'
2. Create the adump directories in all nodes of the cluster (in this case it is a two node cluster)
mkdir -p /opt/app/oracle/admin/asanga/adump
3. Copy the pfile created earlier to one of the RAC nodes and restore the datababase.
SQL> startup nomount pfile='/home/oracle/backup/pfile.ora';

SQL> create spfile from pfile='/home/oracle/backup/pfile.ora';
SQL> startup force nomount;

RMAN> restore controlfile from '/home/oracle/backup/ctlbkp04qfl69p_1_1.ctl';

Starting restore at 28-AUG-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=2275 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
output file name=+DATA/asanga/controlfile/current.270.888927491
output file name=+FLASH/asanga/controlfile/current.447.888927493
Finished restore at 28-AUG-15

RMAN> alter database mount;

RMAN> catalog start with '/home/oracle/backup/';

RMAN> run {
2> restore database;
3> recover database;
4> }

Starting restore at 28-AUG-15
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to +DATA/asanga/datafile/system.309.888838053
channel ORA_DISK_1: restoring datafile 00002 to +DATA/asanga/datafile/sysaux.308.888838053
channel ORA_DISK_1: restoring datafile 00003 to +DATA/asanga/datafile/undotbs1.307.888838053
channel ORA_DISK_1: restoring datafile 00004 to +DATA/asanga/datafile/users.310.888838053
channel ORA_DISK_1: reading from backup piece /home/oracle/backup/full_bkp_ASANGA_20150827_02qfl697_1_1
channel ORA_DISK_1: piece handle=/home/oracle/backup/full_bkp_ASANGA_20150827_02qfl697_1_1 tag=FULL_BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:25
Finished restore at 28-AUG-15

Starting recover at 28-AUG-15
using channel ORA_DISK_1

starting media recovery

channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=2
channel ORA_DISK_1: reading from backup piece /home/oracle/backup/full_arc_ASANGA_20150827_03qfl69n_1_1
channel ORA_DISK_1: piece handle=/home/oracle/backup/full_arc_ASANGA_20150827_03qfl69n_1_1 tag=FULL_ARC_BKP
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
archived log file name=+FLASH/asanga/archivelog/2015_08_28/thread_1_seq_2.444.888927577 thread=1 sequence=2
channel default: deleting archived log(s)
archived log file name=+FLASH/asanga/archivelog/2015_08_28/thread_1_seq_2.444.888927577 RECID=3 STAMP=888927577
unable to find archived log
archived log thread=1 sequence=3
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 08/28/2015 12:19:38
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 3 and starting SCN of 1483282

RMAN> alter database open resetlogs;

database opened
4. Shutdown and restart the database to see if it opens without any issues.

5. The single instance will have only one redo thread. Add anther redo thread (or more if RAC has more nodes) and enable it.
SQL> select * from v$log;

GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ ---------
1 1 1 52428800 512 2 NO CURRENT 1483283 28-AUG-15 2.8147E+14
2 1 0 52428800 512 2 YES UNUSED 0 0
3 1 0 52428800 512 2 YES UNUSED 0 0


alter database add logfile thread 2 group 4 ('+DATA','+FLASH') size 50m ;
alter database add logfile thread 2 group 5 ('+DATA','+FLASH') size 50m ;
alter database add logfile thread 2 group 6 ('+DATA','+FLASH') size 50m ;

SQL> alter database enable public thread 2;

Database altered.

SQL> select * from v$log;

GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ ---------
1 1 1 52428800 512 2 NO CURRENT 1483283 28-AUG-15 2.8147E+14
2 1 0 52428800 512 2 YES UNUSED 0 0
3 1 0 52428800 512 2 YES UNUSED 0 0
4 2 1 52428800 512 2 YES INACTIVE 1484502 28-AUG-15 1484512 28-AUG-15
5 2 0 52428800 512 2 YES UNUSED 0 0
6 2 0 52428800 512 2 YES UNUSED 0 0

6 rows selected.
6. The single instance would have had one undo tablespace. Create undo tablespace for each additional instance
create undo tablespace UNDOTBS2 datafile '+DATA(datafile)' SIZE 600M AUTOEXTEND ON NEXT 64M MAXSIZE UNLIMITED;



7. Create a pfile from the spfile and edit it by removing the *.undo_tablespace='UNDOTBS1' and add instance specific entries
*.cluster_database_instances=2
*.cluster_database=true
asanga1.instance_number=1
asanga2.instance_number=2
asanga1.thread=1
asanga2.thread=2
asanga1.undo_tablespace='UNDOTBS1'
asanga2.undo_tablespace='UNDOTBS2'
Also make sure log archive format has thread number
log_archive_format                   string      %t_%s_%r.dbf
8. Shutdown the database and copy the new pfile to all nodes and start each instance using the pfile.
[oracle@rhel6m1 ~]$ export ORACLE_SID=asanga1

[oracle@rhel6m1 backup]$ sqlplus / as sysdba

SQL> startup pfile='/home/oracle/backup/initasanga.ora';
ORACLE instance started.

Total System Global Area 2137886720 bytes
Fixed Size 2254952 bytes
Variable Size 1811941272 bytes
Database Buffers 318767104 bytes
Redo Buffers 4923392 bytes
Database mounted.
Database opened.

SQL> select instance_name,status from v$instance;

INSTANCE_NAME STATUS
---------------- ------------
asanga1 OPEN

[oracle@rhel6m2 ~]$ export ORACLE_SID=asanga2
[oracle@rhel6m2 ~]$ sqlplus / as sysdba

SQL> startup pfile='/home/oracle/initasanga.ora';
ORACLE instance started.

Total System Global Area 2137886720 bytes
Fixed Size 2254952 bytes
Variable Size 1811941272 bytes
Database Buffers 318767104 bytes
Redo Buffers 4923392 bytes
Database mounted.
Database opened.

SQL> select instance_name,status from v$instance;

INSTANCE_NAME STATUS
---------------- ------------
asanga2 OPEN

SQL> select HOST_NAME,INSTANCE_NUMBER,INSTANCE_NAME,STATUS,THREAD# from gv$instance;

HOST_NAME INSTANCE_NUMBER INSTANCE_NAME STATUS THREAD#
------------------------- --------------- ---------------- ------------ ----------
rhel6m2.domain.net 2 asanga2 OPEN 2
rhel6m1.domain.net 1 asanga1 OPEN 1
9. Once confirmed that all instances are opening without any issue, create a spfile in a shared location.
SQL> create spfile='+data' from pfile='/home/oracle/backup/initasanga.ora';
Make an alias in ASM for the spfile
ASMCMD> mkalias spfile.283.888929113 spfileasanga.ora
ASMCMD> ls -l
Type Redund Striped Time Sys Name
PARAMETERFILE UNPROT COARSE AUG 28 12:00:00 Y spfile.283.888929113
N spfileasanga.ora => +DATA/ASANGA/PARAMETERFILE/spfile.283.888929113
Create instance specific pfile with entries to the spfile alias. Also remove the spfile created earlier from the local node
cat initasanga1.ora
spfile='+DATA/ASANGA/PARAMETERFILE/spfileasanga.ora'

scp initasanga1.ora rhel6m2:`pwd`/initasanga2.ora
10. Create oracle password files for each node

11. Run the following script to create cluster related views
@?/rdbms/admin/catclust.sql
After running this scrip the database registry will show RAC component entry
COMP_ID    COMP_NAME                           STATUS     VERSION
---------- ----------------------------------- ---------- ----------
RAC Oracle Real Application Clusters VALID 11.2.0.4.0
12.Add the database to the cluster
srvctl add database -d asanga -o $ORACLE_HOME -p "+DATA/ASANGA/PARAMETERFILE/spfileasanga.ora"
srvctl add instance -d asanga -i asanga1 -n rhel6m1
srvctl add instance -d asanga -i asanga2 -n rhel6m2

[oracle@rhel6m1 dbs]$ srvctl config database -d asanga -a
Database unique name: asanga
Database name: rhel6m1
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_4
Oracle user: oracle
Spfile: +DATA/ASANGA/PARAMETERFILE/spfileasanga.ora
Domain: local
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: asanga
Database instances: asanga1,asanga2
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed
13. Shutdown both instances and start the database using srvctl.
srvctl start database -d asanga
srvctl status database -d asanga
Instance asanga1 is running on node rhel6m1
Instance asanga2 is running on node rhel6m2

srvctl config database -d asanga -a
Database unique name: asanga
Database name: rhel6m1
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_4
Oracle user: oracle
Spfile: +DATA/ASANGA/PARAMETERFILE/spfileasanga.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: asanga
Database instances: asanga1,asanga2
Disk Groups: DATA,FLASH
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed
14. Verify the database is also listed as a cluster resource
crsctl stat res ora.asanga.db
NAME=ora.asanga.db
TYPE=ora.database.type
TARGET=ONLINE , ONLINE
STATE=ONLINE on rhel6m1, ONLINE on rhel6m2

Moving Grid Infrastructure Management Repository (GIMR) Database (MGMTDB) Out of Default Disk Group

$
0
0
With 12.1.0.2 creation of Grid Infrastructure Management Repository (GIMR) is mandatory. This results in a CDB called -MGMTDB being created at the end of the GI installation. This CDB (-MGMTDB) contains a single PDB which has the same name as the cluster name. If ASM is used for storing OCR/Vote then, when creating the cluster, by default this database is created on the same disk group where ocr/vote resides. As of 12.1.0.2 there's no way to specify a different disk group for GIMR alone. This post shows steps to moving the GIMR to a different disk group from the default disk group.
1. Create a new ASM disk group to store GIMR. In this case this new disk group is called GIMR and has external redundancy. The GIMR currently reside in a disk group called CLUSFS with normal redundancy. It's important that newly created disk group has compatible.asm and compatible.rdbms set to 12.1.
SQL> select name,type total_mb,free_mb,REQUIRED_MIRROR_FREE_MB,USABLE_FILE_MB from v$asm_diskgroup;

NAME TOTAL_ FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------------------------ ------ ---------- ----------------------- --------------
CLUSFS NORMAL 17400 10236 3582
GIMR EXTERN 10108 0 10108

SQL> select name,os_mb,total_mb,free_mb from v$asm_disk where name is not null;

NAME OS_MB TOTAL_MB FREE_MB
------------------------------ ---------- ---------- ----------
GIMR_0000 10236 10236 10108
CLUSFS_0002 10236 10236 5804
CLUSFS_0000 10236 10236 5812
CLUSFS_0001 10236 10236 5800
2. Stop and disable ora.crf resource
# crsctl stop res ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded

# crsctl modify res ora.crf -attr ENABLED=0 -init
3. Find the node GIMR database is running and run the delete database command from that node. This could drop the current GIMR databse.
$ srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node rhel12c1

[grid@rhel12c1 grid2]$ dbca -silent -deleteDatabase -sourceDB -MGMTDB
Connecting to database
4% complete
9% complete
14% complete
19% complete
23% complete
28% complete
47% complete
Updating network configuration files
48% complete
52% complete
Deleting instance and datafiles
76% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb.log" for further details.
4. Create the GIMR with the name "-MGMTDB" and specifying new ASM diskgroup
$ dbca -silent -createDatabase -sid -MGMTDB -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc 
-gdbName _mgmtdb -storageType ASM -diskGroupName +GIMR -datafileJarLocation $ORACLE_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
37% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.


5. Before creating the PDB connected with the GIMR find out the cluster name. There are many ways to find out the cluster name simplest is to use cemutlo.
$ cemutlo -n
rhel12c-cluster
6. Trying to create PDB with above cluster name will fail.
$ dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName rhel12c-cluster -createPDBFrom RMANBACKUP -PDBBackUpfile $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb0.log" for further details.

$ more /opt/app/oracle/cfgtoollogs/dbca/_mgmtdb0.log
The Container database selected is in the open state
rhel12c-cluster: PDB Name must be at least 1 character and at most 30 characters,
should start with an alphabetical character and must include only alphabetical characters, numbers or the '_' character.
The reason is database names cannot contain - (dash) and cluster name cannot contain _ (underscore). "The cluster name is case-insensitive, must be unique across your enterprise, must be at least one character long and no more than 15 characters in length, must be alphanumeric, cannot begin with a numeral, and may contain hyphens (-). Underscore characters (_) are not allowed.".
So how did the GIMR got created in the first place? It seems if the cluster name contains (-) installer implicitly replace them with (_). This could be verified by looking in the $ORACLE_BASE/cfgtoollogs/dbca/_mgmtdb folder. This will contain a folder called rhel12c_cluster which has all the log files related to original PDB creation. Therefore if the cluster name contains (-) then replace it with (_) and run the PDB create command again.
$ dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName rhel12c_cluster -createPDBFrom RMANBACKUP -PDBBackUpfile $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck
Creating Pluggable Database
4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/rhel12c_cluster/_mgmtdb0.log" for further details.
7. Find out on which node the -MGMTDB database is running and run mgmtca from that node to secure the GIMR. Running mgmtca doesn't produce any output.
$ srvctl status MGMTDB
Database is enabled
Instance -MGMTDB is running on node rhel12c1
[grid@rhel12c1 _mgmtdb]$ mgmtca
8. Finally enable and start the ora.crf resource.
# crsctl modify res ora.crf -attr ENABLED=1 -init
# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
Useful metalink notes
Managing the Cluster Health Monitor Repository [ID 1921105.1]
FAQ: 12c Grid Infrastructure Management Repository (GIMR) [ID 1568402.1]
How to Move GI Management Repository to Different Shared Storage (Diskgroup, CFS or NFS etc) [ID 1589394.1]

12c Encryption

$
0
0
This post gives a highlight of using TDE in 12c. For detail information refer advance security guide. In 12c the orapki/alter system commands related to key management has been replaced with ADMINISTER KEY MANAGEMENT commands.
The first set of steps shows setting a non-CDB for use of TDE. 1. Create a location for wallet files (key store location).
mkdir -p /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde
chmod 700 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde
2. Add ENCRYPTION_WALLET_LOCATION to sqlnet.ora file
ENCRYPTION_WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA =
(DIRECTORY = /opt/app/oracle/product/12.1.0/dbhome_2/network/admin/tde)
)
)
3. Create the software key store (wallet) by specifying key store location and password or key store (asanga123 in this case)
SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde' identified by asanga123;
Result of this is a key store file (wallet file).
SQL> ! ls
ewallet.p12
At this stage the status of the key store would be closed
SQL> select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID WRL_PARAMETER
---------- ------- ------------- --------- --------- ---------- -----------------------------------------------------------
FILE CLOSED UNKNOWN SINGLE UNDEFINED 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/
4. Open the key store by providing the key store password
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY asanga123;
keystore altered.

SQL> select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID WRL_PARAMETER
---------- ------------------ ------------- --------- --------- ---------- -----------------------------------------------------------
FILE OPEN_NO_MASTER_KEY PASSWORD SINGLE NO 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/
5. Usually at this stage auto login key store is also created. But this lead to an issue (refer 1944507.1).
SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY asanga123 WITH BACKUP;
ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY asanga123 WITH BACKUP
*
ERROR at line 1:
ORA-28417: password-based keystore is not open

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE open identified by asanga123;
ADMINISTER KEY MANAGEMENT SET KEYSTORE open identified by asanga123
*
ERROR at line 1:
ORA-28354: Encryption wallet, auto login wallet, or HSM is already open
Therefore master encryption key is created before creating auto login wallet.
SQL>  ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY asanga123 with backup ;
keystore altered.
This will change wallet status from open_no_master_key to open.
SQL> select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID WRL_PARAMETER
---------- ------- ------------- --------- --------- ---------- -----------------------------------------------------------
FILE OPEN PASSWORD SINGLE NO 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/
On the database alert log following could be observed
Creating new database key for new master key and wallet
Creating new database key with the new master key
Retiring: ena 2 flag 6 mkloc 0
encrypted key 8dd09c987ef966198af992379477f13900000000000000000000000000000000
mkid b81a02de82664fbcbf2c9bcdcec4a3ae
Creating: ena 2 flag e mkloc 1
encrypted key 92c72aeada0197dda6da3e4d64ac875c00000000000000000000000000000000
mkid ac6f7e2c97ff4fdfbf9d900a653e2c21
Switching out all online logs for the new master key
6. Finally create the auto login key store so that key store is auto opened when the database starts
SQL> ADMINISTER KEY MANAGEMENT CREATE local AUTO_LOGIN KEYSTORE FROM KEYSTORE '/opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde' identified by asanga123;
keystore altered.
This will create the cwallet.sso file and wallet type will be changed to local_autologin
SQL> ! ls
cwallet.sso ewallet_2015101517371593.p12 ewallet.p12 ewallet.p12.bak

select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID WRL_PARAMETER
---------- ------ ----------------- --------- --------- ---------- -----------------------------------------------------------
FILE OPEN LOCAL_AUTOLOGIN SINGLE NO 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/
Encryption could be enabled on CDB with PDB same way as above. Below is an example of enabling TDE for CDB with two PDBs.
SQL> show pdbs

CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 PDBONE READ WRITE NO
4 PDBTWO READ WRITE NO
Create key store, open and set master encryption key.
SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde' identified by asanga123;
keystore altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY asanga123 container=all;
keystore altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY asanga123 with backup container=all;
keystore altered.

SQL> select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID WRL_PARAMETER
---------- ------- ------------- --------- --------- ---------- -----------------------------------------------------------
FILE OPEN PASSWORD SINGLE NO 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/
Enable auto login
SQL> ADMINISTER KEY MANAGEMENT CREATE local AUTO_LOGIN KEYSTORE FROM KEYSTORE '/opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde' identified by asanga123;
keystore altered.
Login to a PDB and check the wallet status
SQL> show con_name

CON_NAME
-----------
PDBTWO

select WRL_TYPE,STATUS,WALLET_TYPE,WALLET_ORDER,FULLY_BACKED_UP,CON_ID,WRL_PARAMETER from v$ENCRYPTION_WALLET;

WRL_TYPE STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID WRL_PARAMETER
---------- ------ ----------------- --------- --------- ---------- -----------------------------------------------------------
FILE OPEN LOCAL_AUTOLOGIN SINGLE NO 0 /opt/app/oracle/product/12.1.0/dbhome_1/network/admin/tde/
v$encryption_keys view could be used to identify each containers key ids even though container id remains 0 for all.
SQL> select CON_ID,KEY_ID,KEYSTORE_TYPE,CREATOR_DBNAME,CREATOR_PDBNAME from v$encryption_keys;

CON_ID KEY_ID KEYSTORE_TYPE CREATOR_DBNAME CREATOR_PDBNAME
---------- ------------------------------------------------------ ----------------- --------------- ------------------------------
0 AQugdpFHIk9yv1tQiZj0EhUAAAAAAAAAAAAAAAAAAAAAAAAAAAAA SOFTWARE KEYSTORE PDBENC CDB$ROOT
0 Af8ZFXnsWk+/v9Z4RFalEHwAAAAAAAAAAAAAAAAAAAAAAAAAAAAA SOFTWARE KEYSTORE PDBENC PDBONE
0 AU3e99wEOk8lv4QEBp4Ow3AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA SOFTWARE KEYSTORE PDBENC PDBTWO
Following test could be used to verify TDE is working. In this case a tablespace is created with encryption enabled. Few rows are inserted to tables created in those table spaces. At times buffer cache flushing or manual check points may be required to force database writer to write to data files
SQL>  create tablespace enctest datafile size 10m ENCRYPTION DEFAULT STORAGE(ENCRYPT);
Tablespace created.
SQL> create tablespace nonenctest datafile size 10m ;
Tablespace created.

SQL>select TABLESPACE_NAME,ENCRYPTED from dba_tablespaces;

TABLESPACE_NAME ENC
------------------------------ ---
ENCTEST YES
NONENCTEST NO

create table enctable (a varchar2(100)) tablespace ENCTEST;
create table nonecntbale (a varchar2(100)) tablespace NONENCTEST;

begin
for i in 1 .. 100
loop
insert into enctable values ('top secret text');
insert into nonecntbale values ('top secret text');
end loop;
commit;
end;
/
alter system checkpoint;
Strings command is used to look into the data file content.First the data file belonging to non-encrypted tablespace
strings /opt/app/oracle/oradata/ENT12C1/datafile/o1_mf_nonencte_c1zsml08_.dbf
top secret text,
top secret text,
Secondly the data file belonging to the encrypted tablespace
strings /opt/app/oracle/oradata/ENT12C1/datafile/o1_mf_enctest_c1zsl0d9_.dbf
Vj#>{
O+l;2
c1ax|
drl
fzc1
Lbqby%}
u;Fa
=B]Wv
~/th
9hHW
=Jc;
@s|J84
|3M*
2ATG
As seen above encrypted tablespace doesn't show it's content in clear text.



Encryption requires additional CPU. Next is a minor test that was done to compare the CPU usage for inserting to a encryption enabled tablespace vs non-encrypted tablespace.
create table NOencryptiontbl(a number, b varchar2(100)) tablespace NONENCTEST;
create table encryptiontbl(a number, b varchar2(100)) tablespace ENCTEST;

begin
for i in 1 .. 1000000
loop
--insert into encryptiontbl values (i,'asdfghjkllqwertyuiopzxcvbnm134567890'||i);
insert into NOencryptiontbl values (i,'asdfghjkllqwertyuiopzxcvbnm134567890'||i);
commit;
end loop;
end;
/
The inserts were first done on table created on non-encrypted and then on table created on encryption enabled tablespace. Value of CPU used by this session statistic was used to compare the CPU usage of each session once the inserts have completed. Graph below shows the CPU used, with TDE enabled it took on average 51 CPU seconds more for inserts to complete.

If TDE is used in a data guard environment then the standby must have the key store (wallet) copied over from the primary. Without wallet open the recovery at standby will fail. In the below output datafile 17 was part of a encrypted tablespace created at primary.
Datafile #17: '/opt/app/oracle/oradata/ENT12C1S/datafile/o1_mf_t2_bv215f2m_.dbf'
kcrf_decrypt_redokey: wallet is not opened..(err 28365)
Errors with log /opt/app/oracle/fast_recovery_area/ENT12C1S/archivelog/2015_07_23/o1_mf_1_2726_bv214gg4_.arc
MRP0: Background Media Recovery terminated with error 28365
Thu Jul 23 16:13:49 2015
Errors in file /opt/app/oracle/diag/rdbms/ent12c1s/ent12c1s/trace/ent12c1s_pr00_2541.trc:
ORA-28365: wallet is not open
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!

Recovered data files to a consistent state at change 41627576
Thu Jul 23 16:13:50 2015
Errors in file /opt/app/oracle/diag/rdbms/ent12c1s/ent12c1s/trace/ent12c1s_pr00_2541.trc:
ORA-28365: wallet is not open
Thu Jul 23 16:13:51 2015
MRP0: Background Media Recovery process shutdown (ent12c1s)
Another thing to watch out for is exporting using exp as oppose to expdp. Tables residing in encrypted tablespaces cannot be exported using exp.
exp asanga/asa file=asa.dmp tables=p1,p2
Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
About to export specified tables via Conventional Path ...
EXP-00111: Table P1 resides in an Encrypted Tablespace T2 and will not be exported
EXP-00111: Table P2 resides in an Encrypted Tablespace T2 and will not be exported
Export terminated successfully with warnings.
Useful metalink notes
TDE Wallet Problem in 12c: Cannot do a Set Key operation when an auto-login wallet is present [ID 1944507.1]
Master Note For Transparent Data Encryption ( TDE ) [ID 1228046.1]
How to migrate a non pluggable database that uses TDE to pluggable database ? [ID 1678525.1]
Known TDE Wallet Issues [ID 1301365.1]
Auto Login Wallet Cannot Be Closed [ID 1204604.1]
Removing TDE, but V$ENCRYPTION_WALLET / GV$ENCRYPTION_WALLET still has rows. [ID 2003528.1]
Step by Step Troubleshooting Guide for TDE Error ORA-28374 [ID 1541818.1]
How To Import Encrypted Datapump Data from 11G To 12c DB [ID 1642059.1]

Relink Fails due to Requirement Checks Failure

$
0
0
Oracle recommends re-linking oracle binaries after OS upgrade. There are few earlier posts on this topic. In this case the server in question was a Amazon EC2 server and didn't have any swap configured.
$ cat /proc/meminfo | grep "Swap"
SwapCached: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
After the OS upgrade , relinking of oracle binaries failed due to insufficient swap space.
$ ./relink all
writing relink log to: /opt/app/oracle/product/testbox/11.2.0/install/relink.log

$ more ../install/relink.log
Starting Oracle Universal Installer...

Checking swap space: 0 MB available, 500 MB required. Failed <<<<

Some requirement checks failed. You must fulfill these requirements before

continuing with the installation,

Exiting Oracle Universal Installer, log for this session can be found at /opt/app/oracle/oraInventory/logs/installActions2015-09-25_12-11-54PM.log
Only parameters that could be passed on to relink is all and as_installed. However looking inside relink script it could be seen that it is using runInstaller.
#-----------------------------------
# location of runInstaller executable
RUNINSTALLER=$ORACLE_HOME/oui/bin/runInstaller

#--------------------------
Runinstaller command gets set of arguments.
#-----------------------------------
# full argument list for runInstaller
#
ARGS="-relink -waitForCompletion -maketargetsxml $MAKEORDER $LOGDIR_ARG ORACLE_HOME=$ORACLE_HOME"

#--------------------------
Adding -ignoreSysPrereqs as the first argument resolved the issue.
ARGS="-ignoreSysPrereqs -relink -waitForCompletion -maketargetsxml $MAKEORDER $LOGDIR_ARG ORACLE_HOME=$ORACLE_HOME"
Relink completes successfully afterwards and relink.log will end with the following for oracle home relinks
test ! -f /opt/app/oracle/product/testbox/11.2.0/bin/oracle ||\
mv -f /opt/app/oracle/product/testbox/11.2.0/bin/oracle /opt/app/oracle/product/testbox/11.2.0/bin/oracleO
mv /opt/app/oracle/product/testbox/11.2.0/rdbms/lib/oracle /opt/app/oracle/product/testbox/11.2.0/bin/oracle
chmod 6751 /opt/app/oracle/product/testbox/11.2.0/bin/oracle
and with the following for GI home relinking
test ! -f /grid/oracle/product/11.2.0/bin/oracle ||\
mv -f /grid/oracle/product/11.2.0/bin/oracle /grid/oracle/product/11.2.0/bin/oracleO
mv /grid/oracle/product/11.2.0/rdbms/lib/oracle /grid/oracle/product/11.2.0/bin/oracle
chmod 6751 /grid/oracle/product/11.2.0/bin/oracle
Once the relinking is completed remove the ignoreSysPrereqs parameter from the relink file.
If the server already has swap created and want to recreate the same scenario as above for testing, use
swapoff -a
swapon -a
to disable and enable swapping
Also if the environment uses role separation then after the relink it may be necessary to set the correct permission on the oracle binary using setasmgidwrap.

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment (RHEL 5)
Upgrading ASMLib and OS in 11gR1 RAC Environment


Parameterizing Backup Tags

$
0
0
At times it may be necessary to customize the back tag value. This could be achieved in two ways.
First method is used when a backups are run using a separate rman backup script file, which is used by the session invoking the rman. In this case the customized backup tags could be passed with the "USING" clause. Below is a simple rman backup script which takes a full database backup and archive log
cat backup.rmn
run {
backup database tag='&1' plus archivelog tag='&2';
delete noprompt obsolete;
}
Tags has been parameterized with the use of '&n'. Values for these are passed via the shell script that invokes rman
cat backup.sh
export ORACLE_SID=racse11g2
...
today_date=$(date +%F)
arctag=arch_bkp_$(date +%Y_%m_%d)
bkptag=full_bkp_$(date +%Y_%m_%d)
logfile=$ORACLE_SID"_"$today_date".log"
rman target / @/home/oracle/cronjobs/backup.rmn log /home/oracle/cronjobs/logs/$logfile using $bkptag $arctag
Result of this is customized tags for each days backup.
List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
79 B A A DISK 06-OCT-15 1 1 NO ARCH_BKP_2015_10_06
80 B F A DISK 06-OCT-15 1 1 NO FULL_BKP_2015_10_06
81 B A A DISK 06-OCT-15 1 1 NO ARCH_BKP_2015_10_06
...
179 B A A DISK 07-OCT-15 1 1 NO ARCH_BKP_2015_10_07
180 B F A DISK 07-OCT-15 1 1 NO FULL_BKP_2015_10_07
181 B A A DISK 07-OCT-15 1 1 NO ARCH_BKP_2015_10_07



Second method is used when rman is invoked within a shell script. In this case customized tag is appended to a string
cat backupdisk.sh
export ORACLE_SID=racse11g1
...
logfile="disk_"$ORACLE_SID"_"$today_date".log"
tag=$(date +%Y_%m_%d)

rman target / log /home/oracle/cronjobs/logs/$logfile <<EOF
run
{
backup database tag = 'full_disk_$tag' format '/backup/full_bkp_%d_%T_%U' plus archivelog tag = 'full_arc_disk_$tag' format '/backup/full_arc_%d_%T_%U';
delete noprompt obsolete;
}
exit;
EOF
This will create customized tags as follows
List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
83 B A A DISK 06-OCT-15 1 1 NO FULL_ARC_DISK_2015_10_06
84 B F A DISK 06-OCT-15 1 1 NO FULL_DISK_2015_10_06
85 B A A DISK 06-OCT-15 1 1 NO FULL_ARC_DISK_2015_10_06
...
183 B A A DISK 07-OCT-15 1 1 NO FULL_ARC_DISK_2015_10_07
184 B F A DISK 07-OCT-15 1 1 NO FULL_DISK_2015_10_07
185 B A A DISK 07-OCT-15 1 1 NO FULL_ARC_DISK_2015_10_07

Direct NFS Setup

$
0
0
This post shows steps for setting up directNFS for 11.2 and 12.1 (steps are identical). For 11.1 please refer here.
Direct NFS requires a working NFS setup to already exists. Create a NFS setup and mount the NFS on the local server with oracle recommended nfs mount options (refer 359515.1) For this test case (tested with 11.2 and 12.1 both SE, SE2 and EE) nfs mount is as follows
# mount
...
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
192.168.0.104:/opt/backup on /usr/local/ochm/nfsdir type nfs (rw,hard,rsize=32768,wsize=32768,nfsvers=3,nointr,timeo=600,tcp,nolock,actimeo=0,addr=192.168.0.104)
Add an entry to fstab as well
# cat /etc/fstab
...
192.168.0.104:/opt/backup /usr/local/ochm/nfsdir nfs rw,hard,rsize=32768,wsize=32768,nfsvers=3,nointr,timeo=600,tcp,nolock,actimeo=0,addr=192.168.0.104 0 0
Test to see if the oracle user has write permission on the nfs mount point by creating dummy files. When the nfs is working without any issues configure the oracle to use direct nfs.
Enable direct nfs by executing the rdbms make with dnfs on.
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk dnfs_on
rm -f /opt/app/oracle/product/12.1.0/std2/rdbms/lib/odm/libnfsodm12.so; \
cp /opt/app/oracle/product/12.1.0/std2/lib/libnfsodm12.so /opt/app/oracle/product/12.1.0/std2/rdbms/lib/odm/libnfsodm12.so
To disable direct NFS use
make -f ins_rdbms.mk dnfs_off
rm -f /opt/app/oracle/product/12.1.0/std2/rdbms/lib/odm/libnfsodm12.so
Verify if the oradism is owned by root and has setuid.
# ls -l $ORACLE_HOME/bin/oradism
-rwsr-x--- 1 root oinstall 71758 Sep 17 2011 /opt/app/oracle/product/12.1.0/std2/bin/oradism
If oradism has wrong permission or ownership correct it as follows
# chown root:oinstall $ORACLE_HOME/bin/oradism
# chmod 4755 $ORACLE_HOME/bin/oradism
Direct NFS searches for mount entries in following files in the given order
1.$ORACLE_HOME/dbs/oranfstab
2. /etc/oranfstab
3. /etc/mtab
In this case an oranfstab file is created as follows
cat oranfstab
server: hpc1nfsmount
local: 192.168.0.66
path: 192.168.0.104
export: /opt/backup mount: /usr/local/ochm/nfsdir
local: 192.168.0.66 is the IP of the local server where the database is running. path: 192.168.0.104 is the IP of the server which provides the NFS storage (i.e. NFS server). export: /opt/backup is the export location on the NFS server. mount: /usr/local/ochm/nfsdir is the mount point local server.

Once the dnfs is enabled start the database. If the dnfs is in use the alert log will have the following line
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0




However there won't be any dnfs related info available on the views
SQL> select * from v$dnfs_servers;

no rows selected

SQL> select * from v$dnfs_files;

no rows selected
Create a tablespace using the nfs location.
SQL> create tablespace nfstbs datafile '/usr/local/ochm/nfsdir/nfs.dbf'  size 10m;
When the tablespace is being created the alert log will show the dnfs being used
Fri Oct 09 11:16:01 2015
create tablespace nfstbs datafile '/usr/local/ochm/nfsdir/nfs.dbf' size 10m
Fri Oct 09 11:16:02 2015
Direct NFS: channel id [0] path [192.168.0.104] to filer [192.168.0.104] via local [] is UP
Completed: create tablespace nfstbs datafile '/usr/local/ochm/nfsdir/nfs.dbf' size 10m
Once the tablespace is created the dnfs views will show the relevant information
SQL> select * from v$dnfs_servers;

ID SVRNAME DIRNAME MNTPORT NFSPORT WTMAX RTMAX
---------- -------------------- ------------------------------ ---------- ---------- ---------- ----------
1 192.168.0.104 /opt/backup 56904 2049 1048576 1048576

SQL> select * from v$dnfs_files;

FILENAME FILESIZE PNUM SVR_ID
------------------------------ ---------- ---------- ----------
/usr/local/ochm/nfsdir/nfs.dbf 10493952 10 1
The same servers and setup was used to test the direct nfs on 11.1 SE as well. The dNFS ODM version is lower than 11.2 and 12.1
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0
The NFS version was v3
# nfsstat | grep nfs
Server nfs v3:
Client nfs v3:
However this did not succeed creating tablespaces in the 11.1 even after mounting the nfs with a lower version
192.168.0.104:/opt/backup on /usr/local/ochm/nfsdir type nfs (rw,hard,rsize=32768,wsize=32768,nfsvers=2,nointr,timeo=600,tcp,nolock,actimeo=0,addr=192.168.0.104)
Related Post
RMAN Backups on NFS

Changing The Cluster Name

$
0
0
This post shows the steps for renaming the cluster. The cluster consists of two nodes and current resources are
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CLUSFS.dg
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.DATA.dg
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.FRA.dg
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.asm
ONLINE ONLINE rhel12c1 Started,STABLE
ONLINE ONLINE rhel12c2 Started,STABLE
ora.net1.network
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.ons
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rhel12c2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rhel12c2 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rhel12c1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rhel12c1 169.254.75.103 192.1
68.1.87,STABLE
ora.cvu
1 ONLINE ONLINE rhel12c2 STABLE
ora.gns
1 ONLINE ONLINE rhel12c2 STABLE
ora.gns.vip
1 ONLINE ONLINE rhel12c2 STABLE
ora.mgmtdb
1 ONLINE ONLINE rhel12c1 Open,STABLE
ora.oc4j
1 ONLINE ONLINE rhel12c2 STABLE
ora.rhel12c1.vip
1 ONLINE ONLINE rhel12c1 STABLE
ora.rhel12c2.vip
1 ONLINE ONLINE rhel12c2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rhel12c2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rhel12c2 STABLE
ora.scan3.vip
1 ONLINE ONLINE rhel12c1 STABLE
ora.std12c1.db
1 ONLINE ONLINE rhel12c1 Open,STABLE
2 ONLINE ONLINE rhel12c2 Open,STABLE
--------------------------------------------------------------------------------
The cluster name is
$ cemutlo -n
rhel12c-cluster
This will be renamed to prod-cluster.
Renaming the cluster involves re-configuring the current cluster setup. To deconfigure the use rootcrs.pl -deconfig options. This is run on all remotes as root
[root@rhel12c2 grid]# $ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force -verbose
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
Network 1 exists
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node rhel12c1
VIP IPv4 Address: -/rhel12c1-vip/192.168.0.90
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
VIP exists: network number 1, hosting node rhel12c2
VIP IPv4 Address: -/rhel12c2-vip/192.168.0.91
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL false
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c2'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2015/10/30 14:05:37 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:06:12 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:06:13 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node
Once all the remote nodes are deconfigured, run the same with -lastnode option on the local node.
[root@rhel12c1 grid2]# $ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force -verbose -lastnode
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 14:07:10 CLSRSC-332: CRS resources for listeners are still configured

Network 1 exists
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node rhel12c1
VIP IPv4 Address: -/rhel12c1-vip/192.168.0.90
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL false
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c1'
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c1'
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c1'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
ASM de-configuration trace file location: /tmp/asmcadc_clean2015-10-30_02-15-19-PM.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2015-10-30_02-15-19-PM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2015/10/30 14:28:04 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:28:25 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:28:26 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node

2015/10/30 14:28:26 CLSRSC-559: Ensure that the GPnP profile data under the 'gpnp' directory in /opt/app/12.1.0/grid2 is deleted on each node before using the software in the current Grid Infrastructure home for reconfiguration.
To configure the cluster again with desired cluster name run the following command as grid user
$ $GI_HOME/crs/config/config.sh
This opens the OUI. Select configure cluster option.
As this is a 12c cluster setup it could be either setup as a flex cluster or standard cluster.
Specify the new cluster name
This OUI will detect the data base related disk groups (Data and Flash) but the ASM diskgroup that contained the ocr/vote disk and mgmtdb is no longer there.
Create a new diskgroup to store the ocr and vote disks.
At the last step select install to begin the configuration.
During the configuration (install) root.sh execution will be prompted. However execution of root.sh failed with the following error
# /opt/app/12.1.0/grid2/root.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /opt/app/12.1.0/grid2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 14:44:46 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:45:24 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:45:26 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2015/10/30 14:46:31 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded

Disk Group CLUSFS creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/oracleasm/ocr3' matches no disks
ORA-15014: path '/dev/oracleasm/ocr3' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr2' matches no disks
ORA-15014: path '/dev/oracleasm/ocr2' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr1' matches no disks
ORA-15014: path '/dev/oracleasm/ocr1' is not in the discovery set



2015/10/30 14:49:43 CLSRSC-184: Configuration of ASM failed

2015/10/30 14:49:45 CLSRSC-258: Failed to configure and start ASM

Died at /opt/app/12.1.0/grid2/crs/install/crsinstall.pm line 2017.
The command '/opt/app/12.1.0/grid2/perl/bin/perl -I/opt/app/12.1.0/grid2/perl/lib -I/opt/app/12.1.0/grid2/crs/install /opt/app/12.1.0/grid2/crs/install/rootcrs.pl ' execution failed
According to 431013.1 clearing the underlying disk would resolve this issue.
[root@rhel12c1 ~]# dd if=/dev/zero of=/dev/oracleasm/ocr3 bs=8192 count=1000
[root@rhel12c1 ~]# dd if=/dev/zero of=/dev/oracleasm/ocr2 bs=8192 count=1000
[root@rhel12c1 ~]# dd if=/dev/zero of=/dev/oracleasm/ocr1 bs=8192 count=1000
but this did not always work. The real reason seem to be the missing asm_diskstring. When the root.sh fail it leaves a empty asm instance (without any disk groups). Trying to create the same disk group fails with the same error
SQL> CREATE DISKGROUP CLUSFS NORMAL REDUNDANCY  DISK '/dev/oracleasm/ocr1',
'/dev/oracleasm/ocr2',
'/dev/oracleasm/ocr3' ATTRIBUTE 'compatible.asm'='12.1.0.0.0','au_size'='4M' 2 3 ;
CREATE DISKGROUP CLUSFS NORMAL REDUNDANCY DISK '/dev/oracleasm/ocr1',
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/oracleasm/ocr3' matches no disks
ORA-15014: path '/dev/oracleasm/ocr3' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr2' matches no disks
ORA-15014: path '/dev/oracleasm/ocr2' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr1' matches no disks
ORA-15014: path '/dev/oracleasm/ocr1' is not in the discovery set
The asm_diskstring is empty
SQL> show parameter string

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring string
However is asm_disktring was updated then diskgroup gets created without a problem
SQL>  alter system set asm_diskstring='/dev/oracleasm/*';

System altered.

SQL> CREATE DISKGROUP CLUSFS NORMAL REDUNDANCY DISK '/dev/oracleasm/ocr1',
'/dev/oracleasm/ocr2',
'/dev/oracleasm/ocr3' ATTRIBUTE 'compatible.asm'='12.1.0.0.0','au_size'='4M' 2 3 ;

Diskgroup created.


So it seems the missing asm_diskstring is the cause of the problem. To fix this the $GI_HOME/crs/install/crsconfig_params file was updated with the asm_diskstring
ASM_DISCOVERY_STRING=/dev/oracleasm/*
Afterwards the root.sh was rerun and there was no issue.
[root@rhel12c1 grid]# /opt/app/12.1.0/grid2/root.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /opt/app/12.1.0/grid2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 16:06:21 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:06:21 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:06:24 CLSRSC-363: User ignored prerequisites during installation

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.


CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
diskgroup CLUSFS not mounted ()
CRS-5017: The resource action "ora.storage start" encountered the following error:
Storage agent start action aborted. For details refer to "(:CLSN00107:)" in "/opt/app/oracle/diag/crs/rhel12c1/crs/trace/ohasd_orarootagent_root.trc".
CRS-2674: Start of 'ora.storage' on 'rhel12c1' failed
CRS-2679: Attempting to clean 'ora.storage' on 'rhel12c1'
CRS-2681: Clean of 'ora.storage' on 'rhel12c1' succeeded

ASM created and started successfully.

Disk Group CLUSFS created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 92718a80064a4fdebf30cdd4ed314115.
Successful addition of voting disk 48f9d1873a394f4dbf63ce93cf8b8b03.
Successful addition of voting disk a96f1382267b4fbdbf084b7595fd3643.
Successfully replaced voting disk group with +CLUSFS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 92718a80064a4fdebf30cdd4ed314115 (/dev/oracleasm/ocr1) [CLUSFS]
2. ONLINE 48f9d1873a394f4dbf63ce93cf8b8b03 (/dev/oracleasm/ocr2) [CLUSFS]
3. ONLINE a96f1382267b4fbdbf084b7595fd3643 (/dev/oracleasm/ocr3) [CLUSFS]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/10/30 16:33:07 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c1'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c1'
CRS-2676: Start of 'ora.gns' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CLUSFS.dg' is already running on 'rhel12c1'
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/10/30 16:38:34 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
On the last node
[root@rhel12c2 grid]# /opt/app/12.1.0/grid2/root.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /opt/app/12.1.0/grid2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 16:43:33 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:44:09 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:44:12 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2015/10/30 16:45:41 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/10/30 16:51:06 CLSRSC-343: Successfully started Oracle Clusterware stack

2015/10/30 16:51:28 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
At the end of the cluster configuration the cluster stack would be up and running
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.CLUSFS.dg
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.net1.network
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.ons
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rhel12c2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rhel12c1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rhel12c1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rhel12c1 169.254.229.238 192.
168.1.87,STABLE
ora.asm
1 ONLINE ONLINE rhel12c1 Started,STABLE
2 ONLINE ONLINE rhel12c2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE rhel12c1 STABLE
ora.gns
1 ONLINE ONLINE rhel12c1 STABLE
ora.gns.vip
1 ONLINE ONLINE rhel12c1 STABLE
ora.mgmtdb
1 ONLINE ONLINE rhel12c1 Open,STABLE
ora.oc4j
1 ONLINE ONLINE rhel12c1 STABLE
ora.rhel12c1.vip
1 ONLINE ONLINE rhel12c1 STABLE
ora.rhel12c2.vip
1 ONLINE ONLINE rhel12c2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rhel12c2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rhel12c1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rhel12c1 STABLE
--------------------------------------------------------------------------------
with the new cluster name
 cemutlo -n
prod-cluster
One thing notable is that database related asm disk groups and database itself are not part of the cluster yet. For the cluster setup to be same as before the rename the database must be manually added to the cluster. Mount the asm disk groups data and fra
SQL> alter diskgroup fra mount;
Diskgroup altered.

SQL> alter diskgroup data mount ;
Diskgroup altered.

SQL> select inst_id,name,state from gv$asm_diskgroup;

INST_ID NAME STATE
---------- ------------------------------ -----------
2 CLUSFS MOUNTED
2 FRA MOUNTED
2 DATA MOUNTED
1 CLUSFS MOUNTED
1 FRA MOUNTED
1 DATA MOUNTED
As oracle user add the database to the cluster
[oracle@rhel12c1 ~]$ srvctl add database -db std12c1 -oraclehome $ORACLE_HOME -spfile "+DATA/STD12C1/PARAMETERFILE/spfilestd12c1.ora"
[oracle@rhel12c1 ~]$ srvctl add instance -db std12c1 -instance std12c11 -n rhel12c1
[oracle@rhel12c1 ~]$ srvctl add instance -db std12c1 -instance std12c12 -n rhel12c2
[oracle@rhel12c1 ~]$ srvctl start database -d std12c1
Database configuration is added to the cluster
$ srvctl config database -d std12c1
Database unique name: std12c1
Database name:
Oracle home: /opt/app/oracle/product/12.1.0/dbhome_2
Oracle user: oracle
Spfile: +DATA/STD12C1/PARAMETERFILE/spfilestd12c1.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,FRA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: std12c11,std12c12
Configured nodes: rhel12c1,rhel12c2
Database is administrator managed
Number of cluster resources are same as before the rename
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.CLUSFS.dg
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.DATA.dg
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.FRA.dg
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.net1.network
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
ora.ons
ONLINE ONLINE rhel12c1 STABLE
ONLINE ONLINE rhel12c2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rhel12c2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rhel12c1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rhel12c1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rhel12c1 169.254.229.238 192.
168.1.87,STABLE
ora.asm
1 ONLINE ONLINE rhel12c1 Started,STABLE
2 ONLINE ONLINE rhel12c2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE rhel12c1 STABLE
ora.gns
1 ONLINE ONLINE rhel12c1 STABLE
ora.gns.vip
1 ONLINE ONLINE rhel12c1 STABLE
ora.mgmtdb
1 ONLINE ONLINE rhel12c1 Open,STABLE
ora.oc4j
1 ONLINE ONLINE rhel12c1 STABLE
ora.rhel12c1.vip
1 ONLINE ONLINE rhel12c1 STABLE
ora.rhel12c2.vip
1 ONLINE ONLINE rhel12c2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rhel12c2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rhel12c1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rhel12c1 STABLE
ora.std12c1.db
1 ONLINE ONLINE rhel12c1 Open,STABLE
2 ONLINE ONLINE rhel12c2 Open,STABLE
--------------------------------------------------------------------------------
This concludes the renaming of the cluster.

Useful metalink notes
How to Configure or Re-configure Grid Infrastructure With config.sh/config.bat [ID 1354258.1]
How to change the cluster name in RAC [ID 1967916.1]

For input string: SP2-0640: Not connected

$
0
0
Upgrade of a 11.1.0.7 instance to 11.2.0.4 fails with the following error message on DBUA.
Following errors could be seen when DBUA trace.log is examined.
[Thread-12] [ 2016-01-15 11:43:16.187 GMT ] [OracleHome.initOptions:1256]  ORA-27102: out of memory

oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-27102: out of memory

at oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1658)
at oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeSql(SQLEngine.java:1925)
at oracle.sysman.assistants.util.OracleHome.initOptions(OracleHome.java:1249)
at oracle.sysman.assistants.dbma.backend.CompManager.doPreMigrationChecks(CompManager.java:2955)
at oracle.sysman.assistants.dbma.backend.CompManager.obtainDatabaseInformation(CompManager.java:4269)
at oracle.sysman.assistants.dbma.ui.DatabasesPage.doProcessing(DatabasesPage.java:811)
at oracle.sysman.assistants.util.WaitDialog.run(WaitDialog.java:187)
at java.lang.Thread.run(Thread.java:637)
..
[Thread-12] [ 2016-01-15 11:43:16.260 GMT ] [OracleHome.initOptions:1324] Database Options queried: 1
java.lang.NumberFormatException: For input string: "SP2-0640: Not connected"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:447)
at java.lang.Integer.parseInt(Integer.java:497)
at oracle.sysman.assistants.dbma.backend.CompManager.doPreMigrationChecks(CompManager.java:2956)
at oracle.sysman.assistants.dbma.backend.CompManager.obtainDatabaseInformation(CompManager.java:4269)
at oracle.sysman.assistants.dbma.ui.DatabasesPage.doProcessing(DatabasesPage.java:811)
at oracle.sysman.assistants.util.WaitDialog.run(WaitDialog.java:187)
at java.lang.Thread.run(Thread.java:637)
Even though the latter stack trace matches the error output (For input string: "SP2-0640: Not connected") seen on DBUA it's not the root cause but the ORA-27102: out of memory. Key to solving the issue is to identify which instance runs out of the memory.
The current shm* parameters are
kernel.shmmax = 5370806272
kernel.shmall = 1311236
kernel.shmmni = 4096

ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 5244928
max total shared memory (kbytes) = 5244944
min seg size (bytes) = 1
And the database is using a one segment that's equal to the SGA size
ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x12982ecc 3112962 oracle 660 5370806272 36

SQL> show parameter sga

NAME TYPE VALUE
------------------------------------ --------------- ----------
sga_max_size big integer 5G
sga_target big integer 5G
As it is configured now there's sufficient shared segment to run the database. However it is not the database that's been upgraded that runs out of memory but a dummy database instance created by DBUA for a short period during information gathering phase. Information on this dummy instance is also logged on the trace.log
[OracleHome.initOptions:1248]  executing: startup nomount pfile='/opt/app/oracle/product/11.2.0/std3/dbs/initDBUA4304056.ora'
When the DBUA succeed all the files created for this dummy instance are deleted and there's no trace of this dummy instance whatsoever but if it fails the init file used will remain and could be examined.
cd /opt/app/oracle/product/11.2.0/std3/dbs/  
-rw-r----- 1 oracle oinstall 1536 Jan 15 11:43 orapwDBUA4304056
-rw-r----- 1 oracle oinstall 161 Jan 15 11:43 initDBUA4304056.ora
Looking at the init file the log directory could be identified.
more initDBUA4304056.ora
db_name=DBUA4304
db_unique_name=DBUA4304056
shared_pool_size=128m
cpu_count=1
diagnostic_dest=/opt/app/oracle/product/11.2.0/std3/log
_enable_NUMA_support=false
Within the log directory will be a diagnostic directory structure which will have the alert log and the trace file of the failed dummy instance
cd /opt/app/oracle/product/11.2.0/std3/log
cd diag/rdbms/dbua4304056/DBUA4304056/trace/
ls
alert_DBUA4304056.log DBUA4304056_ora_6420.trc
These files will have content similar to following
more DBUA4304056_ora_6420.trc

Switching to regular size pages for segment size 8388608
skgm warning: ENOSPC creating segment of size 0000000000800000
fix shm parameters in /etc/system or equivalent

more alert_DBUA4304056.log
Fri Jan 15 11:43:15 2016
The value of parameter shared_pool_size is below the required minimum
It has been reset to the minimum value
Starting ORACLE instance (normal)


As seen on the trace file, the dummy segment is trying acquire a segment of size 8388608. But for the DBUA to be successful it need to create at least 2 other segments. Following segment list captured during a successful DBUA run
ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x12982ecc 3112962 oracle 660 5370806272 37 <-- DB for upgrade
0x00000000 4423684 oracle 777 13160 2 dest <-- DBUA itself
0x00000000 4882437 oracle 640 8388608 1 <-- Dummy instance
0x00000000 4915206 oracle 640 230686720 1 <-- Dummy instance
0xbcf2a6a8 4947975 oracle 640 2097152 1 <-- Dummy instance
Dummy instance trace file show this information
*** 2016-01-15 12:10:27.997
Switching to regular size pages for segment size 8388608
Switching to regular size pages for segment size 230686720
Switching to regular size pages for segment size 2097152

CELL communication is configured to use 0 interface(s):
kcrfwy: minimum sleep is 1922 usecs (overhead is 922 usecs)
Running with 1 strand for Non-Enterprise Edition
Running without dynamic strands for Non-Enterprise Edition
kgfmIsAppliance=0 (due to init)
The dummy instance's alert log shows the SGA size of it
****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 0 KB (0%)

Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB

RECOMMENDATION:
Total Shared Global Region size is 230 MB. For optimal performance,
prior to the next instance restart increase the number
of unused Large Pages by atleast 115 2048 KB Large Pages (230 MB)
system wide to get 100% of the Shared
Global Region allocated with Large pages
***********************************************************
...
Starting up:
Oracle Database 11g Release 11.2.0.3.0 - 64bit Production.
ORACLE_HOME = /opt/app/oracle/product/11.2.0/std3
Using parameter settings in client-side pfile /opt/app/oracle/product/11.2.0/std3/dbs/initDBUA0520596.ora on machine hpc1.domain.net
System parameters with non-default values:
cpu_count = 1
_enable_NUMA_support = FALSE
shared_pool_size = 148M
db_name = "DBUA0520"
db_unique_name = "DBUA0520596"
diagnostic_dest = "/opt/app/oracle/product/11.2.0/std3/log"
With the increase demand for the shared memory segments the minimum kernel.shmall value must be at least 1370116 (theoretical value. The sum of all the segments size listed above / PAGE_SIZE). However it should be slightly larger than that in practice, in this case kernel.shmall had to be set for 1370123 before DBUA worked.
Point to remember is that if DBUA fails with above error then it's not the database being upgraded that ran out of memory. It is also good idea to set kernel.shmall to a higher value than SGA size.

Useful metalink notes
11.2.0.3 DBUA Fails : ORA-27102: Out Of Memory, "SP2-0640: Not Connected"[ID 1526424.1]
DBUA Fails With Error "SP2-0640 Not Connected"[ID 1384814.1]
Upon startup of Linux database get ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device [ID 301830.1]

Installing 11.2.0.4 Standalone Server with ASM and Role Separation on RHEL 7

$
0
0
There are two earlier posts which shows the highlights of installing 11gR2 on standalone server with ASM and role separation on RHEL 6 and RHEL 5. This posts only mention the steps that are different from the earlier setups, it's not an extensive how to install guide.
One of the main issues that existed was the failure of root.sh on RHEL 7. This was due to the fact that startup scripts on RHEL 7 are created as service while root.sh was trying to use the pre-RHEL7 method of inittab entries
# /opt/app/oracle/product/11.2.0/grid/root.sh
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel7 successfully pinned.
Adding Clusterware entries to inittab
ohasd failed to start

Failed to start the Clusterware. Last 20 lines of the alert log follow:
2015-05-27 14:00:52.832:
[client(23215)]CRS-2101:The OLR was formatted using version 3.
2015-05-27 14:00:53.700:
[client(23242)]CRS-1001:The OCR was formatted using version 3.

ohasd failed to start at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 377, line 4.
/opt/app/oracle/product/11.2.0/grid/perl/bin/perl -I/opt/app/oracle/product/11.2.0/grid/perl/lib -I/opt/app/oracle/product/11.2.0/grid/crs/install /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl execution failed
Workaround was to manually create a service file and later on Oracle also created a MOS on this (1959008.1). This has now been fixed.
It is assumed that all necessary pre-reqs are completed and users have been created for role separate installation similar to RHEL 6 setup. The software used is the 13390677 patchset (11.2.0.4 software).
1. As grid user unzip the grid infrastructure software (p13390677_112040_Linux-x86-64_3of7.zip). Before the installation begins apply the patch 19404309. This is not a typical patch apply with OPatch, but a copying of files that eliminate the issues associated with installing 11.2.0.4 on RHEL 7.
2. Run cluster verify tool for high availability service option. The failure of pdksh could be ignored (refer 1962046.1). This invalidation is also visible on the OUI and could also be ignored similar to 11.2.0.3 on RHEL 6.
./runcluvfy.sh  stage -pre hacfg

Performing pre-checks for Oracle Restart configuration
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "rhel7:/tmp"
Check for multiple users with UID value 1002 passed
User existence check passed for "grid"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "grid" in group "oinstall" [as Primary] passed
Membership check for user "grid" in group "dba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "make"
Package existence check passed for "binutils"
Package existence check passed for "gcc(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "elfutils-libelf(x86_64)"
Package existence check passed for "elfutils-libelf-devel"
Package existence check passed for "glibc-common"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "glibc-headers"
Package existence check passed for "gcc-c++(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check failed for "pdksh"
Check failed on nodes:
rhel7
Package existence check passed for "expat(x86_64)"
Check for multiple users with UID value 0 passed
Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed

Pre-check for Oracle Restart configuration was unsuccessful.
3. Run the installer and proceed as before. However do not run the root.sh script when prompted.
Patch 18370031 must be applied before root.sh could be run. Refer the patch's readme on how to apply it before GI home is configured
$GI_HOME/OPatch/opatch napply -oh $GI_HOME -local ./18370031
Once the patch is applied run the root.sh. With the patch in place the root.sh correctly configures a ohas service on RHEL 7. There's no need to manually create a service file anymore (steps in 1959008.1 not needed).
# /opt/app/oracle/product/11.2.0/grid/root.sh

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel7 successfully pinned.
Adding Clusterware entries to oracle-ohasd.service

rhel7 2016/01/26 13:49:04 /opt/app/oracle/product/11.2.0/grid/cdata/rhel7/backup_20160126_134904.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server


4. Once root.sh has finished continue with the rest of the installation to completion.
crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE rhel7
ora.FRA.dg
ONLINE ONLINE rhel7
ora.LISTENER.lsnr
ONLINE ONLINE rhel7
ora.asm
ONLINE ONLINE rhel7 Started
ora.ons
OFFLINE OFFLINE rhel7
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE rhel7
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE rhel7
5. To install database software extract the relevant zip files (p13390677_112040_Linux-x86-64_1of7.zip and p13390677_112040_Linux-x86-64_2of7.zip). Similar to GI installer, before installation could begin apply the patch 19404309. Once the patch is applied run the installer.
6. During the installation the following error could be observed, which was mentioned in an earlier post.
Ignore the error and continue the installation to completion. Once database software is installed apply the patch 19692824 before database is created.
7. As the last step create the database. There are no errors or issues related to RHEL 7 during the database creation using DBCA.
8. If a GI patch is applied using opatch auto option it would fail with the following
$ORACLE_HOME/OPatch/opatch auto ./21523375 -ocmrf ../ocm.rsp
Can't locate Switch.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /opt/app/oracle/product/11.2.0/grid/OPatch/crs/auto_patch.pl line 2730.
Reason is RHEL 7 has a newer version of perl
 perl -v

This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi
(with 25 registered patches, see perl -V for more detail)
where switch module is deprecated. Patch could be applied manually or download and install the switch.pm module from cpan and try opatch auto again. Refer 1915430.1 for more.
cd Switch-2.17
perl Makefile.PL
make test
make install
Manifying blib/man3/Switch.3pm
Installing /usr/local/share/perl5/Switch.pm
Installing /usr/local/share/man/man3/Switch.3pm
Appending installation info to /usr/lib64/perl5/perllocal.pod
Afterwards the opatch auto continues without any issues.

Useful metalink notes
Opatch Auto fails with: Can't locate Switch.pm in @INC [ID 1915430.1]
Installation walkthrough - Oracle Grid/RAC 11.2.0.4 on Oracle Linux 7 [ID 1951613.1]
Install of Clusterware fails while running root.sh on OL7 - ohasd fails to start [ID 1959008.1]
Requirements for Installing Oracle 11.2.0.4 RDBMS on RHEL7 or OL7 64-bit (x86-64) [ID 1962100.1]
Missing pdksh-5.2.14 package during Oracle database 11.2.0.4 install on Oracle Linux 7 [ID 1962046.1]
Installation of Oracle 11.2.0.4 on OL7 fails with “undefined reference to symbol ‘B_DestroyKeyObject’” error [ID 1965691.1]

Related Posts
ASM for Standalone Server in 11gR2 with Role Separation (on RHEL 5)
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11.2.0.3 on RHEL 6
ins_emagent.mk Related Error When Installing 11.2.0.4 Database on RHEL 7
Installing Oracle Database 12.1.0.2 on RHEL 7

Restoring OCR and Vote Disks on Block Devices in a 11gR2 Cluster (11.2.0.4)

$
0
0
Using block or raw devices for OCR and vote disks is not supported for new installation of 11.2. However it is supported for upgraded systems. Following is from the Oracle clusterware admin guide "Oracle Universal Installer for Oracle Clusterware 11g release 2 (11.2), does not support the use of raw or block devices. However, if you upgrade from a previous Oracle Clusterware release, then you can continue to use raw or block devices. Oracle recommends that you use Oracle ASM to store OCR and voting disks." On 12c block devices are not supported at all and ocr and vote disks must be moved to ASM before the upgrade.
However if chose to the ocr and vote disks can remain in block devices such as after upgrade from 11.1.0.7 to 11.2.0.4 (or 11.2.0.3). In 10g and 11gR1 clusters the vote disk was backed up using dd command. However this is not supported on 11.2. Following is from the clusterware admin guide "The dd commands used to back up and recover voting disks in previous versions of Oracle Clusterware are not supported in Oracle Clusterware 11g release 2 (11.2). Restoring voting disks that were copied using dd or cp commands can prevent the Oracle Clusterware 11g release 2 (11.2) stack from coming up." On 11.2 vote disks are not needed to be backed up separately. From clusterware admin "In Oracle Clusterware 11g release 2 (11.2), you no longer have to back up the voting disk. The voting disk data is automatically backed up in OCR as part of any configuration change and is automatically restored to any voting disk added."
So only the OCR is needed to be backed up and this could be used to restore both ocr and vote disk. This post shows steps for restoring OCR and vote disks that are stored on block devices (after all of the ocr and vote disks have failed or corrupted). The environment used for this is a 11.2.0.4 two node cluster which was previously upgraded from 11.1.0.7.
Two OCR files are
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 292924
Used space (kbytes) : 6452
Available space (kbytes) : 286472
ID : 675013742
Device/File Name : /dev/sdb1
Device/File integrity check succeeded
Device/File Name : /dev/sde1
Device/File integrity check succeeded
Cluster registry integrity check succeeded

Logical corruption check succeeded
and the vote disks
##  STATE    File Universal Id                File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 6cc871618b1e5f9fbf795315f36b3c21 (/dev/sdh1) []
2. ONLINE 8f988bcfda0b5fe0bf783f44e81d3f66 (/dev/sdf1) []
3. ONLINE 6af6320f0979dff7bf6fa474cea2765a (/dev/sdg1) []
OCR and vote disks all were corrupted with dd command
 for i in /dev/sdb1 /dev/sde1 /dev/sdh1 /dev/sdg1 /dev/sdf1
> do
> dd if=/dev/zero of=$i bs=8192 count=1000
> done
This results in OCR and vote disks becoming unusable. Below output shows vote disk state pending offline
crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. PENDOFFL 6cc871618b1e5f9fbf795315f36b3c21 (/dev/sdh1) []
2. PENDOFFL 8f988bcfda0b5fe0bf783f44e81d3f66 (/dev/sdf1) []
3. PENDOFFL 6af6320f0979dff7bf6fa474cea2765a (/dev/sdg1) []
Following message could be observed on ocssd.log
2016-01-15 17:08:16.921: [    CSSD][3422501184]clssnmvVoteDiskValidation: Voting disk /dev/sdh1 is corrupted
2016-01-15 17:08:16.921: [ CSSD][3422501184]clssnmvWorkerThread: disk /dev/sdh1 corrupted
2016-01-15 17:08:16.921: [ CSSD][3422501184]clssnmvDiskAvailabilityChange: voting file /dev/sdh1 now offline
2016-01-15 17:08:17.405: [ SKGFD][3419347264]Lib :UFS:: closing handle 0x85937a0 for disk :/dev/sdh1:

2016-01-15 17:08:20.746: [ CSSD][3412990272]clssnmvVoteDiskValidation: Voting disk /dev/sdg1 is corrupted
2016-01-15 17:08:20.746: [ CSSD][3412990272]clssnmvWorkerThread: disk /dev/sdg1 corrupted
2016-01-15 17:08:20.746: [ CSSD][3412990272]clssnmvDiskAvailabilityChange: voting file /dev/sdg1 now offline
2016-01-15 17:08:20.901: [ SKGFD][3422501184]Lib :UFS:: closing handle 0x86afd90 for disk :/dev/sdh1:
Eventually the two nodes got rebooted after which the OCR and vote disk restore process began.
Stop clusterware on all nodes with -f option.
# crsctl stop crs -f
At times this could take a while. Quickest option was to disable crs and reboot the nodes. Once nodes starts enable crs again but do not start the cluster stack.



When clusterware stack is down on all nodes, start it only on a single node with exclusive and nocrs options.
crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
If crsd process is running stop it.
crsctl stop resource ora.crsd -init
Restore the OCR with a backup
#ocrconfig -restore /opt/app/11.2.0/grid/cdata/cg_11g_cluster/backup_20160115_170721.ocr
# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 292924
Used space (kbytes) : 6452
Available space (kbytes) : 286472
ID : 675013742
Device/File Name : /dev/sdb1
Device/File integrity check succeeded
Device/File Name : /dev/sde1
Device/File integrity check succeeded

Cluster registry integrity check succeeded

Logical corruption check succeeded
As seen from the ocrcheck output the ocr files were restored to the original location listed on the ocr.loc file.
At this stage the vote disks are still not listed.
crsctl query css votedisk
Located 0 voting disk(s).
To restore the vote disks run add votedisk specifying the block device locations (replace votedisk is not applicable for non-ASM locations).
# crsctl add css  votedisk /dev/sdh1 /dev/sdg1 /dev/sdf1
Now formatting voting disk: /dev/sdh1.
Now formatting voting disk: /dev/sdg1.
Now formatting voting disk: /dev/sdf1.
CRS-4603: Successful addition of voting disk /dev/sdh1.
CRS-4603: Successful addition of voting disk /dev/sdg1.
CRS-4603: Successful addition of voting disk /dev/sdf1.
[root@rac1 oracle]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 274a68a133e84f65bf3b7b7e966f3862 (/dev/sdh1) []
2. ONLINE b0615b1176ef4f12bfdd34c115620249 (/dev/sdg1) []
3. ONLINE b1eb954b7a454f32bf0c54f252776c1d (/dev/sdf1) []
Stop the clusterware stack on this node
 crsctl stop crs -f
and start the clusteware stack on all nodes
crsctl start crs -nowait
This concludes steps for restoring OCR and vote disk on a 11.2.0.4 cluster when they are stored on block devices.
Useful Metalink noteHow to restore ASM based OCR after complete loss of the CRS diskgroup on Linux/Unix systems [ID 1062983.1]Bug 12543757 : UNABLE TO MOVE VOTING DISK FROM RAW DEVICES TO NFSRelated Posts
Restoring OCR due to ASM disk failures - 1Restoring OCR due to ASM disk failures - 2Restoring Vote disk due to ASM disk failures - 1Restoring Vote disk due to ASM disk failures - 2Restoring Vote disk due to ASM disk failures - 3Restoring OCR & Vote disk due to ASM disk failures - 1Restoring OCR & Vote disk due to ASM disk failures - 2Restoring OCR & Vote disk due to ASM disk failures - 3

Changing Hostname in a Standalone DB Configuration with ASM

$
0
0
Grid infrastructure software is required to setup ASM in a standalone DB configuration. GI has hostname dependent directory paths and components. Changing the hostname in a such configuration requires re-configuring the GI software after the hostname has been changed. This post list the steps for changing the hostname in a standalone DB configuration with ASM.
The current hostname is rhel7.domain.net and this will changed to rhel7s.domain.net
In certain systems (in this case OS RHEL 7.0, kernel 3.10.0-123.el7.x86_64, Oracle 11.2.0.4) may run in to following error due to perl version related issues. Solution for this is to use the perl binary provided with GI.
$GI_HOME/crs/install/roothas.pl -deconfig -force
Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/app/oracle/product/11.2.0/grid/crs/install) at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
Compilation failed in require at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
The MOS notes mention to change the hostname and then de-configure but this fails with the following error
$GI_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -deconfig -force
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Can't open /etc/oracle/scls_scr/rhel7s/grid/ohasdstr for write: No such file or directory at /opt/app/oracle/product/11.2.0/grid/crs/install/s_crsconfig_lib.pm line 1386.
Reason is that deconfig option is looking for a directory path that does not exists
# ls /etc/oracle/scls_scr/
rhel7
It maybe possible to rename this directory and run the deconfig. But instead the deconfig was run (as root) without changing the hostname (reverting to original hostname).
# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -deconfig -force
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
CRS resources for listeners are still configured
PRKO-2573 : ONS daemon is already stopped.
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel7'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel7'
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel7'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel7' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel7' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel7' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel7'
CRS-2677: Stop of 'ora.asm' on 'rhel7' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel7'
CRS-2677: Stop of 'ora.cssd' on 'rhel7' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel7'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel7'
CRS-2677: Stop of 'ora.evmd' on 'rhel7' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel7' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle Restart stack
Once the deconfig is complete change the hostname
hostname
rhel7s.domain.net
Run (as root) the roothas.pl script to configure the HAS again.
# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel7s successfully pinned.
Adding Clusterware entries to oracle-ohasd.service

rhel7s 2016/02/12 13:10:13 /opt/app/oracle/product/11.2.0/grid/cdata/rhel7s/backup_20160212_131013.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
At this stage the services would be in the following states
Resource Name                       Type                           Target             State              Host
------------- ------ ------- -------- ----------
ora.cssd ora.cssd.type OFFLINE OFFLINE
ora.diskmon ora.diskmon.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE rhel7s
ora.ons ora.ons.type OFFLINE OFFLINE
As grid user, enable the cssd auto start and stop and start the HAS again
$ crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
$ crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel7s'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel7s'
CRS-2677: Stop of 'ora.evmd' on 'rhel7s' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel7s' has completed
CRS-4133: Oracle High Availability Services has been stopped.
$ crsctl start has
CRS-4123: Oracle High Availability Services has been started.
Check the states of the services and HAS service
Resource Name                       Type                           Target             State              Host
------------- ------ ------- -------- ----------
ora.cssd ora.cssd.type ONLINE ONLINE rhel7s
ora.diskmon ora.diskmon.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE rhel7s
ora.ons ora.ons.type OFFLINE OFFLINE

$ crsctl check has
CRS-4638: Oracle High Availability Services is online
GI has its services but listener, ASM and DB are missing. These need to be manually added to the configuration. First add and start the listener
$ srvctl add listener
$ srvctl start listener

Resource Name Type Target State Host
------------- ------ ------- -------- ----------
ora.LISTENER.lsnr ora.listener.type ONLINE ONLINE rhel7s
ora.cssd ora.cssd.type ONLINE ONLINE rhel7s
ora.diskmon ora.diskmon.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE rhel7s
ora.ons ora.ons.type OFFLINE OFFLINE


Add the ASM and set auto start option to 1. In this case the ASM Spfile was not residing in a ASM diskgroup.
$ srvctl add asm
$ crsctl modify resource "ora.asm" -attr "AUTO_START=1"
$ srvctl modify asm -p /opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora -l listener

$ srvctl config asm
ASM home: /opt/app/oracle/product/11.2.0/grid
ASM listener: LISTENER
Spfile: /opt/app/oracle/product/11.2.0/grid/dbs/spfile+ASM.ora
ASM diskgroup discovery string: ++no-value-at-resource-creation--never-updated-through-ASM++

$ crsctl stop has
$ crsctl start has

Resource Name Type Target State Host
------------- ------ ------- -------- ----------
ora.DATA.dg ora.diskgroup.type ONLINE ONLINE rhel7s
ora.FRA.dg ora.diskgroup.type ONLINE ONLINE rhel7s
ora.LISTENER.lsnr ora.listener.type ONLINE ONLINE rhel7s
ora.asm ora.asm.type ONLINE ONLINE rhel7s
ora.cssd ora.cssd.type ONLINE ONLINE rhel7s
ora.diskmon ora.diskmon.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE rhel7s
ora.ons ora.ons.type OFFLINE OFFLINE
Finally add the database and start it
$ srvctl add database -d std11g2 -o $ORACLE_HOME -p +DATA/std11g2/spfilestd11g2.ora
$ srvctl config database -d std11g2
Database unique name: std11g2
Database name:
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_4
Oracle user: oracle
Spfile: +DATA/std11g2/spfilestd11g2.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: std11g2
Disk Groups:
Services:
$ srvctl start database -d std11g2

Resource Name Type Target State Host
------------- ------ ------- -------- ----------
ora.DATA.dg ora.diskgroup.type ONLINE ONLINE rhel7s
ora.FRA.dg ora.diskgroup.type ONLINE ONLINE rhel7s
ora.LISTENER.lsnr ora.listener.type ONLINE ONLINE rhel7s
ora.asm ora.asm.type ONLINE ONLINE rhel7s
ora.cssd ora.cssd.type ONLINE ONLINE rhel7s
ora.diskmon ora.diskmon.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE rhel7s
ora.ons ora.ons.type OFFLINE OFFLINE
ora.std11g2.db ora.database.type ONLINE ONLINE rhel7s
This conclude the changing of hostname in a standalone DB configuration with ASM.

Useful metalink notes
How to Reconfigure Oracle Restart on 12c / 12.1 [ID 1570358.1]
rootcrs.pl/roothas.pl Fails With Can't locate Env.pm [ID 2019784.1]
rootcrs.pl/roothas.pl Fails With "Can't locate Env.pm"[ID 1925577.1]
How to change Hostname / IP for a Grid Infrastructure Oracle Restart Standalone Configuration (SIHA) [ID 1552810.1]

Related Posts
Changing The Cluster Name

Opatch util Cleanup and .patch_storage

$
0
0
During a recent PSU apply it was noticed that grid home has grown to a considerable size.
du -sh grid4
16G grid4
Out of the 16GB majority of space was consumed by the .patch_storage directory
du -sh grid4/.patch_storage
9.4G grid4/.patch_storage
.patch_storage could be described loosely as the directory where opatch copies backup files during patch application. In case of rollback backup files from .patch_storage are copied to their original location.
In earlier versions (10g, 11gR1) optach util cleanup would remove older backup that are not needed for rollback and reduce the size of the .patch_storage. However in this case running this command didn't result in any reduction of the .patch_storage.
opatch util cleanup
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.


Oracle Home : /opt/app/11.2.0/grid4
Central Inventory : /opt/app/oraInventory
from : /opt/app/11.2.0/grid4/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.4.0
Log file location : /opt/app/11.2.0/grid4/cfgtoollogs/opatch/opatch2016-03-09_15-20-16PM_1.log

Invoking utility "cleanup"
OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
You will be still able to rollback patches after this cleanup.
Do you want to proceed? [y|n]
y
User Responded with: Y
Size of directory "/opt/app/11.2.0/grid4/.patch_storage"before cleanup is 10031648985 bytes.
Size of directory "/opt/app/11.2.0/grid4/.patch_storage"after cleanup is 10031648985 bytes.

UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
deleted, Please refer log file.

OPatch succeeded.
Looking inside the .patch_storage it could be seen that some of the directories and files were created as a result of earlier PSU application far back as two years ago. The list below shows the size of the sub directories inside .patch_storage and the patch that resulted in creating this backup directory. Read 1641136.1 for GI PSU supplemental read which has all the patches released for GI 11.2.0.4
14M     17478514_Dec_6_2013_04_22_19  -  Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
33M 18031668_Feb_20_2014_05_15_58 - Sub-patch 18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
791M 18031731_Mar_17_2014_06_35_22 - Patch description: "ACFS Patch Set Update : 11.2.0.4.2 (18031731)"
555M 18031740_Mar_19_2014_09_06_37 - Patch description: "OCW Patch Set Update : 11.2.0.4.2 (18031740)"
28M 18522509_Jun_30_2014_08_14_42 - Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
791M 18522514_May_6_2014_01_05_19 - Patch description: "ACFS Patch Set Update : 11.2.0.4.3 (18522514)"
555M 18522515_Jun_16_2014_08_05_14 - Patch description: "OCW Patch Set Update : 11.2.0.4.3 (18522515)"
555M 19121549_Oct_6_2014_03_27_01 - Patch description: "OCW Patch Set Update : 11.2.0.4.4 (19121549)"
30M 19121551_Oct_6_2014_10_07_57 - Sub-patch 19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
791M 19121552_Oct_6_2014_03_48_39 - Patch description: "ACFS Patch Set Update : 11.2.0.4.4 (19121552)"
791M 19769469_Nov_13_2014_04_37_23 - Patch description: "ACFS Patch Set Update : 11.2.0.4.5 (19769469)"
555M 19769476_Dec_3_2014_02_08_05 - Patch description: "OCW Patch Set Update : 11.2.0.4.5 (19769476)"
33M 19769489_Dec_28_2014_21_22_44 - Sub-patch 19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
1.7M 19852360_Oct_20_2014_08_17_43 - Patch 19852360 : applied on Thu Jan 22 17:08:25 GMT 2015
40M 20299013_Mar_4_2015_02_27_44 - Sub-patch 20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
791M 20299019_Mar_27_2015_15_26_30 - Patch description: "ACFS Patch Set Update : 11.2.0.4.6/7 (20299019)"
4.0M 20760982_Jun_4_2015_00_23_20 - Sub-patch 20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
548M 20831122_Jul_1_2015_06_26_45 - Patch description: "OCW Patch Set Update : 11.2.0.4.7 (20831122)"
796K 21352635_Sep_1_2015_07_49_44 - Patch description: "Database Patch Set Update : 11.2.0.4.8 (21352635)"
791M 21352642_Sep_3_2015_00_03_11 - Patch description: "ACFS Patch Set Update : 11.2.0.4.8 (21352642)"
548M 21352649_Sep_2_2015_23_43_49 - Patch description: "OCW Patch Set Update : 11.2.0.4.8 (21352649)"
17M 21948347_Dec_14_2015_03_31_48 - Patch description: "Database Patch Set Update : 11.2.0.4.160119 (21948347)"
548M 21948348_Dec_13_2015_23_42_28 - Patch description: "OCW Patch Set Update : 11.2.0.4.160119 (21948348)"
791M 21948355_Nov_18_2015_00_55_35 - Patch description: "ACFS Patch Set Update : 11.2.0.4.160119 (21948355)"
From this it seems .patch_storage always accumulate new backups and never expire any. When asked Oracle via a SR it was confirmed that due to new composite patching mechanism introduced in 11.2, .patch_storage will keep on growing as some of the rollback files would have to come from earlier backups. Therefore it is not possible to delete any files in .patch_storage unless rollback is not expected (and even then it's not advised. read 403218.1).


It seems only time that opatch util Cleanup does any cleaning is during patch application itself. Following from the opatch log
[Mar 9, 2016 3:02:35 PM]     OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
You will be still able to rollback patches after this cleanup.
Do you want to proceed? [y|n]
[Mar 9, 2016 3:02:38 PM] Y (auto-answered by -silent)
[Mar 9, 2016 3:02:38 PM] User Responded with: Y
[Mar 9, 2016 3:02:39 PM] Size of directory "/opt/app/11.2.0/grid4/.patch_storage"before cleanup is 10243859708 bytes.
[Mar 9, 2016 3:02:39 PM] Deleting the directory "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/backup"
...
[Mar 9, 2016 3:02:39 PM] Deleted the file "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/rac/mode.txt"
[Mar 9, 2016 3:02:39 PM] Deleted the file "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/rac/make_cmds.txt"
[Mar 9, 2016 3:02:39 PM] Deleted the file "/opt/app/11.2.0/grid4/.patch_storage/21948348_Dec_13_2015_23_42_28/rac/remote_cmds_21948348.txt"
...
[Mar 9, 2016 3:02:40 PM] Size of directory "/opt/app/11.2.0/grid4/.patch_storage"after cleanup is 9203930259 bytes.
[Mar 9, 2016 3:02:40 PM] UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
deleted, Please refer log file.
In this case where the GI home has all the 11.2.0.4 PSU applied over a 2 year period has resulted in .patch_storage size of 10GB, on a 12.1.0.2 single instance environment applying GI PSU 12.1.0.2.2 - 12.1.0.2.160119 has resulted in a .patch_storage of 6.1G. So it's expected that throughout the life span of the database the .patch_storage (for both GI home and Oracle home) could grow larger than the minimum storage requirement listed for GI and Oracle home and installation locations must be sized with this growth in mind.

Useful metalink notes
Can You Delete $ORACLE_HOME/.patch_storage Directory ? [ID 403218.1]
How To Avoid Disk Full Issues Because OPatch Backups Take Big Amount Of Disk Space. [ID 550522.1]
Oracle Grid Infrastructure 11.2.0.4.x Patch Set Update SUPPLEMENTAL README [ID 1641136.1]

Upgrading RHEL 7 OS in a 11gR2 (11.2.0.4) and 12cR1 (12.1.0.2) Oracle Restart Environments

$
0
0
As per Oracle documentation "You must relink the Oracle Clusterware and Oracle ASM binaries every time you apply an operating system patch or after an operating system upgrade." There are two earlier posts which shows upgrading of RHEL5 and RHEL6 OS in a 11gR2 RAC environment. This post shows relinking of GI and Oracle homes after upgrading of RHEL 7 in 11.2.0.4 and 12.1.0.2 single instance (Oracle restart) environments.
1. The current RHEL version is 7.1 and the kernel is
uname -r
3.10.0-123.el7.x86_64
2. Before the OS upgrade stop the HAS and disable the auto start on reboot.
crsctl stop has
crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.
3. Comment the entries in oracle-ohasd.service
cat /etc/systemd/system/oracle-ohasd.service
# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup

[Unit]
#Description=Oracle High Availability Services
#After=syslog.target

[Service]
#ExecStart=/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
#Type=simple
#Restart=always

[Install]
#WantedBy=multi-user.target graphical.target
4. Upgrade the RHEL 7 OS. Kernel and RHEL7 version after the upgrade are
uname -r
3.10.0-327.el7.x86_64
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
5. Uncomment the entires in oracle-ohasd.service file and enable auto start of HAS
crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.
Until this point the steps are common for both 11.2.0.4 and 12.1.0.2.

Relinking 11.2.0.4 GI and Oracle Homes
1.Relinking Oracle Home as oracle user. Make sure the relink binary in the PATH is from Oracle home
which relink
/opt/app/oracle/product/11.2.0/dbhome_4/bin/relink
Run relink all
cd $ORACLE_HOME/bin
./relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_4/install/relink.log
At the end of the relink the log file have the following
test ! -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle ||\
mv -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_4/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle
2. Relinkning GI Home
Set ORACLE_HOME=$GI_HOME
As root unlock the GI Home. Using the RHEL 7 provided perl version will result in following error
# perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/app/oracle/product/11.2.0/grid/crs/install) at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
Compilation failed in require at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
Solution for this was mentioned on the changing of hostname post which is to use oracle provided perl
# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Successfully unlock /opt/app/oracle/product/11.2.0/grid
3. As grid user run the relink
$ORACLE_HOME/bin/relink
writing relink log to: /opt/app/oracle/product/11.2.0/grid/install/relink.log
The log file will have the following entry at the end
tail /opt/app/oracle/product/11.2.0/grid/install/relink.log

log tail
test ! -f /opt/app/oracle/product/11.2.0/grid/bin/oracle ||\
mv -f /opt/app/oracle/product/11.2.0/grid/bin/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracleO
mv /opt/app/oracle/product/11.2.0/grid/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/grid/bin/oracle
4. As root patch the GI Home
# cd $ORACLE_HOME/rdbms/install
# ./rootadd_rdbms.sh

$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -patch
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Broadcast message from systemd-journald@rhel7s.codegen.net (Thu 2016-02-18 16:50:14 GMT):
dracut[16456]: dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed

Message from syslogd@rhel7s at Feb 18 16:50:14 ...
dracut:dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed
ACFS driver install actions failed
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
Timed out waiting for ohasd to start.
During the patching the HAS stack start will fail, this is expected.
5. Reboot the host and HAS stack will be up and running as before
Resource Name                       Type                           Target             State              Host
------------- ------ ------- -------- ----------
ora.DATA.dg ora.diskgroup.type ONLINE ONLINE rhel7s
ora.FRA.dg ora.diskgroup.type ONLINE ONLINE rhel7s
ora.LISTENER.lsnr ora.listener.type ONLINE ONLINE rhel7s
ora.asm ora.asm.type ONLINE ONLINE rhel7s
ora.cssd ora.cssd.type ONLINE ONLINE rhel7s
ora.diskmon ora.diskmon.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE rhel7s
ora.ons ora.ons.type OFFLINE OFFLINE
ora.std11g2.db ora.database.type ONLINE ONLINE rhel7s


Relinking 12.1.0.2 GI and Oracle Homes
1. Relinking Oracle home as oracle user. Make sure the relink binary in PATH is from Oracle home. Run relink all to relink the oracle home.
relink all
writing relink log to: /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log
The log file end with the following
tail /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log
test ! -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle ||\
mv -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracleO
mv /opt/app/oracle/product/12.1.0/dbhome_2/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle
2. As root unlock the GI Home by using the $GI_HOME/crs/install/roothas.sh. Unlike 11.2.0.4 there are no issues in unlocking GI home in 12c
./roothas.sh -unlock
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
2016/02/19 18:52:17 CLSRSC-347: Successfully unlock /opt/app/oracle/product/12.1.0/grid
3. As grid user run the relink.
which relink
/opt/app/oracle/product/12.1.0/grid/bin/relink
$ relink
writing relink log to: /opt/app/oracle/product/12.1.0/grid/install/relink.log
The log file will end with the following
tail /opt/app/oracle/product/12.1.0/grid/install/relink.log

test ! -f /opt/app/oracle/product/12.1.0/grid/bin/oracle ||\
mv -f /opt/app/oracle/product/12.1.0/grid/bin/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracleO
mv /opt/app/oracle/product/12.1.0/grid/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/grid/bin/oracle
4. As root patch the GI Home
# $GI_HOME/rdbms/install/rootadd_rdbms.sh

./roothas.sh -patch
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
2016/02/19 18:59:50 CLSRSC-199: Timed out waiting for OHASD to start

The command '/opt/app/oracle/product/12.1.0/grid/perl/bin/perl -I/opt/app/oracle/product/12.1.0/grid/perl/lib -I/opt/app/oracle/product/12.1.0/grid/crs/install /opt/app/oracle/product/12.1.0/grid/crs/install/roothas.pl -patch' execution failed
5. Similar to 11.2.0.4, start of the HAS stack will fail, this is expected. Reboot the host and HAS will be up and running.
Resource Name             Type                      Target             State              Host
------------- ------ ------- -------- ----------
ora.DATA.dg ora.diskgroup.type ONLINE ONLINE rhel7
ora.FRA.dg ora.diskgroup.type ONLINE ONLINE rhel7
ora.LISTENER.lsnr ora.listener.type ONLINE ONLINE rhel7
ora.asm ora.asm.type ONLINE ONLINE rhel7
ora.cssd ora.cssd.type ONLINE ONLINE rhel7
ora.diskmon ora.diskmon.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE rhel7
ora.ons ora.ons.type OFFLINE OFFLINE
ora.se2db.db ora.database.type ONLINE ONLINE rhel7
This concludes the steps for relinking GI and Oracle homes after RHEL 7 OS upgrade.

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment
Upgrading ASMLib and OS in 11gR1 RAC Environment

Valid Node Checking For Registration (VNCR)

$
0
0
Valid Node Checking for Registration (VNCR) allows restriction of instance registration to come only from know servers. This has replaced the use of COST (setting up COST for 11gR1 SE and 11gR2 SE) for protecting against TNS Listener Poison Attack. COST could be still used if it is used for reasons other than listener restriction (for more read 1340831.1).
VNCR is available on 11.2.0.4 and 12c so if the DB is 11.2.0.3 then COST is still the only option available against this attack. Below is an example of setting VNCR on a single instance database. For RAC with SCAN listeners refer the MOS listed at the end of the post.
The listener runs on server with IP 192.168.0.66 (hpc1.domain.net). Currently no DB is registered.
lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 22-FEB-2016 13:06:44

Copyright (c) 1991, 2013, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hpc1.domain.net)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date 22-FEB-2016 13:06:41
Uptime 0 days 0 hr. 0 min. 3 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /opt/app/oracle/product/11.2.0/std4/network/admin/listener.ora
Listener Log File /opt/app/oracle/diag/tnslsnr/hpc1/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hpc1.domain.net)(PORT=1521)))
The listener supports no services
The command completed successfully
The DB (something to register with this listener) runs on server with IP 192.168.0.99. To register with the listener on 192.168.0.66 add a remote listener entry on tnsnames.ora and register the database
cat tnsnames.ora
listener_name =
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.66)(PORT=1521))
)

SQL> alter system set remote_listener='listener_name' scope=both;
System altered.

SQL> alter system register;
System altered.
Check the listener status
lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 22-FEB-2016 13:08:51

Copyright (c) 1991, 2013, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hpc1.domain.net)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date 22-FEB-2016 13:06:41
Uptime 0 days 0 hr. 2 min. 10 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /opt/app/oracle/product/11.2.0/std4/network/admin/listener.ora
Listener Log File /opt/app/oracle/diag/tnslsnr/hpc1/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hpc1.domain.net)(PORT=1521)))
Services Summary...
Service "fgacdb" has 1 instance(s).
Instance "fgacdb", status READY, has 1 handler(s) for this service...
Service "fgacdbXDB" has 1 instance(s).
Instance "fgacdb", status READY, has 1 handler(s) for this service...

The command completed successfully
Above shows that remote database has registered with the listener.



In order to prevent this type of remote registrations enable valid node checking on the listener. To enable VNCR on 11.2.0.4 add following entry to listener.ora file In this case the listener name is "LISTENER".
VALID_NODE_CHECKING_REGISTRATION_LISTENER=1
Reload the listener and check the status. The remote listener registration is blocked.
$ lsnrctl reload
$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 22-FEB-2016 13:24:17

Copyright (c) 1991, 2013, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hpc1.domain.net)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date 22-FEB-2016 13:06:41
Uptime 0 days 0 hr. 17 min. 35 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /opt/app/oracle/product/11.2.0/std4/network/admin/listener.ora
Listener Log File /opt/app/oracle/diag/tnslsnr/hpc1/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hpc1.domain.net)(PORT=1521)))
The listener supports no services
Any attempt to register with the listener is blocked and the listener.log will have an entry similar to following
Listener(VNCR option 1) rejected Registration request from destination 192.168.0.99
22-FEB-2016 13:31:47 * service_register_NSGR * 1182
TNS-01182: Listener rejected registration of service ""
On 12c is VNCR is enabled by default and any attempt for remote registration is rejected and listener log will have an entry as above.

Useful metalink notes
Valid Node Checking For Registration (VNCR) [ID 1600630.1]
How to Enable VNCR on RAC Database to Register only Local Instances [ID 1914282.1]

Related Posts
Implementing Class of Secure Transport (COST) to Restrict Instance Registration in Oracle 11gR2 SE RAC (Solution mentioned in Oracle Security Alert for CVE-2012-1675)
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

Changing SCAN Setting

$
0
0
The post lists steps for changing the SCAN setting in a cluster. This is a test system where the SCAN is currently resolved via the /etc/hosts file with a single IP.
$ srvctl config scan
SCAN name: rhel6m-scan, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /rhel6m-scan/192.168.0.91

$ srvctl config scan_listener
SCAN Listener MYSCANLISTENER_SCAN1 exists. Port: TCP:9120
It uses a non-default scan listener name and non-default port.
This setup will be changed so that a new SCAN setting that resolves SCAN through DNS is used by the cluster. Ideally the scan name should remain the same so that application connecting doesn't have to change any connection strings after the change but in this case the new SCAN name is different to the currently used scan name.
1. Make the changes to /etc/resovle.conf and /etc/nsswitch.conf so that new SCAN look up works on all cluster nodes. Below shows nslookup working on one of the nodes
[grid@rhel6m1 admin]$ nslookup rac-scan.domain.net
Server: 192.168.0.66
Address: 192.168.0.66#53

Name: rac-scan.domain.net
Address: 192.168.0.93
Name: rac-scan.domain.net
Address: 192.168.0.94
Name: rac-scan.domain.net
Address: 192.168.0.92

[grid@rhel6m1 admin]$ nslookup rac-scan.domain.net
Server: 192.168.0.66
Address: 192.168.0.66#53

Name: rac-scan.domain.net
Address: 192.168.0.92
Name: rac-scan.domain.net
Address: 192.168.0.93
Name: rac-scan.domain.net
Address: 192.168.0.94

[grid@rhel6m1 admin]$ nslookup rac-scan.domain.net
Server: 192.168.0.66
Address: 192.168.0.66#53

Name: rac-scan.domain.net
Address: 192.168.0.94
Name: rac-scan.domain.net
Address: 192.168.0.92
Name: rac-scan.domain.net
Address: 192.168.0.93
2. Stop the scan and scan listeners
$ srvctl stop scan_listener
$ srvctl stop scan
3. Modify the scan setting by specifying the new SCAN name.
# srvctl modify scan -n rac-scan.domain.net

# srvctl config scan
SCAN name: rac-scan.domain.net, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /rac-scan.domain.net/192.168.0.93
SCAN VIP name: scan2, IP: /rac-scan.domain.net/192.168.0.94
SCAN VIP name: scan3, IP: /rac-scan.domain.net/192.168.0.92


4. Change the remote_listener parameter to reflect the new SCAN
SQL> alter system set remote_listener='rac-scan.domain.net:9120' scope=both sid='*';
5. Update the scan listener
$ srvctl modify scan_listener -u

$ srvctl config scan_listener
SCAN Listener MYSCANLISTENER_SCAN1 exists. Port: TCP:9120
SCAN Listener MYSCANLISTENER_SCAN2 exists. Port: TCP:9120
SCAN Listener MYSCANLISTENER_SCAN3 exists. Port: TCP:9120
6. Finally start the scan listener
$ srvctl start scan_listener
Useful metalink notes
How to Modify SCAN Setting or SCAN Listener Port after Installation [ID 972500.1]
How To Convert an 11gR2 GNS Configuration To A Standard Configuration Using DNS Only [ID 1489121.1]
How to Modify Public Network Information including VIP in Oracle Clusterware [ID 276434.1]

Related Posts
Changing Listener and SCAN Listener Port in 11gR2 RAC
Changing Listener and SCAN Listener Name in 11gR2 RAC
GNS Setup for RAC
SCAN (Single Client Access Name) Set Up Using DNS

Identity Column Vs Trigger/Sequence Method for Populating Primary Key Column

$
0
0
Oracle 12c introduced a new featured called identity column which allows auto population of primary key columns, perfect for database models that use surrogate keys. Before the identity column was introduced same was achieved with the use of trigger and sequence (identity column also use a sequence behind the scene). This post is to compare the two methods and see if there's any performance regression or benefits of using one over the other.
Two tables were created first one for the trigger base approach.
SQL> create table seqpritable (a number primary key, b number, c varchar2(100));
SQL> create sequence IDSEQ cache 100 noorder;

Create Or Replace Trigger Seqinstrig Before Insert On Seqpritable REFERENCING NEW AS NEW OLD AS OLD for each row
Begin
select idseq.nextval into :NEW.a from dual;
end;
/
The second table with identity column
SQL>  create table seqnotrigtable (a number generated as identity cache 100 noorder primary key, b number, c varchar2(100));
Looking at the sequence it could seen the system generated sequence for the identity column
SQL> select * from user_sequences;

SEQUENCE_NAME MIN_VALUE MAX_VALUE INCREMENT_BY C O CACHE_SIZE LAST_NUMBER PARTITION_COUNT S K
--------------- ---------- ---------- ------------ - - ---------- ----------- --------------- - -
IDSEQ 1 1.0000E+28 1 N N 100 1 N N
ISEQ$$_26555 1 1.0000E+28 1 N N 100 1 N N
A multi-threaded java code was used to insert some rows into the tables (code is given at the end of the post). The testing was done on a 2 node RAC running Oracle 12 SE2 (12.1.0.2.160419). Two sets of tests were carried out, first 2 concurrent threads inserting and on second test 4 concurrent threads were inserting. The active session usage is used for comparison and is shown for the tests below (first two spikes refer to 2 threads inserting and second 2 spikes refer to 4 threads inserting).
The cluster (gc current block busy) and other (gcs log flush sync) are present on all the test same level and not related to use of sequence but on insert of data to the table. In terms of CPU usage there's not much difference either. Looking at this test result it's clear that no method is beneficial over the other in terms of resource usage/performance. However in terms of administration and management the identity column tops as there's no need to maintain a separate sequence or trigger. So when migrating to 12c it may be a worthwhile to move trigger based surrogate key population to identity columns and take advantage of this feature.



Java code used for inserts
public class SeqInsert extends Thread {

private Connection con;

public SeqInsert(Connection con) {
this.con = con;
}

public static void main(String[] args) {
try {

OracleDataSource pool = new OracleDataSource();
pool.setURL("jdbc:oracle:thin:@rac-scan.domain.net:1521/std12csrv");
pool.setUser("asanga");
pool.setPassword("asa");

SeqInsert[] ts = new SeqInsert[Integer.parseInt(args[0])];

for(int i = 0 ; i < ts.length; i++){
Connection con = pool.getConnection();
con.setAutoCommit(false);
ts[i] = new SeqInsert(con);
}

for(int i = 0 ; i < ts.length; i++){
ts[i].start();
}

} catch (Exception ex) {
ex.printStackTrace();
}
}

@Override
public void run() {

try {

// String SQL = "insert into seqpritable (b,c) values (?,?)";
String SQL = "insert into seqnotrigtable (b,c) values (?,?)";
PreparedStatement pr = con.prepareStatement(SQL);
for (int i = 0; i < 10000; i++) {

pr.setInt(1, i);
pr.setString(2, getName()+ i);
pr.execute();
}
con.commit();
pr.close();
con.close();

} catch (Exception ex) {
ex.printStackTrace();
}
}
}

mount to NFS server failed: RPC Error: Program not registered.

$
0
0
Mounting to an NFS location was failing with following error. This was an existing NFS mount location which worked fine earlier.
mount -t nfs -o rw,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp,nolock,actimeo=0 nfs-server:/home/oracle/backup/ /home/oracle/dbbackup
mount: mount to NFS server 'nfs-server' failed: RPC Error: Program not registered.
Showmount to the nfs-server also fails with
showmount -e nfs-server
mount clntudp_create: RPC: Program not registered
To fix this restart the nfs related services in the following order on the nfs server. First rpcbind service
[root@nfs-server ~]# service rpcbind start
Starting rpcbind: [ OK ]
Second the nfs service

[root@nfs-server ~]# service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS mountd: [ OK ]
Starting NFS daemon: [ OK ]
Starting RPC idmapd: [ OK ]
Finally the portmap service

[root@nfs-server ~]# service portmap start
Starting portmap: [ OK ]


Afterwards NFS mount works without issue
# mount -t nfs -o rw,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp,nolock,actimeo=0 10.10.0.102:/home/oracle/backup/ /home/oracle/dbbackup
# df -h
Filesystem Size Used Avail Use% Mounted on

xx.xx.x.xxx:/home/oracle/backup/
526G 407G 93G 82% /home/oracle/dbbackup
Related Posts
RMAN Backups on NFS
Direct NFS Setup

Secure External Password Store and Oracle Restart with Role Separation

$
0
0
Oracle secure external password store allows storing of password needed for database connection in a client side Oracle wallet. It allows password-less connection to the database and useful in running scripts without having to put the password in them. A secure external password store could be setup as below.
Create a wallet (usually in TNS_ADMIN but could be different location as well)
cd $ORACLE_HOME/network/admin/

orapki wallet create -wallet . -pwd "welcome1" -auto_login_local
Oracle PKI Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

[oracle@rhel7 admin]$ ls
cwallet.sso ewallet.p12 samples shrept.lst tnsnames.ora
The option "auto_login_local" prevents the wallet being moved to another host and used (1114599.1). It also ties the wallet to the operating system user preventing other users in the same host using the wallet as well (1505040.1). Create a credential, in this case std11g2 is a TNS entry that exists in tnsnames.ora file and asanga and asa are username and password
mkstore -wrl . -createCredential std11g2 asanga asa
Oracle Secret Store Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:
Create credential oracle.security.client.connect_string1
Add the wallet location to sqlnet.ora file and set WALLET_OVERRIDE to true which makes user names and passwords from the wallet to be used for authentication.
more sqlnet.ora
WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA = (DIRECTORY = /opt/app/oracle/product/11.2.0/dbhome_4/network/admin))
)

SQLNET.WALLET_OVERRIDE = TRUE
The existing credentials could be listed with the following command
mkstore -wrl . -listCredential
Oracle Secret Store Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:
List credential (index: connect_string username)
1: std11g2 asanga
With the external secure password store now setup, one could connected to the database without specifying the password as show below
[oracle@rhel7 orawallet]$ sqlplus  /@std11g2

SQL> show user
USER is "ASANGA"
Problem is that when secure external password store is setup for a Oracle restart (Single instance with ASM) database that has role separation (GI installed grid user and DB installed as oracle), the database fails to start during restarts and when srvctl is used. Below for a 11.2.0.4 setup
srvctl start database -d std11g2
PRCR-1079 : Failed to start resource ora.std11g2.db
CRS-5017: The resource action "ora.std11g2.db start" encountered the following error:
ORA-01078: failure in processing system parameters
. For details refer to "(:CLSN00107:)" in "/opt/app/oracle/product/11.2.0/grid_4/log/rhel6m1/agent/ohasd/oraagent_grid//oraagent_grid.log".

CRS-2674: Start of 'ora.std11g2.db' on 'rhel6m1' failed
Below for a 12.1.0.2 setup
srvctl start database -d se2db
PRCR-1079 : Failed to start resource ora.se2db.db
CRS-5017: The resource action "ora.se2db.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/se2db/spfilese2db.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA/se2db/spfilese2db.ora
ORA-12578: TNS:wallet open failed
. For details refer to "(:CLSN00107:)" in "/opt/app/oracle/diag/crs/rhel7/crs/trace/ohasd_oraagent_grid.trc".

CRS-2674: Start of 'ora.se2db.db' on 'rhel7' failed
MOS notes 1612712.1,1383938.1 says to set the permission to 750 for this issue but it didn't work.



During further investigation of the issue following observations were made.
1. The problem doesn't happen when both grid home and oracle home are installed under one user (usually as oracle user).
2. Problem doesn't happen in RAC environments even if grid home and oracle home are installed as different users (grid home as grid user and oracle home as oracle user)
3. Problem only happens in single instance environments (tested on 12c and 11.2.0.4) where grid home is owned by grid user and oracle home is owned by oracle user.
Looking at the trace file shows that srvctl fails to connect to idle instance to start the database
2016-03-30 11:11:12.474778 :CLSDYNAM:2930841344: [ora.se2db.db]{0:0:45668} [start] clsnInstConnection::makeConnectStr UsrOraEnv ,ORACLE_BASE= m_oracleHome /opt/app/oracle/product/12.1.0/dbhome_2 Crshome /opt/app/oracle/product/12.1.0/grid
2016-03-30 11:11:12.474822 :CLSDYNAM:2930841344: [ora.se2db.db]{0:0:45668} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle)(ARGV0=oraclese2db)(ENVS='ORACLE_HOME=/opt/app/oracle/product/12.1.0/dbhome_2,ORACLE_SID=se2db,LD_LIBRARY_PATH=,ORACLE_BASE=')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))')(CONNECT_DATA=(SID=se2db))))
2016-03-30 11:11:12.475460 :CLSDYNAM:2930841344: [ora.se2db.db]{0:0:45668} [start] InstAgent::stop non pool pConnxn 1 9434f9e0
2016-03-30 11:11:12.475519 :CLSDYNAM:2930841344: [ora.se2db.db]{0:0:45668} [start] InstConnection::connectInt: server not attached
2016-03-30 11:11:13.498240 :CLSDYNAM:2930841344: [ora.se2db.db]{0:0:45668} [start] InstConnection:connectInt connected
2016-03-30 11:11:13.498315 :CLSDYNAM:2930841344: [ora.se2db.db]{0:0:45668} [start] InstConnection::shutdown mode 4
2016-03-30 11:11:13.499364 :CLSDYNAM:2930841344: [ora.se2db.db]{0:0:45668} [start] ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0

2016-03-30 11:11:13.499453 :CLSDYNAM:2930841344: [ora.se2db.db]{0:0:45668} [start] InstConnection::disassociateEdition OCI error 1034
Then the start of the database was tested a wallet created without the "local" option. As mentioned earlier local option prevents wallet being used in another host and is tied to the OS user. On 11.2.0.4
orapki wallet create -wallet . -pwd "welcome1"-auto_login
Oracle PKI Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

srvctl start database -d std11g2
srvctl status database -d std11g2
Database is running.
On 12.1.0.2
orapki wallet create -wallet . -pwd "welcome1" -auto_login
Oracle PKI Tool : Version 12.1.0.2
Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved.

srvctl start database -d se2db
srvctl status database -d se2db
Database is running.
So it appears the problem seem to be use of auto_login_local since there's no issue when using auto_login. Since wallet wasn't moved to a different server, the only reason that could prevent DB from starting with a wallet created with auto_login_local option is if the starting command is executed by a user other than the user tied to the wallet. Right now the wallet has been created as Oracle user and srvctl start was executed as oracle user. So it appears that grid user gets involved in starting the DB even when srvctl is run as oracle user (why the same is not happening in RAC role separated config is under SR)
So the solution in this case was to create an empty wallet as grid user and set permission to 750.
[grid@rhel7 wallet]$ orapki wallet create -wallet . -pwd "welcome1" -auto_login_local
[grid@rhel7 wallet]$ chmod 750 cwallet.sso ewallet.p12
[grid@rhel7 wallet]$ ls -lrt
-rw-rw-rw-. 1 grid oinstall 0 Apr 15 17:35 ewallet.p12.lck
-rwxr-x---. 1 grid oinstall 75 Apr 15 17:35 ewallet.p12
-rw-rw-rw-. 1 grid oinstall 0 Apr 15 17:35 cwallet.sso.lck
-rwxr-x---. 1 grid oinstall 120 Apr 15 17:35 cwallet.sso
No credentials are added to this wallet as such nothing to list
[grid@rhel7 wallet]$  mkstore -wrl . -listCredential
Oracle Secret Store Tool : Version 12.1.0.2
Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:
List credential (index: connect_string username)
Used a non-default location to this wallet file and this was added to $ORACLE_HOME/network/admin/sqlnet.ora file(not to $GI_HOME/network/admin/sqlnet.ora)
pwd
/opt/app/oracle/product/12.1.0/dbhome_2/network/admin

more sqlnet.ora
WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA = (DIRECTORY = /usr/local/wallet))
)

SQLNET.WALLET_OVERRIDE = TRUE
With this setup in place srvctl works fine and database get started automatically after restarts. However password-less access fails as the wallet is owned by grid user
[oracle@rhel7 orawallet]$ sqlplus  /@se2db

ERROR:
ORA-12578: TNS:wallet open failed
To fix this, create another wallet at a different location as Oracle user.
ls /usr/local/wallet/orawallet

-rwxrwx---. 1 oracle oinstall 528 Apr 15 17:20 ewallet.p12
-rwxrwx---. 1 oracle oinstall 573 Apr 15 17:20 cwallet.sso
Create another sqlnet.ora file in the same directory (orawallet). Add this directory path as the wallet location
WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA = (DIRECTORY = /usr/local/wallet/orawallet))
)

SQLNET.WALLET_OVERRIDE = TRUE
Whenever password-less access using external password store is needed export this location as the TNS_ADMIN and connect.
export TNS_ADMIN=/usr/local/wallet/orawallet
sqlplus /@se2db

SQL> show user
USER is "ASANGA"
If any database scripts require password-less access before running export the TNS_ADMIN to location where second sqlnet.ora file was set.
Why the auto_login_local fails this way in Oracle restart with role separation is still under SR.

Useful metalink notes
Using a Secure External Password Store with the JDBC Thin Driver [ID 1441745.1]
Using The Secure External Password Store [ID 340559.1]
How To Avoid Expdp And Impdp Passwords Being Visible By "ps" Unix Command? [ID 869825.1]
How To Prevent The Secure Password Store Wallet From Being Moved to Another Host [ID 1114599.1]
Oracle Cluster failed to start with ASM instance getting ORA-00443 [ID 2000868.1]
Database Failed To Start [ID 1922401.1]
Oracle Restart: srvctl fails to start database with error CRS-5010 if RDBMS and Grid under different users [ID 1335607.1]
How to configure SEPS for the pluggable databases [ID 1980698.1]
Bug 11706168 - ORA-00000 during STARTUP with SQLNET.WALLET_OVERRIDE=TRUE [ID 11706168.8]
How To Configure The Secure External Password Store To Allow The Connection To RMAN Catalog? [ID 1383938.1]
ORA-15055 and ORA-12578 on database startup with external wallet store [ID 1612712.1]
How To Check Whether The Wallet Is A Local Auto Login Wallet [ID 1505040.1]

Moving Role Separated Oracle Restart Setup to Single User Setup

$
0
0
Oracle role separation setup involves installing the grid infrastructure and oracle database software as different users. Commonly GI is installed as grid user and oracle database software as oracle user. On rare occasions this role separation could result in behavior that is not experienced when both GI and Oracle DB is installed using a single user. Issues with secure password store and external tables are just two examples.
This post shows steps to moving from a role separated Oracle restart setup to a single user setup.
The way to do this is to remove the GI software installed with grid user and reinstall it with oracle user. However it is not possible to remove the the existing GI home while oracle database home is configured on it.
[grid@rhel7 deinstall]$ ./deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /opt/app/oraInventory/logs/
...
Traces log file: /opt/app/oraInventory/logs//crsdc_2016-02-24_07-28-15PM.log
ERROR: Can't deconfigure Oracle Restart before removing or downgrading managed Oracle databases.
So the option is to manually remove the grid home. This oracle restart environment used here is 12.1.0.2 on RHEL 7
1. Create a pfile from the ASM spfile
SQL> create pfile='/home/grid/asmpfile.ora' from spfile;

[grid@rhel7 ~]$ more asmpfile.ora
+ASM.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from in memory value
+ASM.asm_diskgroups='FRA'#Manual Mount
*.asm_diskstring='/dev/sd*'
*.asm_power_limit=1
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
2. Backup the oratab file

3. Stop the HAS service
crsctl stop has
4. As root user, delete the GI home installed with grid user
[root@rhel7 12.1.0]# pwd
/opt/app/oracle/product/12.1.0
[root@rhel7 12.1.0]# rm -rf grid
5. Remove other HAS related files
cd /etc
rm -rf ora*
find . -name *ohasd -exec rm {} \;

rm /etc/systemd/system/oracle-ohasd.service
rm /etc/init.d/*ohasd
rm -rf /opt/ORCLfmap
rm -rf /var/tmp/.oracle
Removing of "/var/tmp/.oracle" is important without removing it, trying to reinstall will fail. More on 1997268.1
2016/02/24 20:04:38 CLSRSC-318: Failed to start Oracle OHASD service
Died at /opt/app/oracle/product/12.1.0/grid_2/crs/install/crsinstall.pm line 3041.
6. Backup the existing oraInventory and create a new oraInventory directory as oracle user
drwxrwx---.  6 grid   oinstall   97 Feb  5 19:38 oraInventory

# mv oraInventory oraInventorybak
# mkdir oraInventory
# chown oracle:oinstall oraInventory
7. Change udev file so that owner is the oracle user
OWNER="oracle", GROUP="asmadmin", MODE="0660"
8. Remove grid user owned GI related folders from ORACLE_BASE
 cd $ORACLE_BASE
# ls -l
drwxrwx---. 4 grid oinstall 29 Sep 29 16:58 admin
drwxr-x---. 3 oracle oinstall 18 Sep 29 17:26 audit
drwxrwxr-x. 6 grid oinstall 56 Sep 30 15:38 cfgtoollogs
drwxr-xr-x. 2 grid oinstall 6 Sep 29 16:43 checkpoints
drwxrwx---. 4 grid oinstall 32 Sep 29 16:39 crsdata
drwxr-x---. 19 grid oinstall 4096 Sep 29 16:38 diag
drwxr-xr-x. 3 grid oinstall 17 Feb 4 17:44 log
drwxrwxr-x. 3 grid oinstall 19 Sep 29 16:33 product
drwxr-xr-x. 3 grid oinstall 24 Sep 29 16:39 rhel7

rm -rf admin/+ASM
rm -rf cfgtoollogs
rm -rf checkpoints
rm -rf crsdata
rm -rf diag
rm -rf log
rm -rf rhel7/

# ls -l

drwxrwx---. 3 grid oinstall 18 Feb 24 19:39 admin
drwxr-x---. 3 oracle oinstall 18 Sep 29 17:26 audit
drwxrwxr-x. 3 grid oinstall 19 Sep 29 16:33 product
Change the ownership of the remaining files to oracle user
chown oracle:oinstall product/12.1.0
chown oracle:oinstall product
chown oracle:oinstall admin

# ls -l

drwxrwx---. 3 oracle oinstall 18 Feb 24 19:39 admin
drwxr-x---. 3 oracle oinstall 18 Sep 29 17:26 audit
drwxrwxr-x. 3 oracle oinstall 19 Sep 29 16:33 product
9. Reboot the server. Once restarted verify the permission on the block devices used for ASM
ls -l /dev/sd*
brw-rw----. 1 oracle asmadmin 8, 17 Feb 24 19:42 /dev/sdb1
brw-rw----. 1 oracle asmadmin 8, 33 Feb 24 19:42 /dev/sdc1
10. Add to oracle user asm related os groups. The final group list for oracle user is as below
id oracle
uid=1001(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba),1003(oper),1005(asmoper),1006(asmdba),1007(asmadmin),1008(backupdba),1009(dgdba),1010(kmdba)
11. Install the GI software selecting "software only" option.

12. Run the root scripts when prompted
# /opt/app/oraInventory/orainstRoot.sh
Changing permissions of /opt/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /opt/app/oraInventory to oinstall.
The execution of the script is complete.

# /opt/app/oracle/product/12.1.0/grid_2/root.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /opt/app/oracle/product/12.1.0/grid_2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/opt/app/oracle/product/12.1.0/grid_2/perl/bin/perl -I/opt/app/oracle/product/12.1.0/grid_2/perl/lib -I/opt/app/oracle/product/12.1.0/grid_2/crs/install /opt/app/oracle/product/12.1.0/grid_2/crs/install/roothas.pl


To configure Grid Infrastructure for a Cluster execute the following command as oracle user:
/opt/app/oracle/product/12.1.0/grid_2/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.
Run the command to configure Oracle restart.
# /opt/app/oracle/product/12.1.0/grid_2/perl/bin/perl -I/opt/app/oracle/product/12.1.0/grid_2/perl/lib -I/opt/app/oracle/product/12.1.0/grid_2/crs/install /opt/app/oracle/product/12.1.0/grid_2/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid_2/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel7 successfully pinned.
2016/03/01 18:39:59 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'

rhel7 2016/03/01 18:40:32 /opt/app/oracle/product/12.1.0/grid_2/cdata/rhel7/backup_20160301_184032.olr 0
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel7'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel7'
CRS-2677: Stop of 'ora.evmd' on 'rhel7' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel7' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/03/01 18:40:49 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ons
OFFLINE OFFLINE rhel7 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 OFFLINE OFFLINE STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE rhel7 STABLE
--------------------------------------------------------------------------------


13. Add ASM home entry to the /etc/oratab file
+ASM:/opt/app/oracle/product/12.1.0/grid_2:N
14. Add listener and ASM to HAS configuration
$ srvctl add listener -l listener -o /opt/app/oracle/product/12.1.0/grid_2 -p 1521
$ srvctl start listener -l listener
$ srvctl status listener -l listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rhel7

$ srvctl add asm -l listener -d "/dev/sd*"
$ srvctl start asm
$ srvctl config asm
ASM home:
Password file:
ASM listener: LISTENER
Spfile:
ASM diskgroup discovery string: /dev/sd*

[oracle@rhel7 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE rhel7 STABLE
ora.asm
ONLINE ONLINE rhel7 Started,STABLE
ora.ons
OFFLINE OFFLINE rhel7 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE rhel7 STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE rhel7 STABLE
--------------------------------------------------------------------------------
15. Modify the inventory for GI Home to include crs=true. The inventory.xml content before and after the modification
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/opt/app/oracle/product/12.1.0/grid_2" TYPE="O" IDX="1"/>
</HOME_LIST>

./runInstaller -updateNodeList ORACLE_HOME=/opt/app/oracle/product/12.1.0/grid_2 CRS=TRUE -silent

<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/opt/app/oracle/product/12.1.0/grid_2" TYPE="O" IDX="1" CRS="true"/>
</HOME_LIST>
16. Patch the GI home to the same level as the Oracle database home.

17. Attach the Oracle database home
./runInstaller -silent -attachHome ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME="oracle_home"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 4097 MB Passed
The inventory pointer is located at /etc/oraInst.loc
'AttachHome' was successful.
Inventory content before and after oracle home attached
Before 
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/opt/app/oracle/product/12.1.0/grid" TYPE="O" IDX="1" CRS="true"/>
</HOME_LIST>

After
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/opt/app/oracle/product/12.1.0/grid_2" TYPE="O" IDX="1" CRS="true"/>
<HOME NAME="oracle_home" LOC="/opt/app/oracle/product/12.1.0/dbhome_2" TYPE="O" IDX="2"/>
</HOME_LIST>
18. Mount the ASM disk groups
SQL>  select name,state from v$asm_diskgroup;

NAME STATE
------------------------------ -----------
FRA DISMOUNTED
DATA DISMOUNTED

alter diskgroup data mount;
alter diskgroup fra mount;
19. If does not exists create spfile alias and add the database to HAS configuration. Also db entry to oratab file.
mkalias parameterfile/spfile.266.891711477 spfilese2db.ora

srvctl add database -d se2db -o /opt/app/oracle/product/12.1.0/dbhome_2 -p +DATA/se2db/spfilese2db.ora -a "data,fra"
srvctl start database -d se2db
20. Add auto start options to the components
crsctl modify resource ora.DATA.dg -attr "AUTO_START"="always" -unsupported
crsctl modify resource ora.FRA.dg -attr "AUTO_START"="always" -unsupported
crsctl modify resource ora.LISTENER.lsnr -attr "AUTO_START"="always" -unsupported
crsctl modify resource ora.se2db.db -attr "AUTO_START"="always" -unsupported
21. The HAS stack should be now up and running
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE rhel7 STABLE
ora.FRA.dg
ONLINE ONLINE rhel7 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rhel7 STABLE
ora.asm
ONLINE ONLINE rhel7 Started,STABLE
ora.ons
OFFLINE OFFLINE rhel7 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE rhel7 STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE rhel7 STABLE
ora.se2db.db
1 ONLINE ONLINE rhel7 Open,STABLE
--------------------------------------------------------------------------------
This concludes the steps for moving a role separated oracle restart setup to a single user setup.

Useful metalink notes
Oracle Restart: 12.1 grid installation root.sh fails while starting ohasd [ID 1997268.1]

Related Posts
Recover From a Clusterware Home Deletion
Moving non-RAC Database and ASM Between Servers
Changing The Cluster Name
Viewing all 315 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>