Master Key Not Set for the Database Shown After Applying DBRU 19.11 (19.11.0.0.210420)

May 14, 2021, 6:00 am

≫ Next: Importing Unified Audit Data From Standby

≪ Previous: ORA-12754 When Setting ADG_ACCOUNT_INFO_TRACKING to Global

After applying the DBRU 19.11 following message was seen on the alert log on a database which has TDE setup.

2021-04-26T21:14:47.432344+05:30
DEVPDB(3):ALTER PLUGGABLE DATABASE OPEN detects that an encrypted tablespace has been restored but the master key has not been set for the database, or the database has been flashback'ed prior to first set key of the master key (pdb 3) .
DEVPDB(3): Resetting the master key is required. Please execute ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY command, or select the latest master key from V$ENCRYPTION_KEYS and execute ADMINISTER KEY MANAGEMENT USE KEY <key_id> command if the SET ENCRYPTION KEY command cannot find and decide the master key to use.

If TDE is not used then no such message is shown.
This error is shown only when an auto login wallet is in place. Doesn't matter if the wallet is local auto login or not message still appears.
Message is not shown when auto login wallet is removed and key store is opened manually after CDB start. Message is shown with respect to the user created PDB.
Even when the above message is shown, there's no issue in accessing the encrypted tablespaces and querying data.
Setting the encryption key or recreating the auto login wallet has no effect on it. Below steps were carried out but still the message remains.

alter session set container=devpdb;
Session altered.

ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY asanga321 WITH BACKUP;
keystore altered.

SELECT con_id,MASTERKEY_ACTIVATED FROM V$DATABASE_KEY_INFO;
CON_ID MAS
---------- ---
3 YES

SQL> conn / as sysdba
Connected.
SQL> ADMINISTER KEY MANAGEMENT CREATE local AUTO_LOGIN KEYSTORE FROM KEYSTORE IDENTIFIED BY asanga321;
keystore altered.

There was no tablespace restore or flashback on it. It seems when auto login wallet is involved for some reason the correct master key is not identified.
Rolling back the DBRU 19.11 (DB went back to 19.10) resolved the issue. So it appears this is something introduced with the DBRU 19.11 patch. To confirm this a new database was created using 19.3 base release and 19.11 DBRU. Setup TDE and auto login wallet (below output from alert log).


TESTPDB(3):Creating new database key for new master key and wallet
TESTPDB(3):Creating new database key with the new master key
TESTPDB(3):New database key and new master key created successfully
TESTPDB(3):create tablespace enctest datafile size 10m ENCRYPTION USING 'AES256' ENCRYPT
TESTPDB(3):Completed: create tablespace enctest datafile size 10m ENCRYPTION USING 'AES256' ENCRYPT

Then restarted the database and could see the following on the alert log.

TESTPDB(3):ALTER PLUGGABLE DATABASE OPEN detects that an encrypted tablespace has been restored but the master key has not been set for the database, or the database has been flashback'ed prior to first set key of the master key (pdb 3).
TESTPDB(3): Resetting the master key is required. Please execute ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY command, or select the latest master key from V$ENCRYPTION_KEYS and execute ADMINISTER KEY MANAGEMENT USE KEY <key_id> command if the SET ENCRYPTION KEY command cannot find and decide the master key to use.

To further isolate the issue, created a third database was craeted. This time only TDE was setup, no encrypted tablespaces.

2021-05-11T11:23:48.690428+00:00
Creating new database key for new master key and wallet
Creating new database key with the new master key
Switching out all online logs for the new master key
2021-05-11T11:23:48.746184+00:00
Thread 1 advanced to log sequence 16 (LGWR switch),  current SCN: 1288117
  Current log# 1 seq# 16 mem# 0: +DATA/TDETEST/ONLINELOG/group_1.297.1072260997
  Current log# 1 seq# 16 mem# 1: +FRA/TDETEST/ONLINELOG/group_1.530.1072260999
2021-05-11T11:23:48.748707+00:00
Logfile switch for new master key complete
New database key and new master key created successfully
TDETESTPDB(3):Creating new database key for new master key and wallet
TDETESTPDB(3):Creating new database key with the new master key
TDETESTPDB(3):New database key and new master key created successfully

In this case too the alert log showed the same message as before.

2021-05-11T11:24:58.081692+00:00
TDETESTPDB(3):ALTER PLUGGABLE DATABASE OPEN detects that an encrypted tablespace has been restored but the master key has not been set for the database, or the database has been flashback'ed prior to first set key of the master key (pdb 3).
TDETESTPDB(3): Resetting the master key is required. Please execute ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY command, or select the latest master key from V$ENCRYPTION_KEYS and execute ADMINISTER KEY MANAGEMENT USE KEY <key_id> command if the SET ENCRYPTION KEY command cannot find and decide the master key to use.

The auto login wallet does contain the same key ids which could be checked with

SQL> show pdbs

CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 TESTPDB READ WRITE NO

SQL> select * from gv$encryption_wallet;

INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR KEYSTORE FULLY_BAC CON_ID
---------- -------------------- ------------------------------ ------------------------------ -------------------- --------- -------- --------- ----------
1 FILE /opt/app/oracle/wallet/tde/ OPEN AUTOLOGIN SINGLE NONE NO 1
1 FILE OPEN AUTOLOGIN SINGLE UNITED NO 2
1 FILE OPEN AUTOLOGIN SINGLE UNITED NO 3

SQL> select con_id,KEY_ID,CREATION_TIME,activation_time FROM V$ENCRYPTION_KEYS;

CON_ID KEY_ID CREATION_TIME ACTIVATION_TIME
---------- ------------------------------------------------------------ ----------------------------------- --------------------
3 AVQm2JpHDk+jv59DdGa3lz4AAAAAAAAAAAAAAAAAAAAAAAAAAAAA 28-APR-21 04.25.22.932575 PM +05:30 28-APR-21 04.25.23.243620 PM +05:30
1 ATJSoZoecE+jv6iBZKWk2s0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA 28-APR-21 04.25.22.830950 PM +05:30 28-APR-21 04.25.23.064019 PM +05:30 


orapki wallet display -wallet /opt/app/oracle/wallet/tde
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
Subject: CN=oracle
User Certificates:
Oracle Secret Store entries:
ORACLE.SECURITY.DB.ENCRYPTION.ATJSoZoecE+jv6iBZKWk2s0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.AVQm2JpHDk+jv59DdGa3lz4AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY.C107201B5EA38B36E053FE04A8C049CF
ORACLE.SECURITY.ID.ENCRYPTION.
ORACLE.SECURITY.KB.ENCRYPTION.
ORACLE.SECURITY.KM.ENCRYPTION.ATJSoZoecE+jv6iBZKWk2s0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KM.ENCRYPTION.AVQm2JpHDk+jv59DdGa3lz4AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KT.ENCRYPTION.ATJSoZoecE+jv6iBZKWk2s0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KT.ENCRYPTION.AVQm2JpHDk+jv59DdGa3lz4AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Trusted Certificates:

The issue was happening on OCI DBCS VM databases after applying the DBRU 19.11.
Raised a P1 24x7 SR and 48 hours later Oracle confirmed this is internal bug 29528184 filed as happening only on exadata. Reason for issue is given as "The lookup of MKID for PDB was done in PDB's keystore which was not OPEN". It seems the bug is now packaged with 19.11 thus seeing it on OCI DBCS and on-prem DBs.
No fix available yet. Oracle is working on a backport for non exadata environments.

↧

Importing Unified Audit Data From Standby

May 22, 2021, 6:00 am

≫ Next: Database Resident Connection Pool (DRCP) and Data Guard

≪ Previous: Master Key Not Set for the Database Shown After Applying DBRU 19.11 (19.11.0.0.210420)

In a primary database importing audit data involves a export job with INCLUDE=AUDIT_TRAILS parameter. However, situation is different on standby as audit records are written to file system location. In a CDB the root container audit data on standby is available in a folder path similar to $ORACLE_BASE/audit/$ORACLE_SID while the audit data for the PDB is available in a path similar to $ORACLE_BASE/audit/$ORACLE_SID/GUID where GUID is the PDB's generic unique identifier.
The audit files are not text file but binary files. The strings command could be used to get glance at the content of the audit file.

strings ora_audit_0629.bin
ANG Spillover Audit File
ORAAUDNG
oracle
ip-172-31-11-54.eu-west-1.compute.internal
pts/0
(TYPE=(DATABASE));(CLIENT ADDRESS=((ADDRESS=(PROTOCOL=tcp)(HOST=172.31.11.54)(PORT=23959))));
LOCKTEST
sqlplus@ip-172-31-11-54.eu-west-1.compute.intern
9837
ORA_LOGON_FAILURES
LOCKTEST

However, this doesn't give all the information as seen from above there's no date information.
Also same audit file would be appended with audit data for multiple sessions. For example above audit record generated for login failure for session originating in sever 172.31.11.54. However, after a while more records could be seen on the same file such as below. Now there exists another login failure for client originating from server 172.31.15.132.

strings ora_audit_0629.bin
ANG Spillover Audit File
ORAAUDNG
oracle
ip-172-31-11-54.eu-west-1.compute.internal
pts/0
(TYPE=(DATABASE));(CLIENT ADDRESS=((ADDRESS=(PROTOCOL=tcp)(HOST=172.31.11.54)(PORT=23983))));
LOCKTEST
sqlplus@ip-172-31-11-54.eu-west-1.compute.intern
10309
ORA_LOGON_FAILURES
LOCKTESTq
ORAAUDNG
oracle
ip-172-31-15-132.eu-west-1.compute.internal
pts/0
(TYPE=(DATABASE));(CLIENT ADDRESS=((ADDRESS=(PROTOCOL=tcp)(HOST=172.31.15.132)(PORT=30643))));
LOCKTEST
sqlplus@ip-172-31-15-132.eu-west-1.compute.inter
10478
ORA_LOGON_FAILURES
LOCKTEST

Therefore audit file must be queried via unified_audit_trail to get fine details of audit records.

If the unified audit table is queried on the standby DB (if it is open in read only mode) this will show records from both primary DB (available via redo apply) and from standby DB (earlier post on this causing high file waits).
To view the audit data of just the standby DB, the easiest and simplest way is to copy the required audit files (either from the CDB root or PDB location) to a different (test or temporary) database that doesn't have any audit data of its own and query the unified_audit_trail. The location where files are copied to must be similar to $ORACLE_BASE/audit/$ORACLE_SID or $ORACLE_BASE/audit/$ORACLE_SID/GUID. With audit files in above the pre-defined location, they are picked up automatically from their external location and exposed via the unified_audit_trail.
However, if the audit records must be kept in the DB then to improt the audit records into the database use the following.

EXEC DBMS_AUDIT_MGMT.LOAD_UNIFIED_AUDIT_FILES;

This will import all the audit files records in the pre-defined audit file location into the database. At the end of this the audit files are removed from the file system.

↧

Database Resident Connection Pool (DRCP) and Data Guard

May 29, 2021, 6:42 am

≫ Next: Cryptographic Checksum Mismatch Error on EM 13.4 Repository DB Alert Log

≪ Previous: Importing Unified Audit Data From Standby

There are few restrictions that must be considered when using DRCP in a data guard configuration. This post shows these restrictions in action in a physical data guard configuration.

Staring DRCP
The DRCP in standby cannot be started unless the DRCP in the primary is up. Trying to do so will result in an error similar to below.

SQL>  exec dbms_connection_pool.start_pool();
BEGIN dbms_connection_pool.start_pool(); END;

*
ERROR at line 1:
ORA-56501: DRCP: Pool startup failed
ORA-56501: DRCP: Pool startup failed
ORA-06512: at "SYS.DBMS_CONNECTION_POOL", line 4
ORA-06512: at line 1

Starting the DRCP in primary doesn't automatically start the DRCP in the standby. However, if dba_cpool_info is viewed it will show that pool is active. This is misleading. What this shows is the primary's DRCP status due to DG configuration. DRCP pool must be manually started on standby and one way to find out is to make a connection and see if connection. Another is to view on lsnrctl services for N000 and CMON processes.

Instance "testcdb3", status READY, has 2 handler(s) for this service...
    Handler(s):
"DEDICATED" established:6 refused:0 state:ready
         LOCAL SERVER
"N000" established:1 refused:0 current:0 max:40000 state:ready
CMON <machine: ip-172-31-11-54, pid: 6935>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=33362))

Stopping DRCP
Similar to starting, DRCP in the standby could only be stopped if the primary DRCP is stopped first. Trying to stop standby DRCP while the primary DRCP is up will result in an error similar to below.

SQL>  exec dbms_connection_pool.stop_pool();
BEGIN dbms_connection_pool.stop_pool(); END;

*
ERROR at line 1:
ORA-56506: DRCP: Pool shutdown failed
ORA-56506: DRCP: Pool shutdown failed
ORA-06512: at "SYS.DBMS_CONNECTION_POOL", line 16
ORA-06512: at line 1

Stopping the DRCP in primary doesn't stop the DRCP in standby. However, the dba_cpool_info will show pool inactive, simialr to described in starting DRCP section.

ONNECTION_POOL                STATUS
------------------------------ ---------
SYS_DEFAULT_CONNECTION_POOL    INACTIVE

This is misleading. The DRCP will remain open in the standby and new connections could be established. While primary DRCP is down , pool on standby could be stopped. However, once stopped it cannot be started again until primary is started as described earlier.

Auto Starting DRCP
If DRCP is active on the primary DB, then restaritng the primary DB will automatically will bring up the DRCP as well.
If DRCP is active on standby DB then restarting the standby DB will not automatically will bring up the pool even if pool on primary is also up. It has to be manually started after restart of the standby DB. This is due to bug 18116889.
One way to ensure DRCP always start is to use startup trigger similar to below.

create or replace trigger start_drcp after startup on database
begin
	sys.dbms_system.ksdwrt(2, 'Starting DRCP ...');
	sys.dbms_connection_pool.start_pool();
	sys.dbms_system.ksdwrt(2, 'DRCP started.');

end;
/

Startup trigger is also governed by earlier mentioned restrictions. For example starting standby while primary is down will not start the pool on the standny. If startup trigger is there it will give an error as below.

ORA-04088: error during execution of trigger 'SYS.START_DRCP'
ORA-56501: DRCP: Pool startup failed
ORA-56501: DRCP: Pool startup failed

Changing DRCP Configuration
DRCP related configuration values could only be changed in the primary. The new values take effect immediately on the primary but on the standby then DRCP would need a restart for changes to take effect. This means stopping and starting the primary pool as well since standby pool cannot be stopped while primary pool is up.
Below is an example where number of brokers is changed from 1 to 2.

ONNECTION_POOL                NUM_CBROK
------------------------------ ---------
SYS_DEFAULT_CONNECTION_POOL    1

EXECUTE DBMS_CONNECTION_POOL.ALTER_PARAM ('','NUM_CBROK',2);

ONNECTION_POOL                NUM_CBROK
------------------------------ ---------
SYS_DEFAULT_CONNECTION_POOL    2

Changes are immediately visible on the primary and could see two broker processes as well.

Instance "testcdb", status READY, has 3 handler(s) for this service...
    Handler(s):
"DEDICATED" established:2 refused:0 state:ready
         LOCAL SERVER
"N000" established:0 refused:0 current:0 max:40000 state:ready
         CMON <machine: ip-172-31-15-132, pid: 11969>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=15748))
"N001" established:0 refused:0 current:0 max:40000 state:ready
         CMON <machine: ip-172-31-15-132, pid: 12532>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=31368))

However, on the standby instance there will be only a single broker.

Instance "testcdb3", status READY, has 2 handler(s) for this service...
    Handler(s):
"DEDICATED" established:12 refused:0 state:ready
         LOCAL SERVER
"N000" established:3 refused:0 current:1 max:40000 state:ready
         CMON <machine: ip-172-31-11-54, pid: 9874>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=33761))

Restarting the DRCP on the standby result in additional broker being created.

Instance "testcdb3", status READY, has 3 handler(s) for this service...
    Handler(s):
"DEDICATED" established:12 refused:0 state:ready
         LOCAL SERVER
"N000" established:0 refused:0 current:0 max:40000 state:ready
         CMON <machine: ip-172-31-11-54, pid: 11119>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=26996))
"N001" established:0 refused:0 current:0 max:40000 state:ready
         CMON <machine: ip-172-31-11-54, pid: 11121>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=14744))

Switchover and DRCP
In a multi-standby configuration, switchover between a primary and standby doesn't effect the availabilty of the DRCP on other standby instances. For example if there are multiple terminal standbys working as part of a reader farm then primary switching roles will not have any effect on the DRCP in the reader farm.

↧

Cryptographic Checksum Mismatch Error on EM 13.4 Repository DB Alert Log

June 5, 2021, 6:00 am

≫ Next: Move PDB to a Different ASM Disk Group

≪ Previous: Database Resident Connection Pool (DRCP) and Data Guard

Following error was observed in the EM (EM 13.4) repository database alert log.

NI cryptographic checksum mismatch error: 12599.

  VERSION INFORMATION:
        TNS for Linux: Version 19.0.0.0.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production
  Version 19.6.0.0.0
  Time: 24-MAR-2020 13:54:03
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12599

TNS-12599: TNS:cryptographic checksum mismatch
    ns secondary err code: 12656
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
2020-03-24T13:54:03.565478+00:00

The sqlnet.ora in the Oracle home had the following parameters set which are related to encrypting while in transit.

SQLNET.ENCRYPTION_SERVER=required
SQLNET.ENCRYPTION_CLIENT=required

SQLNET.ENCRYPTION_TYPES_SERVER=aes256
SQLNET.ENCRYPTION_TYPES_CLIENT=aes256

SQLNET.CRYPTO_CHECKSUM_SERVER=required
SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER=(SHA1)

SQLNET.CRYPTO_CHECKSUM_CLIENT=required
SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT=(SHA1)

The error is differnt to when there's no common encryption or checksum between client and server. Following shows the output for such a case (output is same whether the mismatch is on encryption or checksum).

Fatal NI connect error 12650, connecting to:
 (ADDRESS=(PROTOCOL=tcp)(HOST=10.17.16.19)(PORT=62960))

  VERSION INFORMATION:
        TNS for Linux: Version 19.0.0.0.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production
  Version 19.8.0.0.0
  Time: 17-SEP-2020 14:46:49
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12650

TNS-12650: No common encryption or data integrity algorithm
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
opiodr aborting process unknown ospid (6796) as a result of ORA-609
2020-09-17T14:46:50.321241+05:30

MOS note 2332486.1 this is due to bug 26933408. As a solution it offers two patches, 26933408 for OMS (server side) and 31840839 for agent (client side). After the patches were applied the error message did appear any more.

Related Metalink Notes
TNS-12599: TNS:cryptographic Checksum Mismatch in alert.log after enabling of encryption on the server side [ID 1927120.1]
EM 13c: Enterprise Manager 13c Cloud Control Target Database Repeating Alert Log Errors: TNS-12599: TNS:cryptographic Checksum Mismatch [ID 2332486.1]

↧

Move PDB to a Different ASM Disk Group

June 12, 2021, 6:00 am

≫ Next: Installing Oracle Restart Without ASM

≪ Previous: Cryptographic Checksum Mismatch Error on EM 13.4 Repository DB Alert Log

This post shows the steps for moving PDB related data files from one disk group to another. Currently the PDB data files reside in a disk group called +DATA and data file path is

+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/

TESTCDB is the CDB name and 9CBA2DF91A8C7012E053F4071FAC36E9 is the PDBs GUID.
The PDB will be moved to a ASM disk group called PDBDG.
1. Run a RMAN backup as copy to create data files copies in the destination diskgroup.

RMAN>backup as copy pluggable database testpdb2 format '+pdbdg';

Starting backup at 08-FEB-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=141 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=877 device type=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: SID=13 device type=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: SID=137 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00045 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/system.301.1063988727
channel ORA_DISK_2: starting datafile copy
input datafile file number=00046 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/sysaux.300.1063988727
channel ORA_DISK_3: starting datafile copy
input datafile file number=00047 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/undotbs1.299.1063988727
channel ORA_DISK_4: starting datafile copy
input datafile file number=00049 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/oradbaudit.304.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/sysaux.257.1063989107 tag=TAG20210208T163145 RECID=1 STAMP=1063989121
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_2: starting datafile copy
input datafile file number=00050 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/box.303.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/system.256.1063989107 tag=TAG20210208T163145 RECID=3 STAMP=1063989121
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:16
channel ORA_DISK_1: starting datafile copy
input datafile file number=00051 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/lobs.306.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/box.260.1063989121 tag=TAG20210208T163145 RECID=5 STAMP=1063989122
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_2: starting datafile copy
input datafile file number=00052 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/indexes.309.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/undotbs1.258.1063989111 tag=TAG20210208T163145 RECID=4 STAMP=1063989121
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:16
channel ORA_DISK_3: starting datafile copy
input datafile file number=00053 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/repos.308.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/oradbaudit.259.1063989117 tag=TAG20210208T163145 RECID=2 STAMP=1063989121
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:00:18
channel ORA_DISK_4: starting datafile copy
input datafile file number=00054 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/audits.307.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/lobs.261.1063989123 tag=TAG20210208T163145 RECID=6 STAMP=1063989124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting datafile copy
input datafile file number=00055 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/audindexes.310.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/indexes.262.1063989123 tag=TAG20210208T163145 RECID=7 STAMP=1063989124
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_2: starting datafile copy
input datafile file number=00056 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/audlobs.282.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/repos.263.1063989125 tag=TAG20210208T163145 RECID=8 STAMP=1063989125
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_3: starting datafile copy
input datafile file number=00057 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/tbs.280.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/audits.264.1063989125 tag=TAG20210208T163145 RECID=9 STAMP=1063989125
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:00:02
channel ORA_DISK_4: starting datafile copy
input datafile file number=00058 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/cheindexes.281.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/audindexes.265.1063989125 tag=TAG20210208T163145 RECID=10 STAMP=1063989127
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting datafile copy
input datafile file number=00059 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/abs.283.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/audlobs.266.1063989127 tag=TAG20210208T163145 RECID=11 STAMP=1063989127
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_2: starting datafile copy
input datafile file number=00060 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/nstbs.297.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/tbs.267.1063989127 tag=TAG20210208T163145 RECID=12 STAMP=1063989128
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_3: starting datafile copy
input datafile file number=00061 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/lkstbs.298.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/cheindexes.268.1063989129 tag=TAG20210208T163145 RECID=13 STAMP=1063989129
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:00:02
channel ORA_DISK_4: starting datafile copy
input datafile file number=00048 name=+DATA/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/users.305.1063988727
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/abs.269.1063989129 tag=TAG20210208T163145 RECID=14 STAMP=1063989129
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:02
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/nstbs.270.1063989129 tag=TAG20210208T163145 RECID=15 STAMP=1063989130
channel ORA_DISK_2: datafile copy complete, elapsed time: 00:00:02
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/lkstbs.271.1063989131 tag=TAG20210208T163145 RECID=16 STAMP=1063989131
channel ORA_DISK_3: datafile copy complete, elapsed time: 00:00:02
output file name=+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/users.272.1063989131 tag=TAG20210208T163145 RECID=17 STAMP=1063989131
channel ORA_DISK_4: datafile copy complete, elapsed time: 00:00:01
Finished backup at 08-FEB-21

Starting Control File and SPFILE Autobackup at 08-FEB-21
piece handle=+FRA/TESTCDB/AUTOBACKUP/2021_02_08/s_1063989132.284.1063989133 comment=NONE
Finished Control File and SPFILE Autobackup at 08-FEB-21

2. As system tablespaces are also moved the switch to new location cannot be done while PDB is open.

RMAN> switch pluggable database testpdb2 to copy;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of switch to copy command at 02/08/2021 16:32:29
RMAN-06572: database is open and datafile 45 is not offline

So the PDB must be closed (downtime) and then switch to the data file copies.

RMAN> alter pluggable database testpdb2 close;

Statement processed

RMAN>  switch pluggable database testpdb2 to copy;

datafile 45 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/system.256.1063989107"
datafile 46 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/sysaux.257.1063989107"
datafile 47 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/undotbs1.258.1063989111"
datafile 48 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/users.272.1063989131"
datafile 49 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/oradbaudit.259.1063989117"
datafile 50 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/box.260.1063989121"
datafile 51 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/lobs.261.1063989123"
datafile 52 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/indexes.262.1063989123"
datafile 53 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/repos.263.1063989125"
datafile 54 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/audits.264.1063989125"
datafile 55 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/audindexes.265.1063989125"
datafile 56 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/audlobs.266.1063989127"
datafile 57 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/tbs.267.1063989127"
datafile 58 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/cheindexes.268.1063989129"
datafile 59 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/abs.269.1063989129"
datafile 60 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/nstbs.270.1063989129"
datafile 61 switched to datafile copy "+PDBDG/TESTCDB/9CBA2DF91A8C7012E053F4071FAC36E9/DATAFILE/lkstbs.271.1063989131"

3. Recover the PDB

RMAN> recover pluggable database testpdb2;

Starting recover at 08-FEB-21
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 08-FEB-21

4. Open the PDB


RMAN> alter pluggable database testpdb2 open;

Statement processed

Related Posts
Moving 11gR2 RAC to New Set of ASM Diskgroups
Moving non-RAC Database and ASM Between Servers

↧

Installing Oracle Restart Without ASM

June 19, 2021, 6:00 am

≫ Next: Enabling SSL_DH_anon_WITH_3DES_EDE_CBC_SHA to work with TCPS/SSL for JDBC Thin Drvier

≪ Previous: Move PDB to a Different ASM Disk Group

By default Oracle resstart requires at least one ASM disk group to be created during the installation. This post shows steps for installing Oracle restart without creating any ASM disk group.
1. First step in this process is to install the grid infrastructure using the "software only" option. If OUI is used this will be the "Set up Software Only" option. For this post the test steup was done using the grid response file. The response file indicate which section to fill to do a software only instllation.

## To register software for 'Grid Infrastructure'                            ##
##  - Fill out sections A,B and D                                            ##
##  - Provide the cluster nodes in section D when choosing CRS_SWONLY as     ##
##    installation option in section A                                       ##

Section A consists of information for oraIneventory, oracle base and installation option.

#-------------------------------------------------------------------------------
# Specify the location which holds the inventory files.
# This is an optional parameter if installing on
# Windows based Operating System.
#-------------------------------------------------------------------------------
INVENTORY_LOCATION=/opt/app/oraInventory

#-------------------------------------------------------------------------------
# Specify the installation option.
# Allowed values: CRS_CONFIG or HA_CONFIG or UPGRADE or CRS_SWONLY or HA_SWONLY
#   - CRS_CONFIG  : To register home and configure Grid Infrastructure for cluster
#   - HA_CONFIG   : To register home and configure Grid Infrastructure for stand alone server
#   - UPGRADE     : To register home and upgrade clusterware software of earlier release
#   - CRS_SWONLY  : To register Grid Infrastructure Software home (can be configured for cluster
#                   or stand alone server later)
#   - HA_SWONLY   : To register Grid Infrastructure Software home (can be configured for stand
#                   alone server later. This is only supported on Windows.)
#   - CRS_ADDNODE : To add more nodes to the cluster
#   - CRS_DELETE_NODE : To delete nodes to the cluster
#-------------------------------------------------------------------------------
oracle.install.option=CRS_SWONLY

#-------------------------------------------------------------------------------
# Specify the complete path of the Oracle Base.
#-------------------------------------------------------------------------------
ORACLE_BASE=/opt/app/oracle

Section B specify ASM related privileged user groups. Even though ASM is not configured in this case, it is possible to use the same set of user group that would be used in a ASM setup. If not setup the group appropriatly for the enviornment being setup.

################################################################################
#                                                                              #
#                              SECTION B - GROUPS                              #
#                                                                              #
#   The following three groups need to be assigned for all GI installations.   #
#   OSDBA and OSOPER can be the same or different.  OSASM must be different    #
#   than the other two.                                                        #
#   The value to be specified for OSDBA, OSOPER and OSASM group is only for    #
#   Unix based Operating System.                                               #
#   These groups are not required for upgrades, as they will be determined     #
#   from the Oracle home to upgrade.                                           #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
# The OSDBA_GROUP is the OS group which is to be granted SYSDBA privileges.
#-------------------------------------------------------------------------------
oracle.install.asm.OSDBA=asmdba

#-------------------------------------------------------------------------------
# The OSOPER_GROUP is the OS group which is to be granted SYSOPER privileges.
# The value to be specified for OSOPER group is optional.
# Value should not be provided if configuring Client Cluster - i.e. storageOption=CLIENT_ASM_STORAGE.
#-------------------------------------------------------------------------------
oracle.install.asm.OSOPER=asmoper

#-------------------------------------------------------------------------------
# The OSASM_GROUP is the OS group which is to be granted SYSASM privileges. This
# must be different than the previous two.
#-------------------------------------------------------------------------------
oracle.install.asm.OSASM=asmadmin

Section D has many parameters but for Oracle restart setup only the following is needed to be filled, which specify the hostname.

################################################################################
#                                                                              #
#                           SECTION D - CLUSTER & GNS                         #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
#
#-------------------------------------------------------------------------------
oracle.install.crs.config.clusterNodes=ip-172-31-7-187

#-------------------------------------------------------------------------------

2. Install the grid infrastructure using the response file.

./gridSetup.sh -silent -responseFile grid.rsp

3. At the end of the installation the grid infrastructure would be configured to an Oracle restart configuration by running roothas.sh in GI_HOME/crs/install as the root user.

./roothas.sh
Using configuration parameter file: /opt/app/oracle/product/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-12-240/crsconfig/roothas_2021-02-15_04-09-40PM.log
2021/02/15 16:09:41 CLSRSC-363: User ignored prerequisites during installation
Redirecting to /bin/systemctl restart rsyslog.service
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-12-240 successfully pinned.
2021/02/15 16:10:08 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'

ip-172-31-12-240     2021/02/15 16:12:24     /opt/app/oracle/crsdata/ip-172-31-12-240/olr/backup_20210215_161224.olr     1944883066
2021/02/15 16:12:25 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

4. At this stage the resource states would be as below

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.cssd                  ora.cssd.type             OFFLINE            OFFLINE
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-7-187
ora.ons                   ora.ons.type              OFFLINE            OFFLINE

It is important to have the cssd status online. Without it, when managing the databases following errors is thrown

$ srvctl start database -db testfs
PRCD-1024 : Failed to retrieve instance list for database testfs
PRCR-1055 : Cluster membership check failed for node ip-172-31-12-240

5. Enable the cssd to auto start and restart HAS.

crsctl modify resource "ora.cssd" -attr "AUTO_START=1" -unsupported
crsctl stop has
crsctl start has -nowait

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             ip-172-31-7-187
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-7-187
ora.ons                   ora.ons.type              OFFLINE            OFFLINE

6. Add listener

srvctl add listener -endpoints tcp:1521
srvctl start listener

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             ip-172-31-7-187
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             ip-172-31-7-187
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-7-187
ora.ons                   ora.ons.type              OFFLINE            OFFLINE

7. Enable ONS if this is used for data guard and client failover is setup.

srvctl enable ons
srvctl start ons

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             ip-172-31-7-187
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             ip-172-31-7-187
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-7-187
ora.ons                   ora.ons.type              ONLINE             ONLINE             ip-172-31-7-187

8. Create the CDB. File system is given as the storage type for CDB and two directories are specified for datafileDestination and recoveryAreaDestination. These gets set to db_create_file_dest and db_recovery_file_dest since OMF is set to true. The CDB will be automatically registered with the HAS service and could be managed with srvctl commands.

dbca -createDatabase -gdbName testfs -templateName /home/oracle/New_Database.dbt 
-characterSet AL32UTF8 -emConfiguration DBEXPRESS 
-storageType FS -datafileDestination  /opt/data  
-recoveryAreaDestination /opt/fra 
-sysPassword testCDB1234 -systemPassword testCDB1234 
-createAsContainerDatabase true 
-memoryMgmtType AUTO_SGA  -enableArchive false 
-useOMF true  -nationalCharacterSet AL16UTF16 
-databaseConfigType SINGLE -silent

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             ip-172-31-7-187
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             ip-172-31-7-187
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-7-187
ora.ons                   ora.ons.type              ONLINE             ONLINE             ip-172-31-7-187
ora.testfs.db             ora.database.type         ONLINE             ONLINE             ip-172-31-7-187

9. Create the PDB

dbca -silent -createPluggableDatabase -pdbName testpdb -sourceDB $ORACLE_SID -createUserTableSpace true -pdbAdminPassword adminPBD123

10. If a data guard configuraiton is created from a similar setup and data guard broker is created then it will detect the presence of clusterware.

DGMGRL> validate database testfs2

  Database Role:     Physical standby database
  Primary Database:  testfs

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

Managed by Clusterware:
testfs :  YES
    testfs2:  YES

↧

Enabling SSL_DH_anon_WITH_3DES_EDE_CBC_SHA to work with TCPS/SSL for JDBC Thin Drvier

June 28, 2021, 8:12 am

≫ Next: ORA-12696: Double Encryption Turned On Even When IGNORE_ANO_ENCRYPTION_FOR_TCPS=TRUE is Set

≪ Previous: Installing Oracle Restart Without ASM

While testing out "Connect to the database through TCPS for SSL with Encryption Only" on 762286.1 encournted the following error.

java -Doracle.net.tns_admin=. JDBCSSLTester2 test2.properties
Start: Wed Jun 23 12:50:02 UTC 2021
SQL Exception occurred:
java.sql.SQLRecoverableException: IO Error: No appropriate protocol (protocol is disabled or cipher suites are inappropriate), Authentication lapse 0 ms.
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:894)
        at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:807)
        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:77)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:767)
        at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:450)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:324)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:234)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:212)
        at JDBCSSLTester2.getConnection(JDBCSSLTester2.java:74)
        at JDBCSSLTester2.run(JDBCSSLTester2.java:34)
        at JDBCSSLTester2.main(JDBCSSLTester2.java:88)
Caused by: java.io.IOException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate), Authentication lapse 0 ms.
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:890)
        ... 10 more
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
        at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171)
        at sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:101)
        at sun.security.ssl.TransportContext.kickstart(TransportContext.java:221)
        at sun.security.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:98)
        at oracle.net.nt.SSLSocketChannel.doSSLHandshake(SSLSocketChannel.java:430)
        at oracle.net.nt.SSLSocketChannel.write(SSLSocketChannel.java:130)
        at oracle.net.ns.NIOPacket.writeToSocketChannel(NIOPacket.java:355)
        at oracle.net.ns.NIOConnectPacket.writeToSocketChannel(NIOConnectPacket.java:247)
        at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:122)
        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:364)
        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1625)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:606)
        ... 10 more
Ended: Wed Jun 23 12:50:03 UTC 2021

This was intresting as the SQLPlus connection worked fine. So the issue is localized to java. For SSL_DH_anon_WITH_3DES_EDE_CBC_SHA to work with JDBC the cipher suite must be added to both sqlnet.ora and listener.ora (2621754.1, 1434966.1). As these were already in place in both those files missing chipher suite cannot be the reason for this.

The intersting part from error stack was "protocol is disabled or cipher suites are inappropriate". Seems cipher suite SSL_DH_anon_WITH_3DES_EDE_CBC_SHA is not available for java. It's available on Oracle as per security guide. Oracle advices not to use these cipher suites to protect sensitive data. But they are useful in situatation where only encryption of traffic is needed not authentication or if communicating parties want to remain anonymous.

MOS doc 2288489.1 listed similar issue with regard to using Diffie-Hellman on JDK 1.7. It did have a link to external doc which listed enchancements on JDK 1.8 but did not help in resolving this issue.

However, it seems the DH_anon cipher suites used in 762286.1 for encryption only test case (SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_RC4_128_MD5,SSL_DH_anon_WITH_DES_CBC_SHA) seem to be indeed disable by default on 1.8. It is mentioned here"For users of Oracle 11g, the SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_RC4_128_MD5, and SSL_DH_anon_WITH_DES_CBC_SHA cipher suites are disabled by default in Java 8. To allow these cipher suites, see the Test or Revert changes to Oracle's JDK and JRE Cryptographic Algorithms section of the Java documentation".

The test case was run using JDK1.8 and 19.11.0.0.0 driver. Using SSLServerSocketFactory is possible to iterate over available cipher suites and default cipher suites.

SSLServerSocketFactory ssf = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
String[] defaultCiphers = ssf.getDefaultCipherSuites();
String[] availableCiphers = ssf.getSupportedCipherSuites();

This showed following list of cipher suites available by default

TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_256_GCM_SHA384

SSL_DH_anon_WITH_3DES_EDE_CBC_SHA was missing from the list.

Document here shows how to add algorithm to disable list. So to enable then it must be taken out of the disabled list. By deafult $JAVA_HOME/jre/lib/security/java.security has the following for jdk.tls.disabledAlgorithms entry.

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
    include jdk.disabled.namedCurves

To enable DH_anon remove "3DES_EDE_CBC, anon". So the udpate entry looks like below.

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, NULL, \
    include jdk.disabled.namedCurves

The new cipher suite list is shown below

SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
SSL_RSA_WITH_3DES_EDE_CBC_SHA
TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DH_anon_WITH_AES_128_CBC_SHA
TLS_DH_anon_WITH_AES_128_CBC_SHA256
TLS_DH_anon_WITH_AES_128_GCM_SHA256
TLS_DH_anon_WITH_AES_256_CBC_SHA
TLS_DH_anon_WITH_AES_256_CBC_SHA256
TLS_DH_anon_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_anon_WITH_AES_128_CBC_SHA
TLS_ECDH_anon_WITH_AES_256_CBC_SHA
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
TLS_KRB5_WITH_3DES_EDE_CBC_MD5
TLS_KRB5_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_256_GCM_SHA384

SSL_DH_anon_WITH_3DES_EDE_CBC_SHA is available now. The other two *DH_anon_* cipher suite listed on 762286.1 seem to be missing from 19c (as it is not listed on security guide, possibly desupported in newer versions). Only SSL_DH_anon_WITH_3DES_EDE_CBC_SHA is listed on security guide as a possible cipher suite for encryption only. So getting SSL_DH_anon_WITH_3DES_EDE_CBC_SHA available is enough to run the test case mentioned on 762286.1.

java -Doracle.net.tns_admin=. JDBCSSLTester2 test2.properties
Start: Wed Jun 23 13:47:47 UTC 2021
Conncted as DATABASE USER ASANGA
Ended: Wed Jun 23 13:47:49 UTC 2021

There's a good reason why certain cipher suits are disabled by default. Enabling and using them must be carefully considerd inline with security policies being used.

↧

ORA-12696: Double Encryption Turned On Even When IGNORE_ANO_ENCRYPTION_FOR_TCPS=TRUE is Set

June 28, 2021, 8:49 am

≫ Next: Enabling SSL_DH_anon_WITH_3DES_EDE_CBC_SHA to work with TCPS/SSL for JDBC Thin Drvier

≪ Previous: Enabling SSL_DH_anon_WITH_3DES_EDE_CBC_SHA to work with TCPS/SSL for JDBC Thin Drvier

The error "ORA-12696: Double Encryption Turned On, login disallowed" is an expected one when both SSL and Native encryption (ANO) is enabled. By default Oracle doesn't allow both encryption types and return the above error.

However, there is a parameter that could be set so that ANO is ignored for TCPS connections. The parameter is called IGNORE_ANO_ENCRYPTION_FOR_TCPS and setting this to TRUE would allow both TCP with ANO and TCPS connections to be used concurrently.
While settting up this configuration the connections for TCPS were failing with ORA-12696. The parameter IGNORE_ANO_ENCRYPTION_FOR_TCPS could be set on either sqlnet.ora or in the TNS alias. No matter where it was set, the TCPS connections kept getting the above error. It was puzzling as the documentation was followed to the letter.

Apparently the issue is with documetnation. The parameter is actaully called "SQLNET.IGNORE_ANO_ENCRYPTION_FOR_TCPS" (visible on the step 4 below). However the documentation where it shows how to set it and helpful copy buttons all ignore the "SQLNET." part. See below for current documentation at the time of this blog post.

MOS Doc 2614143.1 which also address the same issue shows it being set as IGNORE_ANO_ENCRYPTION_FOR_TCPS=TRUE (though the sqlnet.ora content shown in the same MOS has it set correctly).
SR was raised to correct the documentation so that parameter is reflected correctly with the prefix SQLNET. similar to other parameters such as SQLNET.ENCRYPTION_CLIENT, SQLNET.ENCRYPTION_TYPES_CLIENT and etc which all has been documented with SQLNET. prefix.
Only place this is correctly reflected (time of this blog) is in net services reference guide (which helped to identify the root cause of the issue).
When setting this on the TNS alias then the parameter could be with sqlnet. prefix or without it(documentation is correct for this setting).

↧

Enabling SSL_DH_anon_WITH_3DES_EDE_CBC_SHA to work with TCPS/SSL for JDBC Thin Drvier

July 3, 2021, 6:00 pm

≫ Next: ORA-00800: soft external error, arguments: [Set Priority Failed], [VKTM], [Check traces and OS configuration]

≪ Previous: ORA-12696: Double Encryption Turned On Even When IGNORE_ANO_ENCRYPTION_FOR_TCPS=TRUE is Set

While testing out "Connect to the database through TCPS for SSL with Encryption Only" on 762286.1 encournted the following error.

java -Doracle.net.tns_admin=. JDBCSSLTester2 test2.properties
Start: Wed Jun 23 12:50:02 UTC 2021
SQL Exception occurred:
java.sql.SQLRecoverableException: IO Error: No appropriate protocol (protocol is disabled or cipher suites are inappropriate), Authentication lapse 0 ms.
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:894)
        at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:807)
        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:77)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:767)
        at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:450)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:324)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:234)
        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:212)
        at JDBCSSLTester2.getConnection(JDBCSSLTester2.java:74)
        at JDBCSSLTester2.run(JDBCSSLTester2.java:34)
        at JDBCSSLTester2.main(JDBCSSLTester2.java:88)
Caused by: java.io.IOException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate), Authentication lapse 0 ms.
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:890)
        ... 10 more
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
        at sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:171)
        at sun.security.ssl.ClientHandshakeContext.<init>(ClientHandshakeContext.java:101)
        at sun.security.ssl.TransportContext.kickstart(TransportContext.java:221)
        at sun.security.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:98)
        at oracle.net.nt.SSLSocketChannel.doSSLHandshake(SSLSocketChannel.java:430)
        at oracle.net.nt.SSLSocketChannel.write(SSLSocketChannel.java:130)
        at oracle.net.ns.NIOPacket.writeToSocketChannel(NIOPacket.java:355)
        at oracle.net.ns.NIOConnectPacket.writeToSocketChannel(NIOConnectPacket.java:247)
        at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:122)
        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:364)
        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1625)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:606)
        ... 10 more
Ended: Wed Jun 23 12:50:03 UTC 2021

SSLServerSocketFactory ssf = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
String[] defaultCiphers = ssf.getDefaultCipherSuites();
String[] availableCiphers = ssf.getSupportedCipherSuites();

This showed following list of cipher suites available by default

TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_256_GCM_SHA384

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
    include jdk.disabled.namedCurves

To enable DH_anon remove "3DES_EDE_CBC, anon". So the udpate entry looks like below.

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, NULL, \
    include jdk.disabled.namedCurves

The new cipher suite list is shown below

SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
SSL_RSA_WITH_3DES_EDE_CBC_SHA
TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DH_anon_WITH_AES_128_CBC_SHA
TLS_DH_anon_WITH_AES_128_CBC_SHA256
TLS_DH_anon_WITH_AES_128_GCM_SHA256
TLS_DH_anon_WITH_AES_256_CBC_SHA
TLS_DH_anon_WITH_AES_256_CBC_SHA256
TLS_DH_anon_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_anon_WITH_AES_128_CBC_SHA
TLS_ECDH_anon_WITH_AES_256_CBC_SHA
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
TLS_KRB5_WITH_3DES_EDE_CBC_MD5
TLS_KRB5_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_256_GCM_SHA384

java -Doracle.net.tns_admin=. JDBCSSLTester2 test2.properties
Start: Wed Jun 23 13:47:47 UTC 2021
Conncted as DATABASE USER ASANGA
Ended: Wed Jun 23 13:47:49 UTC 2021

There's a good reason why certain cipher suits are disabled by default. Enabling and using them must be carefully considerd inline with security policies being used.

↧

ORA-00800: soft external error, arguments: [Set Priority Failed], [VKTM], [Check traces and OS configuration]

July 10, 2021, 6:30 pm

≫ Next: Accessing OCI Bucket Using Instance Principal

≪ Previous: Enabling SSL_DH_anon_WITH_3DES_EDE_CBC_SHA to work with TCPS/SSL for JDBC Thin Drvier

While adding a new standby node to the existing DG configuration noticed the following in the new standby databases's alert log.

Starting background process VKTM
2021-03-30T00:08:42.781044+10:00
Errors in file /opt/app/oracle/diag/rdbms/dbx12/dbx12/trace/dbx12_vktm_32410.trc  (incident=41):
ORA-00800: soft external error, arguments: [Set Priority Failed], [VKTM], [Check traces and OS configuration], [Check Oracle document and MOS notes], []
Incident details in: /opt/app/oracle/diag/rdbms/dbx12/dbx12/incident/incdir_41/dbx12_vktm_32410_i41.trc
2021-03-30T00:08:42.782979+10:00
Error attempting to elevate VKTM's priority: no further priority changes will be attempted for this process
VKTM started with pid=5, OS id=32410

MOS Doc 2718971.1 gives a workaround for this issue (seems this doc has now been made internal).
The problme was related to cgroup setup. In a database setup where everythig is working fine the cgroup for the VKTM would have /. For example if VKTM PID is 5207 then following could be be used to find otu cgroup setting

cat /proc/5207/cgroup | grep cpu
10:cpu,cpuacct:/
6:cpuset:/

Any other setting would mean VKTM would run into above issue.
One of the solutions is to set the hidden parameter _high_priority_processes='VKTM'. But this was already set to TRUE so no going to be a solution.
The problem server had the cgroup setting as following.

# ps -eaf | grep -i vktm | grep -v grep
oracle    3315     1  0 17:53 ?        00:00:00 ora_vktm_dbx12
oracle    3357     1  0 14:19 ?        00:01:40 asm_vktm_+ASM
# cat /proc/3315/cgroup | grep cpu
11:cpuset:/
6:cpu,cpuacct:/user.slice

So the next workround was to set the kernel parameter as below.

# echo 0 > /sys/fs/cgroup/cpu,cpuacct/system.slice/cpu.rt_runtime_us
# echo 950000 > /sys/fs/cgroup/cpu,cpuacct/user.slice/cpu.rt_runtime_us

This seem to resolve the situation and was able stop and start the standby instance without the above error message appaering on the alert log.

However, the question reamin why did the cgroup change. This was the first incident of facing this error. Since this is adding a standby database server to exisitng setup there was a reference point to check against any changes. So first the OS. The "good" servers had OEL 7.9 while the "problem" server had OEL 7.7. Both servers are Azure VMs
Then decided to check inside cgroup setting. The good server had following (no user.slice)

 ls -l /sys/fs/cgroup/cpu,cpuacct/
drwxr-xr-x. 3 root root 0 Apr  7 09:32 WALinuxAgent
-rw-r--r--. 1 root root 0 Apr  7 09:32 tasks
-rw-r--r--. 1 root root 0 Apr  7 09:32 cgroup.procs
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.cfs_period_us
-r--r--r--. 1 root root 0 Apr  7 09:32 cgroup.sane_behavior
-r--r--r--. 1 root root 0 Apr  7 09:32 cpu.stat
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu_sys
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.shares
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.stat
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.cfs_quota_us
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_sys
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_all
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu_user
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Apr  7 09:32 notify_on_release
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Apr  7 09:32 release_agent
-rw-r--r--. 1 root root 0 Apr  7 09:32 cgroup.clone_children
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_user

While the problem server had the following

drwxr-xr-x.  2 root root 0 Apr  7 09:29 auoms
drwxr-xr-x.  2 root root 0 Apr  7 09:29 auomscollect
-rw-r--r--.  1 root root 0 Apr  7 09:29 cgroup.clone_children
-rw-r--r--.  1 root root 0 Apr  7 09:29 cgroup.procs
-r--r--r--.  1 root root 0 Apr  7 09:29 cgroup.sane_behavior
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.stat
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_all
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_percpu
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_percpu_sys
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_percpu_user
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_sys
-r--r--r--.  1 root root 0 Apr  7 09:29 cpuacct.usage_user
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.cfs_period_us
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.cfs_quota_us
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.rt_period_us
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.rt_runtime_us
-rw-r--r--.  1 root root 0 Apr  7 09:29 cpu.shares
-r--r--r--.  1 root root 0 Apr  7 09:29 cpu.stat
-rw-r--r--.  1 root root 0 Apr  7 09:29 notify_on_release
-rw-r--r--.  1 root root 0 Apr  7 09:29 release_agent
drwxr-xr-x. 69 root root 0 Apr  7 09:29 system.slice
-rw-r--r--.  1 root root 0 Apr  6 14:35 tasks
drwxr-xr-x.  2 root root 0 Apr  7 09:29 user.slice
drwxr-xr-x.  2 root root 0 Apr  7 09:29 WALinuxAgent

Beside user.slice the auoms* seem to be the different between the two. Auoms is Azure management agent plugin. Could this be the reason why cgroup has a user.slice? To test this out disableed auoms and restarted the server.

# systemctl stop auoms.service
# systemctl disable auoms.service
Removed symlink /etc/systemd/system/multi-user.target.wants/auoms.service.
Removed symlink /etc/systemd/system/auoms.service.
# /sbin/reboot

When the server restart the cgroup didn't have a user.slice

drwxr-xr-x. 3 root root 0 Apr  7 09:32 WALinuxAgent
-rw-r--r--. 1 root root 0 Apr  7 09:32 tasks
-rw-r--r--. 1 root root 0 Apr  7 09:32 cgroup.procs
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.cfs_period_us
-r--r--r--. 1 root root 0 Apr  7 09:32 cgroup.sane_behavior
-r--r--r--. 1 root root 0 Apr  7 09:32 cpu.stat
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu_sys
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.shares
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.stat
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.cfs_quota_us
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_sys
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_all
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_percpu_user
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Apr  7 09:32 notify_on_release
-rw-r--r--. 1 root root 0 Apr  7 09:32 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Apr  7 09:32 release_agent
-rw-r--r--. 1 root root 0 Apr  7 09:32 cgroup.clone_children
-r--r--r--. 1 root root 0 Apr  7 09:32 cpuacct.usage_user

DB was starting without the VKTM related error on alert log (earlier set kernel parameters were not set to be persistent). VKTM had the / for cgroup.

 ps ax  | grep vktm
 3314 ?        Ss     0:03 asm_vktm_+ASM
 3506 ?        Ss     0:03 ora_vktm_dbx12
 4664 pts/0    S+     0:00 grep --color=auto vktm

# cat /proc/3506/cgroup | grep cpu
6:cpu,cpuacct:/
4:cpuset:/

There were several instances to be added to the DG and on other servers the OS was upgraded to 7.9 from 7.7. This upgrade seem to be remove the auoms and there were no cgroup issues.
If facing similar issues first check what has caused the cgroup change before attempting hidden parameter or kernel parameter workarounds.

↧

Accessing OCI Bucket Using Instance Principal

July 29, 2021, 7:56 am

≫ Next: Getting DB Passwords from Vault Secrets for OKE Deployments

≪ Previous: ORA-00800: soft external error, arguments: [Set Priority Failed], [VKTM], [Check traces and OS configuration]

This post shows how to access a OCI bucket using instance principal. Being able to access a bucket based on instance pricipal is useful as it eliminate the need for sharing passwords. Access to OCI storage buckets may be needed for example when importing a data dump into ATP. With instance princiapl a compute instance could be designated as the uploader of the dump files to bucket which could be imported into ATP later.
1. If not already done create a storage bucket.

2. Have a compute instance that would be used to create the instance principal

3. If the compute instance is in a private subnet then create a service gateway and have a rourting rule to sending service related traffic through it.

4. Create a dynamic group using the compute instance's ID as the value to match.

5. Creaet a policy allowing the dynamic group to read the bucket and manage the objects in the bucket.

6. Install OCI client. This only needs to be installed. No need to configure it. If the compute instance is OEL then this would be done via yum.

7. To check if the instance principal works and bucket is accessible, set the OCI_CLI_AUTH=instance_principal and run a list command against tbe bucket using oci cli. Specify the namespace of the OCI tenancy after -ns (omitted in screenshow below).

Output above shows the content of the bucket. It contain just one file called asanga.dmp.
Another way to specify the authentication in the oci cli is to use the --auth. This eliminate the need to set an environment variable.

↧

Getting DB Passwords from Vault Secrets for OKE Deployments

July 31, 2021, 6:30 pm

≫ Next: Unable to Set UNDO_RETENTION in Standby PDB

≪ Previous: Accessing OCI Bucket Using Instance Principal

OCI allows storing of DB passwords as secrets in the vault. These secrets could be retreived using various SDKs, CLI and etc. This post shows how DB password could be retreived from a vault for JDBC Connection pools when a java application is deployed in OKE.
1. In order to retreivew the secrets from vault the user making the request must be authenticated. Instance principal is used to avoid using a password based authentication for this. OCI allows dynamic group be created based compartment, instance id, tag and tag value. As a first step create a dynamic group specifying the compartment where OKE worker node resides and some qualifying tags. The dynamic group is called "test dynamic group".

Compartment is used instead of instance ID in the dynamic group creation. This is due the fact that new worker nodes could be created and old ones destroyed as part of the life cycle of the OKE cluster. Tag values have been used to further reduce the number of instances that qualify for the dynamic group. All worker nodes would have "oke" as the created-by tag value. So this tag could be used to distinguish between OKE cluster service created instances vs other compute instnaces. Further reduction could be made based on project, enviornment and etc.
2. Create the secrets in the vault. Secrets could be created with a prefix which would allow writing of policy capturing only the secrets with the specified prefix text. Below two secrets have been created both with prefix "acme_test_prod".

3. Write a policy allowing dynamic group to access the secret bundles. Use the vault id and prefix of the secret to restrict the dynamic group to specific set of secrets and vaults. Below policy would allow test_dynamic_group to get all the secret bundles with names begining with "acme_test_prod" in the specified vault id.

Policy is written for secret bundles as that's what the java API expect. If this is done for OCI CLI then policy could be written for secrets instead of secret bundles. Java API access fails when policy only allow access to secrets instead of secret bundles.

4. Final step set is to write the java code that would be deployed as part of the application into OKE cluster. Download the java SDK from the link here. Below is an example java code that uses instance principal provider to create a secrets client that could be used to retreive the secrets from the vault.

final InstancePrincipalsAuthenticationDetailsProvider provider;
        try {
            provider = InstancePrincipalsAuthenticationDetailsProvider.builder().build();
        } catch (Exception e) {

            throw e;
        }

SecretsClient secretsDpClient  = new SecretsClient(provider);

        GetSecretBundleByNameResponse getSecretBundleByNameResponse = secretsDpClient.getSecretBundleByName(GetSecretBundleByNameRequest.builder()
        .secretName("acme_test_prod_schema1").vaultId("ocid1.vault.oc1.vault id here...").build());

        Base64SecretBundleContentDetails details = (Base64SecretBundleContentDetails) getSecretBundleByNameResponse.getSecretBundle().getSecretBundleContent();
        byte[] content = Base64.getDecoder().decode(details.getContent());
        //System.out.println("Password : "new String(content));

        PoolDataSource ds = PoolDataSourceFactory.getPoolDataSource();
        ds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
        ds.setURL("jdbc:oracle:thin:@test");
        ds.setUser("asanga");
        ds.setPassword(new String(content));

↧

Unable to Set UNDO_RETENTION in Standby PDB

August 7, 2021, 6:30 pm

≫ Next: ORA-16713: The Oracle Data Guard broker command timed out When Changing LogXptMode

≪ Previous: Getting DB Passwords from Vault Secrets for OKE Deployments

The fix 30577591 which is included in the DBRU 19.9 changed how the undo_retention parameter value is set in a CDB. Prior to this the value was set at CDB root level and all PDBs inherited this value. But with this change this is not the case. The value set at CDB root is only assigned to CDB root. For PDB this must be explicityly set either by login into the PDB or using "container=all" clause if setting at CDB root level and want the value to be inherited by all PDBs. This change has been mentioned here.
It seems Oracle has not foreseen the unintended consequence of this change. Mainly, how this would affect the standby databases. The database used in this case is using local undo.

SQL> select property_value from database_properties where property_name='LOCAL_UNDO_ENABLED';

PROPERTY_VALUE
--------------------------------------------------------------------------------
TRUE

SQL> select ispdb_modifiable from v$parameter where name='undo_retention';

ISPDB
-----
TRUE

To demonstrate the issue at hand, assume the standby databases is in mount mode.

SQL> select open_mode from v$database;

OPEN_MODE
---------
MOUNTED

Current value for undo_retention is same for both cdb root and pdb and is the default 900.

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT
SQL> show parameter undo_retention

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_retention                       integer     900


-- value at PDB
SQL> alter session set container=dgpdb;

Session altered.

SQL> show con_name

CON_NAME
------------------------------
DGPDB
SQL> show parameter undo_retention

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_retention                       integer     900

Set the undo_retention parameter at CDB root level.

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

SQL>  alter system set undo_retention=901 scope=both;

System altered.

This change appears to be applied on both CDB root and PDB level.

SQL> show parameter undo_retention

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_retention                       integer     901


SQL> alter session set container=dgpdb;

Session altered.

SQL>  show con_name

CON_NAME
------------------------------
DGPDB
SQL> show parameter undo_retention

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_retention                       integer     901

Value appears to be set at PDB because at mounted state the PDB seem to inherit it from the CDB.

Now open the CDB in read only mode.

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY WITH APPLY

show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 DGPDB                          READ ONLY  NO

The change reamins on the CDB root but is lost on the PDB.

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

SQL> show  parameter undo_retention

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_retention                       integer     901


SQL>  alter session set container=dgpdb;

Session altered.

SQL> show con_name

CON_NAME
------------------------------
DGPDB

-- change not inherited by pdb

SQL>  show  parameter undo_retention

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_retention                       integer     900

Trying to set the value with container=all fails with the following.

SQL>  alter system set undo_retention=901 container=all scope=both;
 alter system set undo_retention=901 container=all scope=both
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-16000: database or pluggable database open for read-only access

Trying to set it by login into PDB also fails.

SQL> alter session set container=dgpdb;

Session altered.

SQL> show con_name

CON_NAME
------------------------------
DGPDB
SQL> alter system set undo_retention=901;
alter system set undo_retention=901
*
ERROR at line 1:
ORA-32000: write to SPFILE requested but SPFILE is not modifiable

Reason for this is as undo_retention value is not inherited, it is kept as a local parameter of the PDB. To change this the new value must be written to the data dictionary (PDB_SPFILE$ table) this means writing it to the system tablespace where the pdb_spfile$ table resides. But for standby databases this cannot be done as writes are not allowed on standby.
Only workaround found so far is to set the value on the PDB to memory.

SQL> alter system set undo_retention=901 scope=memory;

System altered.

However, this means everytime there's a restart of the standby the change is lost and must be set again.
SR has been raised.

↧

ORA-16713: The Oracle Data Guard broker command timed out When Changing LogXptMode

August 14, 2021, 6:00 pm

≫ Next: Updating the DCS Agent to the Expected Version

≪ Previous: Unable to Set UNDO_RETENTION in Standby PDB

After setting up a new dataguard configuration and broker the change of log transport mode to sync fail as below.

DGMGRL> edit database mystby set property LogXptMode='SYNC';
Error: ORA-16713: The Oracle Data Guard broker command timed out.

Failed.

Sync mode was needed to change the protection mode. There was no other DG issue and redo transport and log apply was working fine. Transport mode change was only failing on the standby at that time. However, the sync mode parameters gets setup on the primary's log archive destination. Log transport mode of the two instances looks like below due to this failure.

  Transport-Related Property Settings:
    Property                        myprod Value             mystby Value
    LogXptMode                      SYNC                     ASYNC

MOS doc 2007507.1 says this is due to rsm* process hanging and to kill it so it will respawn. This didn't help.

 ps ax | grep rsm
26686 pts/3    S+     0:00 grep --color=auto rsm
30386 ?        Ss     0:01 ora_rsm0_myprod
$ kill -9 30386

 ps ax | grep rsm
27157 ?        Ss     0:00 ora_rsm0_myprod  <--------- new process spawn
27161 pts/3    S+     0:00 grep --color=auto rsm

DGMGRL> edit database mystby set property LogXptMode='SYNC';
Error: ORA-16713: The Oracle Data Guard broker command timed out.

Failed.

MOS doc 1322877.1 suggested increasing OperationTimeout. That too didn't help.

DGMGRL>  EDIT CONFIGURATION SET PROPERTY OperationTimeout=600;
Property "operationtimeout" updated

DGMGRL> edit database mystby set property LogXptMode='SYNC';
Error: ORA-16713: The Oracle Data Guard broker command timed out.

Failed.

Few other MOS suggested clearing up ARDCI location. But those docs were related to DG broker hanging during validation. In this case validation ran fine without any issue.

Finally what resolved the issue was disabling the configuration and updating the log transport mode and enabling it again (maybe this respwaned the rsm cleanly than killing it).

DGMGRL> disable configuration;
Disabled.

DGMGRL> edit database mystby set property LogXptMode='SYNC';
Property "logxptmode" updated

DGMGRL> enable configuration;
Enabled.

This worked as expected. The transprot node changes are now visible on the log_archive_dest on the primary.

log_archive_dest_3   string service="mystbytns", SYNC AFFIRM delay=0 optional c

  Transport-Related Property Settings:
    Property                        myprod Value             mystby Value
    LogXptMode                      SYNC                     SYNC

After this it was possible to change the protection mode.

↧

Updating the DCS Agent to the Expected Version

September 1, 2021, 4:00 pm

≫ Next: Converting EM Repository DB from Non-CDB to PDB

≪ Previous: ORA-16713: The Oracle Data Guard broker command timed out When Changing LogXptMode

Following error was given while patching DBCS VM.

 dbcli update-server
{
"jobId" : "a81b17c5-3c22-44a7-ae9e-a6c4b62fe5d6",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "August 24, 2021 12:06:07 PM BST",
"resourceList" : [ ],
"description" : "Server Patching",
"updatedTime" : "August 24, 2021 12:06:07 PM BST",
"percentageProgress" : "0%",
"cause" : null,
"action" : null
}

dbcli describe-job -i a81b17c5-3c22-44a7-ae9e-a6c4b62fe5d6

Job details
----------------------------------------------------------------
                     ID:  a81b17c5-3c22-44a7-ae9e-a6c4b62fe5d6
            Description:  Server Patching
                 Status:  Failure
                Created:  August 24, 2021 12:06:07 PM BST
               Progress:  0%
                Message:  DCS-10205:Operation: Patch update failed. Found unsupported DCS Agent version: 21.2.2.2.0. Expected version: 21.2.3.0.0
                  Cause:  Attempted operation required expected version of DCS Agent.
                 Action:  Update the DCS Agent to the expected version, use the command 'odacli update-dcsagent' to update DCS Agent.

Running the odacli as suggested by the error message also failed.

odacli update-dcsagent
dbcli: 'update-dcsagent' is not an dbcli command.
usage: dbcli    [-h/--help]
<category> [-h/--help]
<operation> [-h/--help]
<command> [-h/--help]
<command> [<args>]

Installed DCS* verions were

rpm -qa|grep dcs
dcs-cli-21.2.2.2.0_210608.0448-1.x86_64
dcs-agent-21.3.1.0.0_210624.0711-16.x86_64
dcs-admin-20.4.2.1.0_210114.1437-1.x86_64

Running cliadm update-dbcli which updates the dbcli didn't resolve it.

Solution was to restart the dcs admin and agent (refer 2473667.1).

systemctl stop initdcsagent
systemctl stop initdcsadmin

systemctl start initdcsagent
systemctl start initdcsadmin

rpm -qa | grep dcs
dcs-agent-21.3.1.0.0_210624.0711-16.x86_64
dcs-cli-21.3.1.0.0_210624.0711-1.x86_64
dcs-admin-20.4.2.1.0_210114.1437-1.x86_64

Afterwards paatching continued without error.

↧

Converting EM Repository DB from Non-CDB to PDB

September 10, 2021, 6:00 pm

≫ Next: Using OCI Bastion for Accessing EM Database Express

≪ Previous: Updating the DCS Agent to the Expected Version

With EM 13.4 it is possible to use either non-CDB or a PDB as the repository database. This post shows the steps for converting a non-CDB EM repository DB to a PDB.
Main thing to look out for is that repository connection string is updated in various files the EM uses. Current connection string could be found out with the following command.

emctl config oms -list_repos_details
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Repository Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=console-srv)(PORT=2020)))(CONNECT_DATA=(SID=emandb1)))
Repository User : SYSMAN

Mos Docs 2431726.1, 2144665.1 and 2214218.1 all mention the files where the connection string is specified. These are available in the following directories.

cd /opt/app/software/em/gc_inst/user_projects/domains/GCDomain/config/fmwconfig

embi-policystoremerge-jpscfg.xml
<property name="jdbc.url" value="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=console-srv)(PORT=2020)))(CONNECT_DATA=(SID=emandb1)))"/>

jps-config-jse.xml
<property name="jdbc.url" value="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=console-srv)(PORT=2020)))(CONNECT_DATA=(SID=emandb1)))"/>

jps-config.xml
<property name="jdbc.url" value="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=console-srv)(PORT=2020)))(CONNECT_DATA=(SID=emandb1)))"/>


cd /opt/app/software/em/gc_inst/user_projects/domains/GCDomain/config/jdbc
all *.xml files in this directory as per 2214218.1

cd /opt/app/software/em/gc_inst/em/EMGC_OMS1
emgc.properties
EM_REPOS_CONNECTDESCRIPTOR=(DESCRIPTION\=(ADDRESS_LIST\=(ADDRESS\=(PROTOCOL\=TCP)(HOST\=console-srv)(PORT\=2020)))(CONNECT_DATA\=(SID\=emandb1)))

Main issue here is that connection string uses SID. Once the repos DB is plugged in as a PDB, the SID cannot be used to connect to it. Service name (SERVICE_NAME) must be used instead. Therefore as the first step the connection string would be updated to use service name instead of SID.
The current repos DB is called emandb1.

It will have a default service called emandb1. The listener status command would give these details.

Service "emandb1" has 1 instance(s).
  Instance "emandb1", status READY, has 1 handler(s) for this service...

If the non-CDB is plugged as a PDB using the same name emandb1 then this will result in a default service for PDB called emandb1. This would allow OMS to connect to the PDB without requring any connection string change. Also changing the connecting string before non-CDB conversion would allow any connectivity related issues to be ironed out beforehand.
With this in mind change the connection string to use service_name instead of SID. First check the connection string with service_name works by using it on a sqlplus connection

sqlplus sysman@"(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=console-srv)(PORT=2020)))(CONNECT_DATA=(SERVICE_NAME=emandb1)))"

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 7 09:10:44 2021
Version 19.12.0.0.0

SQL>

Then use the following command (mentioned in 2214218.1, 1395107.1 and CC Advanced Installation and Configuration Guide) to change the connecting string. The documentation states to shutdown OMS before running this command. But it seems the admin server neeeds to be up for it to work. Good thing is comamnd will bring up the admin server.

emctl stop oms

emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=console-srv)(PORT=2020)))(CONNECT_DATA=(SERVICE_NAME=emandb1)))" -repos_user SYSMAN
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Enter Repository User's Password :
Admin server is down. It is required to update repository details. This command will try to bring it up.
Starting Admin Server only...
Admin Server Successfully Started
Successfully updated datasources and stored repository details in Credential Store.
If there are multiple OMSs in this environment, run this store_repos_details command on all of them.
And finally, restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
It is also necessary to restart the BI Publisher Managed Server.

emctl stop oms -all

emctl start oms

emctl config oms -list_repos_details
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Repository Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=console-srv)(PORT=2020)))(CONNECT_DATA=(SERVICE_NAME=emandb1)))
Repository User : SYSMAN

Above shows the new connetion string uses service_name. Checking in individual files mentioned earlier, all but embi-policystoremerge-jpscfg.xml had the connection string changed correctly. The connectring string in file embi-policystoremerge-jpscfg.xml still refered to SID. Manually edit the connection string to use service_name. Once done login to cloud control and check all is working as expected. If any connectivity errors are there correct them before proceeding to next steps. The EM Repos DB target shown on the console may require re-discovering and promoting

Next step is to create a CDB to plug the repos non-CDB as a PDB. The repos DB has very specific requirements when it comes to their memory parametrs, redo log file sizes, hidden parameters and etc (refer earlier post). The simplest way to accomplish this is to create a template from current non-CDB and use the template to create a the CDB. Below two commands does exactly this.

dbca -createTemplateFromDB -templateName /home/oracle/emrepos13.4.dbt -sourceDB emandb1 -sysDBAUserName sys -maintainFileLocations false  -silent

dbca -silent -createDatabase -templateName /home/oracle/emrepos13.4.dbt -gdbName emandb -sysPassword xxxx -systemPassword xxx  -emConfiguration NONE -storageType ASM -asmsnmpPassword xxxx -diskGroupName DATA -recoveryGroupName FRA -createAsContainerDatabase true

The CDB is called emandb(different to current non-CDB repos DB name). The database created from template will have tablespaces that were created for EM repos use such as MGMT_AD4J_TS,MGMT_ECM_DEPOT_TS,MGMT_TABLESPACE. These could be dropped from the CDB to save space (alternatively edit the template so these are not created in the first place).
Next step is to plug the non-CDB as a PDB. For this test case used the manual method. Last set of steps are shown below. The non-CDB is plugged in as a PDB has the same name, emandb1.

SQL> CREATE PLUGGABLE DATABASE emandb1 USING '/home/oracle/emandb1_non_cdb.xml' copy;

Pluggable database created.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 EMANDB1                        MOUNTED
SQL> alter session set container=emandb1;

Session altered.

SQL> select name from v$datafile;
NAME
----------------------------------------------------------------------------------------
+DATA/EMANDB/A148AAC6957E5A34E0530500160A7405/DATAFILE/system.303.1082632693
+DATA/EMANDB/A148AAC6957E5A34E0530500160A7405/DATAFILE/sysaux.306.1082632693
+DATA/EMANDB/A148AAC6957E5A34E0530500160A7405/DATAFILE/undotbs1.304.1082632693
+DATA/EMANDB/A148AAC6957E5A34E0530500160A7405/DATAFILE/users.299.1082632693
+DATA/EMANDB/A148AAC6957E5A34E0530500160A7405/DATAFILE/oradbaudit.298.1082632693
+DATA/EMANDB/A148AAC6957E5A34E0530500160A7405/DATAFILE/mgmt_ad4j_ts.305.1082632693
+DATA/EMANDB/A148AAC6957E5A34E0530500160A7405/DATAFILE/mgmt_ecm_depot_ts.297.1082632693
+DATA/EMANDB/A148AAC6957E5A34E0530500160A7405/DATAFILE/mgmt_tablespace.307.1082632693

8 rows selected.

The new PDB now containes the datafiles related to EM related tablespaces. Complete the conversion with the following.

@?/rdbms/admin/noncdb_to_pdb.sql

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 EMANDB1                        MOUNTED
SQL> alter pluggable database open;

Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 EMANDB1                        READ WRITE NO

Once the PDB is open the listener will have a service emandb1 (PDB default service) for emandb CDB instance.

Service "emandb" has 1 instance(s).
  Instance "emandb", status READY, has 1 handler(s) for this service...
Service "emandb1" has 1 instance(s).
  Instance "emandb", status READY, has 1 handler(s) for this service...

Use sqlplus with same connection string as before to check connectivity to the PDB.

sqlplus sysman@"(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=console-srv)(PORT=2020)))(CONNECT_DATA=(SERVICE_NAME=emandb1)))"

If connectivity is fine start the OMS.

emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up

Discover the new EM repos db and related PDB targets and promote them. The dbsnmp common user in CDB will be in locked status. Unlock the dbsnmp account and give a new password.

The old EM repos db will be in down state while newly promoted CDB and PDB will be shown up state.

Once confident all is working as expected the old EM repos DB could be dropped.

↧

Using OCI Bastion for Accessing EM Database Express

September 17, 2021, 7:00 pm

≫ Next: Gradual Database Password Rollove and UCP

≪ Previous: Converting EM Repository DB from Non-CDB to PDB

OCI bastion allows a restricted and time-limited way to access private endpoints. This could be used for accessing EM database express of DBCS VM DBs. Since most of the cases database would reside in a private subnet, the bastion service provides a convenient way to accessing these services with private endpoints. Only downside is that maximum time to live (TTL) for a bastion session is 3 hours. So this wouldn't be good for 24/7 monitoring (unless creating a new session every 3 hours is not a hazzel). This post shows the steps for creating a bastion session and accessing the EM database express using it.
First up some details about the DBCS VM DB setup. The database system resides in a private subnet.

If not already done, enable EM Express for the database. For more info on this refer 2453454.1. In this setup the EM express runs on port 5500.
Access bastion service from the OCI console (found under Identity and security). Give a bastion a name and select the same VCN and private subnet where the databsae resides. These sections shown in red. In addition to above provide a list of IPs or hostname that will access the bastion session. In this case a single IP is given (public IP for the test windows pc shown by whatsmyip). This section is shown in blue.

Once the bastion is created make a note of its private endpoint.

Add an ingress rule to the security list associated with the private subnet (where the database resides) using the bastion endpoint as source. This step is not needed if an exiting rule already allows this traffic.

Make a note of the private IP of the database node.

Next create a bastion session by selecting port forwarding as the session type. As the IP address specify the private IP of the database node. Specify the port on which EM express runs, in this case 5500. Finally provide a public key of a ssh key pair. This doesn't have to be the same key used for the database. This key is used only for the bastion session and has no relevence for the database.

Click the menau at the end of the bastion session (three dots at the end) and select copy ssh command. This will copy the ssh command to needed to create the tunnel to the clipboard.

Replace the placeholder values with actual values. Placeholder values include the private key file and the local port. In this case the local port is also set to 5500. If running on windows the power shell could be used to execute the ssh to create the port forwarding ssh tunnel.

Once the ssh tunnel is created access the EM express from the PC browser using localhost as the server.

↧

Gradual Database Password Rollove and UCP

September 25, 2021, 5:49 am

≫ Next: Failover and Reinstate With Multiple Physical Standbys

≪ Previous: Using OCI Bastion for Accessing EM Database Express

Oracle first introduced gradual password rollover as a new feature in 21c. But with RU 19.12 this feature is also available on 19c.
The gradual password rollover feature introduced a new parameter for user profiles called "PASSWORD_ROLLOVER_TIME". When this is set, a user could have two passwords for authentication for the duration specified by the value set for PASSWORD_ROLLOVER_TIME. Once the rollover period ends only the new password is valid.
This could be demo as below using 19.12. A new profile is craeted with password rollver time set to 1.

SQL> create profile test limit PASSWORD_ROLLOVER_TIME 1;

Profile created.

A user is assigned the new profile

SQL> alter user asangaro profile test;

User altered.

The current status of the account is open.

SQL> select username,account_status from dba_users where username='ASANGARO';

USERNAME   ACCOUNT_STATUS
---------- ---------------
ASANGARO   OPEN

Change the password for the user and check the account status

SQL> alter user asangaro identified by hello123##;

User altered.

SQL>  select username,account_status from dba_users where username='ASANGARO';

USERNAME   ACCOUNT_STATUS
---------- -------------------
ASANGARO   OPEN & IN ROLLOVER

Account status is now open but in the rollover period. During this period user can use both the old password and new password set above.
The rollover period could be manually ended by using the following command.

SQL> alter user asangaro expire password rollover period;

User altered.

SQL> select username,account_status from dba_users where username='ASANGARO';

USERNAME   ACCOUNT_STATUS
---------- -----------------
ASANGARO   OPEN

Main thign to remember is the below

Oracle Database does not send any special messages to the database clients that indicate that the user account is in the password rollover period. This design avoids any errors from applications that may not be equipped to handle error and warning messages when a user logs in.

UCP will not automatically update itself with the new password. However, its behaviour may seems it has updated the password and working fine.

Universal Connection Pool (UCP) does the authentication only during the intial creation of the connection. For example if the inital size of the connection pool is set to 10, then those 10 connections will be authenticated using the password provided. There's no authentication happening when these connections are checked out the pool later on. This could be easily tested and verified (no need for PASSWORD_ROLLOVER_TIME configuration) by creating a UCP with initial set of connections and then changing the password. The connection created would still work and would be able to run DB queries. Because of this fact, even after rollover period has ended UCP may continue to function normal unless additoinal connections are created.

If more connections are needed than the intial amount, then authentication take place when those are created in the connection pool. Going by the above example, if password was changed after initializng the pool with 10 connections then during the creation of the 11th connection an ORA-1017 will be thrown. So after the rollover period has ended, if the UCP needed more connection than it had during the rollover period this would result in ORA-1017.

There's no way to update the UCP with the new password without recreating it. The property check interval only concern itself with resizing the pool. Not with password changes. Even if there is a UCP manger, methods such as purge, recycle, refresh would not update the password in the pool.

Only possible solution it seems is to destroy and recreate the pool. During this period the source where password is read must have been updated with the new password.

Alternatively, application could have a rolling restart with the new password.

↧

Failover and Reinstate With Multiple Physical Standbys

September 28, 2021, 6:00 pm

≫ Next: Active Data Guard and PDB Parameters

≪ Previous: Gradual Database Password Rollove and UCP

The data guard configuration consists of three databases.

 DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest3 - Primary database
    dgtest  - Physical standby database
    dgtest2 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 67 seconds ago)

At the moment dgtest3 is the primary. One of the standby (dgtest2) databases is open in mount mode.

SQL> select db_unique_name,open_mode from v$database;

DB_UNIQUE_NAME                 OPEN_MODE
------------------------------ --------------------
dgtest2                        MOUNTED

The other standby (dgtest) is an active data guard and is open in read only mode with log apply.

SQL> select db_unique_name,open_mode from v$database;

DB_UNIQUE_NAME                 OPEN_MODE
------------------------------ --------------------
dgtest                         READ ONLY WITH APPLY

To simulate primary DB failure it is shutdown with abort.

SQL> shutdown abort;
ORACLE instance shut down.

Checking the data guard configuration from dgtest2 shows an error state.

DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest3 - Primary database
    Error: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

    dgtest  - Physical standby database
    dgtest2 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
ERROR   (status updated 0 seconds ago)

One of the existing standbys could be choosen for failover. In this case dgtest2( instance in mount mode) was chosen. Validating the dgtest2 show that it is ready for failover.

DGMGRL> validate database dgtest2

  Database Role:     Physical standby database
  Primary Database:  dgtest3
    Warning: primary database was not reachable

  Ready for Switchover:  No
Ready for Failover:    Yes (Primary Not Running)

  Flashback Database Status:
    dgtest3:  Unknown
    dgtest2:  On

  Managed by Clusterware:
    dgtest3:  Unknown
    dgtest2:  YES
    Validating static connect identifier for the primary database dgtest3...
Unable to connect to database using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ip-132)(PORT=1521))(CONNECT                                                                                                                          _DATA=(SERVICE_NAME=dgtest3_DGMGRL)(INSTANCE_NAME=dgtest3)(SERVER=DEDICATED)(STATIC_SERVICE=TRUE)))
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.
    Warning: Ensure primary database's StaticConnectIdentifier property
    is configured properly so that the primary database can be restarted
    by DGMGRL after switchover

  Temporary Tablespace File Information:
    dgtest3 TEMP Files:  Unknown
    dgtest2 TEMP Files:  3

  Data file Online Move in Progress:
    dgtest3:  Unknown
    dgtest2:  No

  Transport-Related Information:
    Transport On:  No
    Gap Status:    Unknown
    Transport Lag:  0 seconds (computed 24 seconds ago)
    Transport Status:  Success

  Log Files Cleared:
    dgtest3 Standby Redo Log Files:  Unknown
    dgtest2 Online Redo Log Files:   Unknown
    dgtest2 Standby Redo Log Files:  Unknown

Initiate failover to dgtest2

DGMGRL> failover to dgtest2;
Performing failover NOW, please wait...
Failover succeeded, new primary is "dgtest2"

During the failover process the active data guard instance continue to function. It will cancel the current redo apply service and start a new one with the incarnation that results from the failover. At the same time any references to the old primary will be removed (such as fal_server).

2021-09-20T09:53:29.718199+00:00
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
2021-09-20T09:53:29.720039+00:00
PR00 (PID:5312): MRP0: Background Media Recovery cancelled with status 16037
2021-09-20T09:53:29.720747+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5312.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:5312): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 4236455
Stopping change tracking
2021-09-20T09:53:29.943216+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5312.trc:
ORA-16037: user requested cancel of managed recovery operation
2021-09-20T09:53:30.067266+00:00
Background Media Recovery process shutdown (dgtest)
2021-09-20T09:53:30.720982+00:00
Managed Standby Recovery Canceled (dgtest)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
2021-09-20T09:53:32.641452+00:00
ALTER SYSTEM SET fal_server='dgtest2tns' SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT  NODELAY
2021-09-20T09:53:32.658894+00:00
Attempt to start background Managed Standby Recovery process (dgtest)
Starting background process MRP0
2021-09-20T09:53:32.673122+00:00
MRP0 started with pid=63, OS id=5816
2021-09-20T09:53:32.674424+00:00
Background Managed Standby Recovery process started (dgtest)
2021-09-20T09:53:36.270863+00:00
 rfs (PID:5820): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is Foreground (PID:11152)
Deleted Oracle managed file +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_0_seq_0.371.1083750817
2021-09-20T09:53:37.697852+00:00
 Started logmerger process
2021-09-20T09:53:37.712394+00:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:5826): Managed Standby Recovery starting Real Time Apply
max_pdb is 3
2021-09-20T09:53:37.888440+00:00
 rfs (PID:5824): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is SYNC (PID:10754)
 rfs (PID:5824): New archival redo branch: 1083750811 current: 1067878072
 rfs (PID:5824): Primary database is in MAXIMUM AVAILABILITY mode
 rfs (PID:5824): Changing standby controlfile to RESYNCHRONIZATION level
 rfs (PID:5824): Standby controlfile consistent with primary
2021-09-20T09:53:37.951928+00:00
 rfs (PID:5824): Selected LNO:6 for T-1.S-3 dbid 4024401720 branch 1083750811
2021-09-20T09:53:37.974747+00:00
Parallel Media Recovery started with 8 slaves
2021-09-20T09:53:38.018389+00:00
Stopping change tracking
PR00 (PID:5826): Media Recovery Waiting for T-1.S-382
2021-09-20T09:53:38.440514+00:00
 rfs (PID:5855): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is FAL (PID:11166)
2021-09-20T09:53:38.505442+00:00
 rfs (PID:5855): Selected LNO:7 for T-1.S-2 dbid 4024401720 branch 1083750811
 rfs (PID:5855): A new recovery destination branch has been registered
 rfs (PID:5855): New Archival REDO Branch(resetlogs_id): 1083750811  Prior: 1067878072
 rfs (PID:5855): Archival Activation ID: 0xf0d1268f Current: 0xf0d199cf
 rfs (PID:5855): Effect of primary database OPEN RESETLOGS
 rfs (PID:5855): Managed Standby Recovery process is active
2021-09-20T09:53:38.536066+00:00
Incarnation entry added for Branch(resetlogs_id): 1083750811 (dgtest)
2021-09-20T09:53:38.548638+00:00
Setting recovery target incarnation to 2
2021-09-20T09:53:38.678029+00:00
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT  NODELAY
2021-09-20T09:53:39.126620+00:00
PR00 (PID:5826): MRP0: Incarnation has changed! Retry recovery...
2021-09-20T09:53:39.127329+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5826.trc:
ORA-19906: recovery target incarnation changed during recovery
PR00 (PID:5826): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Stopping change tracking
2021-09-20T09:53:39.233028+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5826.trc:
ORA-19906: recovery target incarnation changed during recovery
2021-09-20T09:53:39.432821+00:00
 rfs (PID:5858): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is FAL (PID:11172)
2021-09-20T09:53:39.480429+00:00
 rfs (PID:5858): Opened log for T-1.S-382 dbid 4024401720 branch 1067878072
2021-09-20T09:53:39.509873+00:00
 rfs (PID:5858): Archived Log entry 746 added for B-1067878072.T-1.S-382 ID 0xf0d199cf LAD:2
2021-09-20T09:53:39.573601+00:00
 Started logmerger process
2021-09-20T09:53:39.614023+00:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:5861): Managed Standby Recovery starting Real Time Apply
2021-09-20T09:53:39.622286+00:00
ARC8 (PID:5169): Archived Log entry 747 added for T-1.S-2 ID 0xf0d1268f LAD:1
2021-09-20T09:53:39.678880+00:00
max_pdb is 3
2021-09-20T09:53:39.878314+00:00
Parallel Media Recovery started with 8 slaves
2021-09-20T09:53:39.894206+00:00
Media Recovery start incarnation depth : 1, target inc# : 2, irscn : 4236456
Stopping change tracking
2021-09-20T09:53:39.917344+00:00
 rfs (PID:5863): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is FAL (PID:11158)
2021-09-20T09:53:39.974576+00:00
 rfs (PID:5863): Opened log for T-1.S-1 dbid 4024401720 branch 1083750811
2021-09-20T09:53:40.001952+00:00
 rfs (PID:5863): Archived Log entry 748 added for B-1083750811.T-1.S-1 ID 0xf0d1268f LAD:2
2021-09-20T09:53:40.011908+00:00
PR00 (PID:5861): Media Recovery Log +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_1_seq_382.371.1083750819
2021-09-20T09:53:40.134398+00:00
PR00 (PID:5861): Resetting standby activation ID 4040268239 (0xf0d199cf)
2021-09-20T09:53:40.138397+00:00
Media Recovery End-Of-Redo indicator encountered
2021-09-20T09:53:40.138471+00:00
Media Recovery Continuing
2021-09-20T09:53:40.401844+00:00
PR00 (PID:5861): Media Recovery Log +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_1_seq_1.454.1083750819
2021-09-20T09:53:40.499037+00:00
PR00 (PID:5861): Media Recovery Log +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_1_seq_2.372.1083750819
2021-09-20T09:53:40.692602+00:00
PR00 (PID:5861): Media Recovery Waiting for T-1.S-3 (in transit)
2021-09-20T09:53:40.698462+00:00
Recovery of Online Redo Log: Thread 1 Group 6 Seq 3 Reading mem 0
  Mem# 0: +FRA/DGTEST/ONLINELOG/group_6.423.1067949163
2021-09-20T09:53:41.024853+00:00
 rfs (PID:5824): Changing standby controlfile to MAXIMUM AVAILABILITY level
2021-09-20T09:53:41.041788+00:00
 rfs (PID:5824): Selected LNO:7 for T-1.S-4 dbid 4024401720 branch 1083750811
2021-09-20T09:53:41.045351+00:00
ARC9 (PID:5171): Archived Log entry 749 added for T-1.S-3 ID 0xf0d1268f LAD:1
2021-09-20T09:53:41.118868+00:00
PR00 (PID:5861): Media Recovery Waiting for T-1.S-4 (in transit)
2021-09-20T09:53:41.124519+00:00
Recovery of Online Redo Log: Thread 1 Group 7 Seq 4 Reading mem 0
  Mem# 0: +FRA/DGTEST/ONLINELOG/group_7.424.1067949165

The instance failed over to will stop it's current redo apply service and convert to a read/write primary. It is intresting to see some of the lines in the log output "switchover" instead of failover.

2021-09-20T09:53:24.419649+00:00
Beginning failover to database dgtest2.
Starting background process NSV1
2021-09-20T09:53:24.492112+00:00
NSV1 started with pid=16, OS id=12324
2021-09-20T09:53:29.976461+00:00
ALTER DATABASE FAILOVER TO dgtest2
2021-09-20T09:53:29.976612+00:00
RSM0 (PID:11547): The Time Management Interface (TMI) is being enabled for role transition
RSM0 (PID:11547): information.  This will result in messages beingoutput to the alert log
RSM0 (PID:11547): file with the prefix 'TMI: '.  This is being enabled to make the timing of
RSM0 (PID:11547): the various stages of the role transition available for diagnostic purposes.
RSM0 (PID:11547): This output will end when the role transition is complete.
TMI: dbsdrv failover to target BEGIN 2021-09-20 09:53:29.977021
Terminal Recovery requested in process 11547
TMI: adbdrv termRecovery BEGIN 2021-09-20 09:53:29.978960
RSM0 (PID:11547): Terminal Recovery: Stopping real time apply
2021-09-20T09:53:29.980343+00:00
PR00 (PID:11627): MRP0: Background Media Recovery cancelled with status 16037
2021-09-20T09:53:29.980556+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_pr00_11627.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:11627): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 4236455
Stopping change tracking
2021-09-20T09:53:30.221440+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_pr00_11627.trc:
ORA-16037: user requested cancel of managed recovery operation
2021-09-20T09:53:30.319390+00:00
Background Media Recovery process shutdown (dgtest2)
2021-09-20T09:53:30.981370+00:00
RSM0 (PID:11547): Terminal Recovery: Stopped real time apply
2021-09-20T09:53:30.986937+00:00
Attempt to do a Terminal Recovery (dgtest2)
TMI: adbdrv termRecovery END 2021-09-20 09:53:30.986957
2021-09-20T09:53:30.987291+00:00
Media Recovery Start: Managed Standby Recovery (dgtest2)
2021-09-20T09:53:30.992470+00:00
Serial Media Recovery started
RSM0 (PID:11547): Managed Standby Recovery not using Real Time Apply
max_pdb is 3
Stopping change tracking
RSM0 (PID:11547): Begin: SRL archival
RSM0 (PID:11547): End: SRL archival
RSM0 (PID:11547): Terminal Recovery timestamp is '09/20/2021 09:53:31'
RSM0 (PID:11547): Terminal Recovery: applying standby redo logs.
RSM0 (PID:11547): Terminal Recovery: thread 1 seq# 382 redo required
2021-09-20T09:53:31.259258+00:00
RSM0 (PID:11547): Terminal Recovery:
2021-09-20T09:53:31.264259+00:00
Recovery of Online Redo Log: Thread 1 Group 5 Seq 382 Reading mem 0
  Mem# 0: +FRA/DGTEST2/ONLINELOG/group_5.423.1067949477
Terminal Recovery finished with No-Data-Loss
2021-09-20T09:53:31.297911+00:00
Incomplete Recovery applied until change 4236456 time 09/20/2021 09:52:41
2021-09-20T09:53:31.312164+00:00
Media Recovery Complete (dgtest2)
Terminal Recovery: successful completion
RSM0 (PID:11547): Forcing ARSCN to IRSCN for TR SCN:0x000000000040a4a8
RSM0 (PID:11547): Attempt to set limbo arscn SCN:0x000000000040a4a8 irscn SCN:0x000000000040a4a8
RSM0 (PID:11547): Resetting standby activation ID 4040268239 (0xf0d199cf)
Stopping change tracking
2021-09-20T09:53:31.464571+00:00
ALTER DATABASE SWITCHOVER TO PRIMARY (dgtest2)
Maximum wait for role transition is 15 minutes.
TMI: kcv_commit_to_so_to_primary wait for MRP to finish BEGIN 2021-09-20 09:53:31.466893
TMI: kcv_commit_to_so_to_primary wait for MRP to finish END 2021-09-20 09:53:31.467051
TMI: kcv_commit_to_so_to_primary Switchover from physical BEGIN 2021-09-20 09:53:31.468362
Backup controlfile written to trace file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_rsm0_11547.trc
Standby terminal recovery start SCN: 4236455
RESETLOGS after incomplete recovery UNTIL CHANGE 4236456 time 09/20/2021 09:52:41
RSM0 (PID:11547): ORL pre-clearing operation disabled by switchover
Online log +DATA/DGTEST2/ONLINELOG/group_1.289.1067949469: Thread 1 Group 1 was previously cleared
Online log +FRA/DGTEST2/ONLINELOG/group_1.419.1067949471: Thread 1 Group 1 was previously cleared
Online log +DATA/DGTEST2/ONLINELOG/group_2.290.1067949471: Thread 1 Group 2 was previously cleared
Online log +FRA/DGTEST2/ONLINELOG/group_2.420.1067949473: Thread 1 Group 2 was previously cleared
Online log +DATA/DGTEST2/ONLINELOG/group_3.291.1067949473: Thread 1 Group 3 was previously cleared
Online log +FRA/DGTEST2/ONLINELOG/group_3.421.1067949473: Thread 1 Group 3 was previously cleared
Standby became primary SCN: 4236454
2021-09-20T09:53:31.635945+00:00
Setting recovery target incarnation to 2
2021-09-20T09:53:31.673002+00:00
RSM0 (PID:11547): RT: Role transition work is not done
RSM0 (PID:11547): The Time Management Interface (TMI) is being enabled for role transition
RSM0 (PID:11547): information.  This will result in messages beingoutput to the alert log
RSM0 (PID:11547): file with the prefix 'TMI: '.  This is being enabled to make the timing of
RSM0 (PID:11547): the various stages of the role transition available for diagnostic purposes.
RSM0 (PID:11547): This output will end when the role transition is complete.
RSM0 (PID:11547): Redo network throttle feature is disabled at mount time
2021-09-20T09:53:31.710578+00:00
RSM0 (PID:11547): Database role cleared from PHYSICAL STANDBY [kcvs.c:1099]
Switchover: Complete - Database mounted as primary
TMI: kcv_commit_to_so_to_primary Switchover from physical END 2021-09-20 09:53:31.712243
TMI: dbsdrv failover to target END 2021-09-20 09:53:31.712367
Failover completed with No-Data-Loss.
Completed: ALTER DATABASE FAILOVER TO dgtest2
2021-09-20T09:53:31.975876+00:00
RSM0 (PID:11547): Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST [krsd.c:18222]
2021-09-20T09:53:31.976280+00:00
ARC9 (PID:11172): Becoming the 'no SRL' ARCH
2021-09-20T09:53:31.983848+00:00
ALTER SYSTEM SET log_archive_dest_2='service="dgtesttns"','SYNC AFFIRM delay=0 optional compression=disable max_fai                                                                                                                          lure=0 reopen=300 db_unique_name="dgtest" net_timeout=30','valid_for=(online_logfile,all_roles)' SCOPE=BOTH;
2021-09-20T09:53:32.015296+00:00
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
Completed: ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
ALTER DATABASE OPEN
2021-09-20T09:53:32.030768+00:00
TMI: adbdrv open database BEGIN 2021-09-20 09:53:32.030695
Data Guard Broker initializing...
Ping without log force is disabled:
  instance mounted in exclusive mode.
Buffer Cache Full DB Caching mode changing from FULL CACHING DISABLED to FULL CACHING ENABLED
2021-09-20T09:53:32.116297+00:00
Crash Recovery excluding pdb 2 which was cleanly closed.
2021-09-20T09:53:32.116442+00:00
Crash Recovery excluding pdb 3 which was cleanly closed.
Endian type of dictionary set to little
2021-09-20T09:53:32.136779+00:00
Assigning activation ID 4040238735 (0xf0d1268f)
LGWR (PID:10754): Primary database is in MAXIMUM AVAILABILITY mode
2021-09-20T09:53:32.140956+00:00
LGWR (PID:10754): LAD:2 is UNSYNCHRONIZED
LGWR (PID:10754): LAD:1 is not serviced by LGWR
2021-09-20T09:53:33.177813+00:00
Thread 1 advanced to log sequence 2 (thread open)
Redo log for group 2, sequence 2 is not located on DAX storage
Thread 1 opened at log sequence 2
  Current log# 2 seq# 2 mem# 0: +DATA/DGTEST2/ONLINELOG/group_2.290.1067949471
  Current log# 2 seq# 2 mem# 1: +FRA/DGTEST2/ONLINELOG/group_2.420.1067949473
Successful open of redo thread 1
2021-09-20T09:53:33.246706+00:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Stopping change tracking
2021-09-20T09:53:33.287479+00:00
ARC1 (PID:11156): Archived Log entry 520 added for T-1.S-1 ID 0xf0d1268f LAD:1
2021-09-20T09:53:33.290677+00:00
TT03 (PID:12462): Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2021-09-20T09:53:33.549541+00:00
Undo initialization recovery: Parallel FPTR failed: start:740398 end:740405 diff:7 ms (0.0 seconds)
Undo initialization recovery: err:0 start: 740397 end: 740435 diff: 38 ms (0.0 seconds)
[11547] Successfully onlined Undo Tablespace 2.
Undo initialization online undo segments: err:0 start: 740435 end: 740959 diff: 524 ms (0.5 seconds)
Undo initialization finished serial:0 start:740397 end:740967 diff:570 ms (0.6 seconds)
Dictionary check beginning
Dictionary check complete
2021-09-20T09:53:34.245425+00:00
Database Characterset is AL32UTF8
No Resource Manager plan active
2021-09-20T09:53:35.468394+00:00
joxcsys_required_dirobj_exists: directory object exists with required path /opt/app/oracle/product/19.x.0/dbhome_2/                                                                                                                          javavm/admin/, pid 11547 cid 1
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Starting background process AQPC
2021-09-20T09:53:36.133706+00:00
AQPC started with pid=70, OS id=12472
PDB$SEED(2):Pluggable database PDB$SEED opening in read only
2021-09-20T09:53:36.942191+00:00
PDB$SEED(2):Autotune of undo retention is turned on.
2021-09-20T09:53:37.055480+00:00
TT03 (PID:12462): Enabling archival of deferred physical standby SRLs
2021-09-20T09:53:37.152868+00:00
TT03 (PID:12462): Archived Log entry 521 added for T-1.S-382 ID 0xf0d199cf LAD:1
2021-09-20T09:53:37.246644+00:00
PDB$SEED(2):Endian type of dictionary set to little
PDB$SEED(2):Undo initialization finished serial:0 start:744276 end:744276 diff:0 ms (0.0 seconds)
PDB$SEED(2):Pluggable database PDB$SEED dictionary check beginning
2021-09-20T09:53:38.232036+00:00
Thread 1 advanced to log sequence 3 (LGWR switch),  current SCN: 4236634
  Current log# 3 seq# 3 mem# 0: +DATA/DGTEST2/ONLINELOG/group_3.291.1067949473
  Current log# 3 seq# 3 mem# 1: +FRA/DGTEST2/ONLINELOG/group_3.421.1067949473
2021-09-20T09:53:38.316860+00:00
PDB$SEED(2):Pluggable Database PDB$SEED Dictionary check complete
2021-09-20T09:53:38.327245+00:00
ARC5 (PID:11164): Archived Log entry 522 added for T-1.S-2 ID 0xf0d1268f LAD:1
2021-09-20T09:53:38.328930+00:00
PDB$SEED(2):Database Characterset for PDB$SEED is AL32UTF8
2021-09-20T09:53:38.503847+00:00
QPI: opatch file present, opatch
QPI: qopiprep.bat file present
2021-09-20T09:53:38.642191+00:00
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
2021-09-20T09:53:38.835626+00:00
PDB$SEED(2):SUPLOG: Set PDB SUPLOG SGA at PDB OPEN, old 0x0, new 0x0 (no suplog)
PDB$SEED(2):Opening pdb with no Resource Manager plan active
2021-09-20T09:53:39.622239+00:00
Starting background process CJQ0
Completed: ALTER DATABASE OPEN

After the failover completes the data guard broker status shows old primary is disabled and need reinstating.

DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest2 - Primary database
    dgtest  - Physical standby database
    dgtest3 - Physical standby database (disabled)
      ORA-16661: the standby database needs to be reinstated

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 40 seconds ago)

If the old primary issue is rectified it would be reinstated as a standby (instead of adding as a new standby). When the old primary is started the DG broker will detect failover has happened and will prevent the opening the opening of the DB in read/write mode.

Completed: ALTER DATABASE MOUNT /* db agent *//* {0:0:312} */
ALTER DATABASE OPEN /* db agent *//* {0:0:312} */
Data Guard Broker initializing...
2021-09-20T09:58:19.408297+00:00
Starting Data Guard Broker (DMON)
Starting background process INSV
2021-09-20T09:58:19.425650+00:00
INSV started with pid=49, OS id=5982
2021-09-20T09:58:23.448529+00:00
Data Guard Broker initialization complete
Data Guard: verifying database primary role...
2021-09-20T09:58:23.467893+00:00
Starting background process NSV1
2021-09-20T09:58:23.484569+00:00
NSV1 started with pid=50, OS id=5987
2021-09-20T09:58:27.484306+00:00
Starting background process NSV2
2021-09-20T09:58:27.501947+00:00
NSV2 started with pid=51, OS id=5992
2021-09-20T09:58:30.605237+00:00
Starting background process RSM0
2021-09-20T09:58:30.620725+00:00
RSM0 started with pid=52, OS id=5997
2021-09-20T09:58:31.448386+00:00
Data Guard: broker startup completed
Data Guard determines a failover has occurred - instance will not be opened
ORA-16649 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:0:312} */...

The database will be left at mount mode.

SQL> select db_unique_name,open_mode from v$database;

DB_UNIQUE_NAME                 OPEN_MODE
------------------------------ --------------------
dgtest3                        MOUNTED

From the current primary DG broker run the reinstate command.

DGMGRL> reinstate database dgtest3
Reinstating database "dgtest3", please wait...
Reinstatement of database "dgtest3" succeeded

During the reinstate process the old primary will be flashback to the time where failover occured and converted to a physical standby.

2021-09-20T09:59:51.612704+00:00
FLASHBACK DATABASE TO SCN 4236454
2021-09-20T09:59:51.774069+00:00
Flashback Restore Start
Flashback Restore Complete
Flashback Media Recovery Start
 Started logmerger process
2021-09-20T09:59:52.058152+00:00
max_pdb is 3
2021-09-20T09:59:52.297907+00:00
Parallel Media Recovery started with 8 slaves
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_373.332.1083750351
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_374.264.1083750365
2021-09-20T09:59:52.812785+00:00
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_375.265.1083750375
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_376.267.1083750649
2021-09-20T09:59:53.157016+00:00
Media Recovery End-Of-Redo indicator encountered
2021-09-20T09:59:53.157085+00:00
Media Recovery Continuing
2021-09-20T09:59:53.202162+00:00
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_377.271.1083750659
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_378.269.1083750671
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_379.268.1083750683
2021-09-20T09:59:53.834774+00:00
Recovery of Online Redo Log: Thread 1 Group 1 Seq 380 Reading mem 0
  Mem# 0: +DATA/DGTEST3/ONLINELOG/group_1.299.1069533175
  Mem# 1: +FRA/DGTEST3/ONLINELOG/group_1.460.1069533177
2021-09-20T09:59:53.955443+00:00
Recovery of Online Redo Log: Thread 1 Group 2 Seq 381 Reading mem 0
  Mem# 0: +DATA/DGTEST3/ONLINELOG/group_2.300.1069533177
  Mem# 1: +FRA/DGTEST3/ONLINELOG/group_2.459.1069533177
2021-09-20T09:59:54.017466+00:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 382 Reading mem 0
  Mem# 0: +DATA/DGTEST3/ONLINELOG/group_3.301.1069533179
  Mem# 1: +FRA/DGTEST3/ONLINELOG/group_3.458.1069533179
2021-09-20T09:59:54.086085+00:00
Incomplete Recovery applied until change 4236455 time 09/20/2021 09:52:41
2021-09-20T09:59:54.094097+00:00
Flashback Media Recovery Complete
Completed: FLASHBACK DATABASE TO SCN 4236454
alter database convert to physical standby
2021-09-20T09:59:54.263414+00:00
ALTER DATABASE CONVERT TO PHYSICAL STANDBY (dgtest3)
Clearing standby activation ID 4040262352 (0xf0d182d0)
The primary database controlfile was created using the
'MAXLOGFILES 16' clause.
There is space for up to 13 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 209715200;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 209715200;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 209715200;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 209715200;
Offline data file 2 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 4 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 6 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 8 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 9 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 10 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
RSM0 (PID:5997): Database role changed from PRIMARY to PHYSICAL STANDBY [kcvs.c:8936]
2021-09-20T09:59:54.325231+00:00
RSM0 (PID:5997): Waiting for all non-current ORLs to be archived
2021-09-20T09:59:54.325324+00:00
RSM0 (PID:5997): All non-current ORLs have been archived
RSM0 (PID:5997): Clearing online redo logfile 1 +DATA/DGTEST3/ONLINELOG/group_1.299.1069533175
RSM0 (PID:5997): Clearing online redo logfile 2 +DATA/DGTEST3/ONLINELOG/group_2.300.1069533177
RSM0 (PID:5997): Clearing online redo logfile 3 +DATA/DGTEST3/ONLINELOG/group_3.301.1069533179
Clearing online log 1 of thread 1 sequence number 380
Clearing online log 2 of thread 1 sequence number 381
Clearing online log 3 of thread 1 sequence number 382
2021-09-20T10:00:03.959609+00:00
RSM0 (PID:5997): Clearing online redo logfile 1 complete
RSM0 (PID:5997): Clearing online redo logfile 2 complete
RSM0 (PID:5997): Clearing online redo logfile 3 complete
RSM0 (PID:5997): RT: Role transition work is not done
RSM0 (PID:5997): Redo network throttle feature is disabled at mount time
Physical Standby Database mounted.
2021-09-20T10:00:03.970236+00:00
In-memory operation on ADG is currently only supported on Engineered systems and PaaS.
inmemory_adg_enabled is turned off automatically.
Please contact our support team for EXADATA solutions
CONVERT TO PHYSICAL STANDBY: Complete - Database mounted as physical standby

At the same time the DG broker will add log archive destination on the primary for the reinstated physical standby.

ALTER SYSTEM SET log_archive_dest_3='service="dgtest3tns"','SYNC AFFIRM delay=0 optional compression=disable max_failure=0 reopen=300 db_unique_name="dgtest3" net_timeout=30','valid_for=(online_logfile,all_roles)' SCOPE=BOTH;

The other standby will get an update on its fal_server with reference to the reinstated standby.

ALTER SYSTEM SET fal_server='dgtest2tns','dgtest3tns' SCOPE=BOTH;

At the end the data guard configuration will have same number of standbys as before but with a different primary.

DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest2 - Primary database
    dgtest  - Physical standby database
    dgtest3 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 58 seconds ago)

↧

Active Data Guard and PDB Parameters

October 1, 2021, 6:00 pm

≫ Next: Patching OS in DBCS VM DB Using DBCLI

≪ Previous: Failover and Reinstate With Multiple Physical Standbys

In an earlier post it was shown how undo_retention value in a PDB that is part of an ADG standby could only be changed by updating the value on the primary PDB. In the case of undo_retention parameter Oracle introduced change 30577591 which prevented it from inheriting the value from the standby CDB.
This post looks at the effects of setting parameter values at PDB level in a ADG enviornment. It seems that the PDB in a standby CDB will inherit the init values from cdb$root spfile as long as the parameter is not set at PDB level in the primary.
open_cursors parameter is used as the test case here. The initial values at primary and standby are same and no value has been set at PDB level.

SQL> show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- ----------
open_cursors                         integer     3000

On the primary the value is changed at CDB level.

SQL> alter system set open_cursors=3500 scope=both;

System altered.

SQL> show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- ----------
open_cursors                         integer     3500

As no value has been set at PDB level it will inherit the value from CDB$root.

SQL> alter session set container=dgpdb;

Session altered.

SQL> show parameter cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -----------
open_cursors                         integer     3500

Next the value is changed at PDB level.

SQL> alter session set container=dgpdb;

SQL>  alter system set open_cursors=2500;

System altered.

SQL> show parameter cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -----------
open_cursors                         integer     2500

At this stage primary cdb$root has value 3500 and PDB has 2500. No changes has been made to standby and value there remains 3000.

On the standby checking the value in the PDB shows the initial value.

SQL>  alter session set container=dgpdb;

Session altered.

SQL> show parameter cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -------------
open_cursors                         integer     3000

Changing the parameter value at CDB level and check value in PDB.

SQL>  alter system set open_cursors=4500 scope=both;

System altered.

SQL>  show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- ---------
open_cursors                         integer     4500

Value checked at PDB level.

SQL> alter session set container=dgpdb;

Session altered.

SQL> show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -------
open_cursors                         integer     4500

This shows value being inherited from standby cdb$root.
Next the PDB in the standby instance is restarted and parameter value checked.

SQL> shutdown ;
Pluggable Database closed.

SQL> startup;
Pluggable Database opened.

SQL> show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -----------
open_cursors                         integer     2500

It not longer shows the 4500 value inherited from standby but the value set at primary PDB. It appears during PDB start the table PDB_SPFILE$ is read for PDB level parameter values. Since PDB_SPFILE$ is a database table, any writes to it will generate redo. As such changes made in primary PDB will cascade to all standby instances and to all relevant PDBs. When values are read from this table they are set for PDB overwriting values inherited from cdb$root spfile.
Any subsequent change to the init parameter at cdb$root doesn't override the value set at PDB level.

SQL> show con_name

CON_NAME
---------
CDB$ROOT

SQL> alter system set open_cursors=4600 scope=both;

System altered.

SQL> alter session set container=dgpdb;

Session altered.

SQL>  show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------
open_cursors                         integer     2500

This behaviour, inheriting from local cdb$root spfile vs primary PDB could have consequences for asymmetric data guard deployments where primary and standy differ in the amount of resources (memory,cpu) and user requirments (BI vs OLTP, user counts, etc). If for whatever an init value is set on PDB level at primary, remember that will get applied on all standby PDBs next time those are restarted.

↧