We had an interesting scenario where we did a full rman refresh of a standby database becuase the archivelogs went missing and there was a gap. After doing the restore we got the below error in the RMAN recovery

rman nocatalog target sys/******@mw auxiliary /
connected to target database: MW (DBID=000000000)
using target database control file instead of recovery catalog
connected to auxiliary database: MW (not mounted)
RMAN> DUPLICATE TARGET DATABASE FOR STANDBY NOFILENAMECHECK DORECOVER;
The restore was succesful but the recovery gave below error

Oracle Error:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: ‘+DATAC1/mwdr/datafile/system.572.867661777’
After doing a lot of googling and searching support.oracle.com it turns out it has something to do with the
incarnation of databases. We checked the incarnation for both the Primary and Standby database.

Output for Primary Database
RMAN> list incarnation of database;
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
——- ——- ——– —————- — ———- ———-
1 1 MW 1312370394 PARENT 1 24-AUG-13
2 2 MW 1312370394 CURRENT 925702 10-JUL-14
Output for Standby Database

RMAN> list incarnation of database;

List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
——- ——- ——– —————- — ———- ———-
1 1 MW 1312370394 PARENT 1 24-AUG-13
2 2 MW 1312370394 ORPHAN 925702 10-JUL-14
4 4 MW 1312370394 ORPHAN 20799166 23-OCT-14
3 3 MW 1312370394 CURRENT 22477563 30-OCT-14

Solution:

As we can see clearly from the output the primary database is currenly on a different incarnation from the standby database. The way to address it is to reset the incarnation on the standby database to match the primary incarnation.
RMAN> reset database to incarnation 2;

RMAN> list incarnation of database;
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
——- ——- ——– —————- — ———- ———-
1 1 MW 1312370394 PARENT 1 24-AUG-13
2 2 MW 1312370394 CURRENT 925702 10-JUL-14
4 4 MW 1312370394 ORPHAN 20799166 23-OCT-14
3 3 MW 1312370394 ORPHAN 22477563 30-OCT-14
As you can now the incarnations match. Now do the recovery for the database again and the logs will apply and start shipping like normal again.

For running a single SQL Command via sqlplus on multiple servers, we have to first do a few pre-requisites

1. Oracle client should be installed on the Unix/Linux Server

2. Create Local TNS entry in the tnsnames.ora file for the databases where you will run the commands

3. All databases should have one common user with a common password (this method is not the safest method
since the password will be in plain text in your Shell script)
Now let us proceed to first create a text file called dbnodes.txt which will have the TNSNAMES for the
database we will connect to..

boston
chicago
newyork

Above is the entries in the dbnodes.txt file, these are the databases where we will loop the sql commands

Create a sql script with the commands you have to run on all the databases, lets call the file script.sql

set echo on
set linesize 200
set pages 0
select sysdate from dual;
select user from dual;
select instance_name,host_name from gv$instance;
alter system set cpu_count=8 scope=both sid='*';
show parameter cpu_count;
alter session set nls_date_format='DD/MM/YYYY';
exit;


I do this to dynamically to cap the cpu_count on some of our databases to contain any resouce hogging.

Now the final shell script which will call the dbnodes.txt and script.sql to loop the above sql commands
through all databases. The shell script is called dbloop.sh

#!/bin/bash
cat dbnodes.txt | while read line
do
sqlplus -s user/user123@$line @/u03/scripts/script.sql
done

Copy all 3 files dbnodes.txt, script.sql and dbloop.sh to one directiry and run it like below; to collect the log of the SQL commands.

$--> dbloop.sh > dbrun.log

This will spool the output to a logfile.

So there you see it, one of the easiest ,method to run a set ofcommon sql commands on hundreds of server. I use this script to do basics like checking a certain parameters on all our production databases, to check dataguard status for multiple production DB’s etc. The use cases of this script is unlimited. Hope you enjoyed it :) Keep it Easy and Keep It Oracle !

Oracle has just announced a suite of new Engineered systems. Of particular significance is the announcement of the Generation 6 of Exadata Engineered system. I think the disruptive feature of this release is VM on Exadata. I wonder how well does that fit with the Muti-Tenant option with 12c. It could prove to be counterproductive for Oracle. I don’t see the use of it when you can do workload paritioning using Instance Caging, DBRM and IORM.

Link Below with more information:

https://www.oracle.com/engineered-systems/exadata/database-machine-x5-2/index.html

Oracle Transparent Data Encryption using Oracle Wallet

If you wish to use a wallet specifically for TDE, then you must specify a wallet location in the sqlnet.ora file by using the ENCRYPTION_WALLET_LOCATION parameter. Oracle recommends that you use ENCRYPTION_WALLET_LOCATION parameter to specify a wallet location for TDE.

ENCRYPTION_WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA =
(DIRECTORY = /u02/app/oracle/admin/swxdev/wallet)
)
)

SQL> select * from v$encryption_wallet;

WRL_TYPE
--------------------
WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS
------------------
file
/u02/app/oracle/admin/swxdev/wallet
OPEN

SQL> ALTER SYSTEM SET ENCRYPTION KEY IDENTIFIED BY "***";

SQL> ALTER SYSTEM SET ENCRYPTION WALLET OPEN IDENTIFIED BY "***";

SQL> ALTER SYSTEM SET ENCRYPTION WALLET CLOSE IDENTIFIED BY "****";

-- Modify Wallet for Auto-Login --

orapki wallet create -wallet /u02/app/oracle/admin/swxdev/wallet -auto_login

CREATE BIGFILE TABLESPACE SWX_ENCRYPTED DATAFILE
'+DATAC1' SIZE 25G AUTOEXTEND ON NEXT 250M MAXSIZE 33554431M
LOGGING
ONLINE
EXTENT MANAGEMENT LOCAL AUTOALLOCATE
BLOCKSIZE 8K
SEGMENT SPACE MANAGEMENT AUTO
encryption using 'AES256'
default storage (encrypt);

SELECT * FROM V$ENCRYPTED_TABLESPACES;

— Change Wallet Location to ACFS, has to be done in RAC where the Wallet files needs to be on a Shared Cluster File System —

1. Copy wallet files to new location

cp -p /u02/app/oracle/admin/swxdev/wallet/* /ACFS/wallet

2. Go to the sqlnet.ora file where ENCRYPTION_WALLET_LOCATION  location is defined. Change to new path /ACFS/wallet

ENCRYPTION_WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA =
(DIRECTORY = /ACFS/wallet)
)
)


3. Run orapki utility to modify the wallet location for auto-login

orapki wallet create -wallet /ACFS/wallet -auto_login

4. Bounce the databases for which the encryption wallet has been set

— Add Wallet to RAC Database —

If you check the 2nd instance, you can see the wallet is closed on 2nd node and the location points to the default location

SQL> select * from v$encryption_wallet;

WRL_TYPE
--------------------
WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS
------------------
file
/u02/app/oracle/admin/swxtst/wallet
CLOSED

1. Since we have already moved the Encryption wallet to the ACFS shared file system, we have to copy the sqlnet.ora file from Node1 to

Node 2

scp -r /u02/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora

oracle@exatestdbadm02:/u02/app/oracle/product/11.2.0/dbhome_1/network/admin/

2. Now check the Instance on Node 2

SQL> select * from v$encryption_wallet;

WRL_TYPE
--------------------
WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS
------------------
file
/ACFS/wallet
OPEN

And you can now see the wallet is open.

Starting Testing for In-Memory Column Store on 12.1.0.2. Will post the results for the test very soon on the blog.

Since it was announced in Early 2010 that Oracle GoldenGate will be the RoadMap for Replication for Oracle Database, The adoption of GoldenGate over Streams has been quite Rapid. The complexity of Streams has intimidated many adopters, specially for Multi-Master or Peer-to-Peer Replication. GoldenGate not only makes MultiMaster Replication easy but is so full of options and topologies that anybody who has used Golden Gate would most likely never go back to Oracle Streams.

In the Example below we will create a Bi-Directional Model where both the Tables have same name in different databases and can be used for INSERT’ing transactions. The CR( Conflict Resolution) is handled by GG itself.

You can download the latest version of GoldenGate from OTN >> http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

We will use the Sample Table EMP in Sample Schema SCOTT . Before we begin, the assumption is that you have installed and created Oracle Database 12c in both Site A and Site B for creating the replication, (This setup was done on Windows platform, there isn’t much difference between the Linux and Windows setup)

Site A : GGPRIM

————————

Host OS: Windows 2008R2 X64

DB Version: Oracle 12c 12.1.0.2

GoldenGate Version : 12.1.2

Site A : GGSTANDBY

————————-

Host OS: Windows 2008R2 X64

DB Version: Oracle 12c 12.1.0.2

GoldenGate Version : 12.1.2

With 12c the GoldenGate installation is GUI based. GG12c will need to be installed on both Sites and ensure that both databases are in archive log mode.

1. Enable Supplemental Logging
** Do Below Steps in Both Databases **

SQL> select supplemental_log_data_min from v$database;

SUPPLEME
--------
NO

SQL> alter database add supplemental log data;
Database altered.

SQL> select supplemental_log_data_min from v$database;

SUPPLEME
--------
YES

SQL> create user ggate identified by ggate;

SQL> grant dba to ggate;
SQL> conn scott/scott

2. At SITE A, Create the extract (EXT1) and data pump (DPUMP1) —

Site A Home Directory of GG : “D:\app\sql_admin\product\12.1.2\oggcore_1”

add extract ext1 tranlog begin now

add exttrail D:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\aa extract ext1

add extract dpump1 exttrailsource D:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\aa

— The below Directory is remote directory on other Database Server —

ADD RMTTRAIL F:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ab, EXTRACT DPUMP1

edit params ext1

EXTRACT ext1
USERID ggate, PASSWORD ggate
EXTTRAIL D:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\aa
TRANLOGOPTIONS EXCLUDEUSER ggate
TABLE scott.emp;

edit params dpump1

EXTRACT dpump1
USERID ggate, PASSWORD ggate
RMTHOST 172.21.104.49, MGRPORT 7809, TCPBUFSIZE 100000
RMTTRAIL F:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ab
PASSTHRU
TABLE scott.emp;

3. On SITE B Add Replicat (REP2) —

Site A Home Directory of GG : “F:\app\sql_admin\product\12.1.2\oggcore_1”

ggsci> dblogin userid ggate
password : ******

ADD CHECKPOINTTABLE ggate.ggschkpt

exit

ggsci>

add replicat rep2 exttrail F:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ab checkpointtable ggate.ggschkpt
REPLICAT added.

edit params rep2

REPLICAT rep2
ASSUMETARGETDEFS
USERID ggate, PASSWORD ggate
DISCARDFILE F:\app\sql_admin\product\12.1.2\oggcore_1\discard.txt, append,
MAP scott.emp, TARGET scott.emp;

Create the extract (EXT2) and data pump (DPUMP2) on Site B

add extract ext2 tranlog begin now
EXTRACT added.

add exttrail F:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ac extract ext2
EXTTRAIL added.

add extract dpump2 exttrailsource F:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ac
EXTRACT added.

— Below Directory is Remote Directory of Other Database Server —

add rmttrail D:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ad, extract dpump2
RMTTRAIL added.

edit params ext2

EXTRACT ext2
USERID ggate, PASSWORD ggate
EXTTRAIL F:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ac
TRANLOGOPTIONS EXCLUDEUSER ggate
TABLE scott.emp;

edit params dpump2

EXTRACT dpump2
USERID ggate, PASSWORD ggate
RMTHOST 172.20.4.13, MGRPORT 7809, TCPBUFSIZE 100000
RMTTRAIL D:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ad
PASSTHRU
TABLE scott.emp;

4. On SITE A, Add replicat (REP1) —

ggsci> dblogin userid ggate
password : ******

ADD CHECKPOINTTABLE ggate.ggschkpt

exit

ggsci>

add replicat rep1 exttrail D:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ad checkpointtable ggate.ggschkpt
REPLICAT added.

edit params rep1

REPLICAT rep1
ASSUMETARGETDEFS
USERID ggate, PASSWORD ggate
DISCARDFILE D:\app\sql_admin\product\12.1.2\oggcore_1\discard.txt, append,
MAP scott.emp, TARGET scott.emp;

5. On both SITE A AND SITE B, add trandata, this steps will be done on both sites —

dblogin userid ggate password ggate
Successfully logged into database.

add trandata scott.emp

info trandata scott.emp

exit

6. At SITE A, Start the Extract and Data Pump process —

start mgr

start extract ext1
start extract dpump1
info extract ext1
info all

7. SITE B, Start the Extract and Data Pump process on Site B —

start extract ext2
start extract dpump2
info all

— SITE A —

start replicat rep1

status replicat rep1

— SITE B —

start replicat rep2

status replicat rep2

8. — ERROR in ggserr when starting extract —

If you get below error in starting extract on both sites “Operation not supported because enable_goldengate_replication is not set to true.” then set the parameter enable_goldengate_replication=true

— On Both Sites A and B —

sqlplus "/as sysdba"

alter system set enable_goldengate_replication=true scope=spfile;

Restart both the databases since it is a static parameter(this parameter is only in 11.2.0.4 onwards and in 12c)
Now restart the extract process on both Sites

start extract ext1

start extract ext2

— If you get Directory Level Errors, check on Both sides the RMTTRAIL is pointing to Remote Directory of other server respectively and Remove any Firewall Service blocking the ports servers on both —

-- In Site A --
ADD RMTTRAIL F:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ab, EXTRACT DPUMP1

-- In Site B --
ADD RMTTRAIL D:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ad, EXTRACT DPUMP2

Do this and Start the extract dpump1 and dpump2, replicat rep1 and rep2 in Both sites A and B

9. — Testing The Active Active Replication —

on Site A

Insert into SCOTT.EMP
(EMPNO, ENAME, JOB, MGR, HIREDATE, SAL, DEPTNO)
Values
(1111, 'SHADAB', 'ANALYST', 7566, TO_DATE('12/03/1981 00:00:00', 'MM/DD/YYYY HH24:MI:SS'),
3000, 20);

commit;

Check Record in Site b

On Site B

Insert into SCOTT.EMP
(EMPNO, ENAME, JOB, MGR, HIREDATE, SAL, DEPTNO)
Values
(1112, 'Yusuf', 'ANALYST', 7566, TO_DATE('12/03/1981 00:00:00', 'MM/DD/YYYY HH24:MI:SS'),
3000, 20);

commit;

Check Record in Site A

10.
————————————–
|SUMMARY OF DEFINITIONS ON BOTH SITES |
————————————–

There is a slight modification in definitions below, I created another table called tb_table_1 (code below) on both sites to demonstrate you can add as many tables as you want or even the full schema using * wildcard.

create table Tb_table_1(
user_id number primary key,
field_1 varchar2(30)
);

—- SITE A DEFINITIONS —

edit param mgr

PORT 7809

edit param ext1

EXTRACT ext1
USERID ggate, PASSWORD ggate
EXTTRAIL D:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\aa
TRANLOGOPTIONS EXCLUDEUSER ggate
TABLE scott.emp;
TABLE scott.tb_table_1;

edit param dpump1

EXTRACT dpump1
USERID ggate, PASSWORD ggate
RMTHOST 172.21.104.49, MGRPORT 7809, TCPBUFSIZE 100000
RMTTRAIL F:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ab
PASSTHRU
TABLE scott.emp;
TABLE scott.tb_table_1;

edit param rep1

REPLICAT rep1
ASSUMETARGETDEFS
USERID ggate, PASSWORD ggate
DISCARDFILE D:\app\sql_admin\product\12.1.2\oggcore_1\discard.txt, append,
MAP scott.emp, TARGET scott.emp;
MAP scott.tb_table_1, TARGET scott.tb_table_1;
TABLE scott.tb_table_1;

—- SITE B DEFINITIONS —

edit param mgr

PORT 7809

edit param ext2

EXTRACT ext2
USERID ggate, PASSWORD ggate
EXTTRAIL F:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ac
TRANLOGOPTIONS EXCLUDEUSER ggate
TABLE scott.emp;
TABLE scott.tb_table_1;


edit param dpump2

EXTRACT dpump2
USERID ggate, PASSWORD ggate
RMTHOST 172.20.4.13, MGRPORT 7809, TCPBUFSIZE 100000
RMTTRAIL D:\app\sql_admin\product\12.1.2\oggcore_1\dirdat\ad
PASSTHRU
TABLE scott.emp;
TABLE scott.tb_table_1;

edit param rep2

REPLICAT rep2
ASSUMETARGETDEFS
USERID ggate, PASSWORD ggate
DISCARDFILE F:\app\sql_admin\product\12.1.2\oggcore_1\discard.txt, append,
MAP scott.emp, TARGET scott.emp;
MAP scott.tb_table_1, TARGET scott.tb_table_1;

1. Download Software as Zip file from OTN
http://www.oracle.com/technetwork/developer-
tools/apex/downloads/index.html

2.Unzip apex_5.0_en.zip Software to ORACLE_HOME directory,
it will create a directory called Apex

3. Login to SQLPLUS as SYSDBA in database where you will
create the Apex instance. Create a tablespace called
“apex” for the Apex 5 installation

$ cd $ORACLE_HOME/apex

SELECT dbms_xdb.gethttpport FROM dual;

EXEC DBMS_XDB.SETHTTPPORT(0);

4. — Full development environment. Run apexins.sql

@apexins.sql tablespace_apex tablespace_files
tablespace_temp images

tablespace_apex is the name of the tablespace for the
Oracle Application Express application user.

tablespace_files is the name of the tablespace for the
Oracle Application Express files user.

tablespace_temp is the name of the temporary tablespace or
tablespace group.

images is the virtual directory for Oracle Application
Express images. To support future Oracle Application
Express upgrades, define the virtual image directory as
/i/.

@apexins.sql apex apex temp /i/

images is the virtual directory for Oracle Application
Express images. To support future Oracle Application
Express upgrades, define the virtual image directory as
/i/.

@apxldimg.sql APEX_HOME

APEX_HOME is the directory you specified when unzipping
the file, in our case it is the ORACLE_HOME directory

SQL> @apxldimg.sql /u02/app/oracle/product/11.2.0/dbhome_1

5. Run apxchpwd.sql to create Admin account and Reset the
password —
@apxchpwd.sql


Alter/Unlock the anonymous account


alter user anonymous account unlock;


If you get this error ORA-20001: Password validation
failed.

Then there is a bug in Apex 5 for the password validation

–Execute Below Query and Generate DATA as INSERT
statements–

select t.name, t.value--, t.pref_desc
from apex_040100.wwv_flow_platform_prefs t
where t.name in ('ACCOUNT_LIFETIME_DAYS'
,'PASSWORD_ALPHA_CHARACTERS'
,'PASSWORD_PUNCTUATION_CHARACTERS'
,'STRONG_SITE_ADMIN_PASSWORD'
,'USERNAME_VALIDATION')

— Insert Data into APEX_050000 schema and Check again the
values —

select t.name, t.value--, t.pref_desc
from apex_050000.wwv_flow_platform_prefs t
where t.name in ('ACCOUNT_LIFETIME_DAYS'
,'PASSWORD_ALPHA_CHARACTERS'
,'PASSWORD_PUNCTUATION_CHARACTERS'
,'STRONG_SITE_ADMIN_PASSWORD'
,'USERNAME_VALIDATION')

—————————————————-
Configuring the PL/SQL Embedded Gateway
—————————————————-

6. Running the apex_epg_config.sql Configuration Script to
the base directory where Apex software was unzipped. In
our case it is the DB home directory

@apex_epg_config.sql
/u02/app/oracle/product/11.2.0/dbhome_1


7. Conifuringthe Oracle XML DB Protocol Server Port

SELECT DBMS_XDB.GETHTTPPORT FROM DUAL;

EXEC DBMS_XDB.SETHTTPPORT(8080);

8. Enabling Network Services in Oracle Database 11g or
Later

By default, the ability to interact with network services
is disabled in Oracle Database 11g Release 1 or 2 or
later. Therefore, if you are running Oracle Application
Express with Oracle Database 11g Release 1 or 2 or later,
you must use the new DBMS_NETWORK_ACL_ADMIN package to
grant connect privileges to any host for the APEX_050000
database user. Failing to grant these privileges

Grant connect privileges to any host for the APEX_050000
database user

— for Apex 5 —

DECLARE
ACL_PATH VARCHAR2(4000);
BEGIN
-- Look for the ACL currently assigned to '*' and give APEX_050000
-- the "connect" privilege if APEX_050000 does not have the privilege yet.

SELECT ACL INTO ACL_PATH FROM DBA_NETWORK_ACLS
WHERE HOST = '*' AND LOWER_PORT IS NULL AND UPPER_PORT IS NULL;

IF DBMS_NETWORK_ACL_ADMIN.CHECK_PRIVILEGE(ACL_PATH, 'APEX_050000',
'connect') IS NULL THEN
DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE(ACL_PATH,
'APEX_050000', TRUE, 'connect');
END IF;

EXCEPTION
-- When no ACL has been assigned to '*'.
WHEN NO_DATA_FOUND THEN
DBMS_NETWORK_ACL_ADMIN.CREATE_ACL('power_users.xml',
'ACL that lets power users to connect to everywhere',
'APEX_050000', TRUE, 'connect');
DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL('power_users.xml','*');
END;
/
COMMIT;

— For Apex 4 —

DECLARE
ACL_PATH VARCHAR2(4000);
BEGIN
-- Look for the ACL currently assigned to '*' and give APEX_040200
-- the "connect" privilege if APEX_040200
-- does not have the privilege yet.

SELECT ACL INTO ACL_PATH FROM DBA_NETWORK_ACLS
WHERE HOST = '*' AND LOWER_PORT IS NULL AND UPPER_PORT IS NULL;

IF DBMS_NETWORK_ACL_ADMIN.CHECK_PRIVILEGE(ACL_PATH, 'APEX_040200',
'connect') IS NULL THEN
DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE(ACL_PATH,
'APEX_040200', TRUE, 'connect');
END IF;

EXCEPTION
-- When no ACL has been assigned to '*'.
WHEN NO_DATA_FOUND THEN
DBMS_NETWORK_ACL_ADMIN.CREATE_ACL('power_users.xml',
'ACL that lets power users to connect to everywhere',
'APEX_040200', TRUE, 'connect');
DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL('power_users.xml','*');
END;
/
COMMIT;

Now access the URL for Apex, becuase we configured the
Embedded PL/SQL Gateway, go to:

http://hostname:port/apex/apex_admin


In case you get below error then check if port 8080 is
taken by some other service on the server, change the port
to a new one and restart the database.

“APEX.ERROR.ERROR_PAGE.UNHANDLED_ERROR”

EXEC DBMS_XDB.SETHTTPPORT(8898);

alter system register;

srvctl stop database -d dbname

srvctl start database -d dbname

http://gavinsoorma.com/2012/11/ash-and-awr-performance-tuning-scripts/

prompt ************************************
prompt **** ASH OVERALL WAIT PROFILE
prompt ************************************
set lines 999

SELECT MIN(sample_time) min_ash_available,sysdate-MIN(sample_time) available duration FROM v$active_session_history;

select * from (
select NVL(event,’CPU’) event,count(*),
round((ratio_to_report(sum(1)) over ()*100),1) rr
from gv$active_session_history
WHERE user_id0
AND sample_timetrunc(sysdate-1)
group by event
order by 2 desc
) where rownum<10;

After a memory upgrade in our RAC servers, the Cluster wasn’t coming up after a reboot. We first tried to start the CRS in init mode but even that failed. Upon checking the CRS alert log we discovered that the ASM was failing to start due to below error.

Error:

CRS-5017: The resource action “ora.asm start” encountered the following error:
ORA-01017: invalid username/password; logon denied
. For details refer to “(:CLSN00107:)” in “/u01/app/grid/diag/crs/<hostname>/crs/trace/ohasd_oraagent_grid.trc”

Solution:

We discovered the above error in the CRS alert log. After knocking our heads around, trying to start crsd in init mode without any success. We eventually discovered it was related to SQLNET.AUTHENTICATION_SERVICES parameter in sqlnet.ora file in the Grid Home. We removed the entry in the sqlnet.ora file from both nodes of the cluster and restarted the cluster. Everything was back to normal after that. Another day of troubleshooting CRS without any help from Oracle Support.

A 600 slide presentation by Oracle DB Product Manager Mike Dietrich, to upgrade and migrate to Oracle 12c.

Upgrade_and_Migrate_12c

ORA-00245: control file backup failed; target is likely on a local file system

RMAN-08132: WARNING: cannot update recovery area reclaimable file list
Solution : This is due to the snapshot of the control file being on a local filesystem in a RAC database. Change the snapshot control file back to a shared disk

RMAN > CONFIGURE SNAPSHOT CONTROLFILE NAME TO ‘+DATAC1/DB004/snapcf.f’;

One of the best sites for scripts related to checking library cache locks…

http://allappsdba.blogspot.qa/2012/04/to-check-library-cache-lock-contention.html

While installing Oracle 12c on Linux you can get this error. The solution is very easy just unset the ORA_CRS_HOME environment variable and run the runInstaller again

$ unset ORA_CRS_HOME

Synonym is an alternative name for a table, view, sequence, procedure, stored function, package, materialized view, Java class schema object, user-defined object type, or another synonym.

— Create a Public Synonym for a Table —

CREATE OR REPLACE PUBLIC SYNONYM EASYNAME for EASYORADBA.TABLE00001 ;

— Drop a Public Synonym for a Table —

DROP PUBLIC SYNONYM EASYNAME ;

— View to Check Synonyms in Oracle —

SELECT * from DBA_SYNONYMS;

Before we begin this is a note from the Oracle Website

The name of these bundle patches was changed to “Database Proactive Bundle Patch” in April 2016.

The patches include fixes for both Engineered Systems and for DB In-Memory.
They can be used on both Exadata and non-Exadata systems, and can be used for both RAC and non-RAC configurations.

Quarterly patches are released on all platforms.
Monthly patches may only be released on a subset of platforms.

DB Version : 12.1.0.2

OS : OEL 6 Update 6, Linux x86_64

Single Instance No RAC

Multitenant Option Enabled

1 CDB, 1 PDB

CDB Name: CDBDEV

PDB Name: PDBDEV

The database bundled patches are now called “Database Proactive Bundle Patch” from April 2016.

Patch 22291127 – Database Patch Set Update 12.1.0.2.160419 (Includes CPUApr2016) – Single Instance DB

Patch 22899531: DATABASE PROACTIVE BUNDLE PATCH 12.1.0.2.160419 (APR2016)- DB Plus Grid Infrastructure

Also the naming convention has changed like below

Patch description:  “Database Patch Set Update : 12.1.0.2.160419 (22291127)”

   Created on 6 Apr 2016, 03:46:21 hrs PST8PDT

Sub-patch  21948354; “Database Patch Set Update : 12.1.0.2.160119 (21948354)”

Sub-patch  21359755; “Database Patch Set Update : 12.1.0.2.5 (21359755)”

Sub-patch  20831110; “Database Patch Set Update : 12.1.0.2.4 (20831110)”

Sub-patch  20299023; “Database Patch Set Update : 12.1.0.2.3 (20299023)”

Sub-patch  19769480; “Database Patch Set Update : 12.1.0.2.2 (19769480)”

The left most part denotes “yymmdd” as a post-fix to now which date is the PSU from

1. Download the latest patches for your platform from Oracle Support Website.
2. unzip the patch

unzip p22291127_12102_<platform>.zip

cd 22291127

Go to Directory of Patch and Apply the Patch

[oracle@localhost 22291127]$ /home/oracle/app/oracle/product/12.1.0/dbhome_1/OPatch/opatch prereq CheckCOnflictAgainstOHWithDetail -ph ./

Oracle Interim Patch Installer version 12.1.0.1.3

Copyright (c) 2016, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /home/oracle/app/oracle/product/12.1.0/dbhome_1

Central Inventory : /home/oracle/app/oraInventory

   from           : /home/oracle/app/oracle/product/12.1.0/dbhome_1/oraInst.loc

OPatch version    : 12.1.0.1.3

OUI version       : 12.1.0.2.0

Log file location : /home/oracle/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-05-13_08-38-21AM_1.log

Invoking prereq “checkconflictagainstohwithdetail”

Prereq “checkConflictAgainstOHWithDetail” passed.

OPatch succeeded.

1. Download the latest version of Patch before applying the patches. You can search latest Patch by searching for “Patch 6880880”
2. Shutdown the Database and Listener
3. Go to Directory of the Patch and Apply the Patch

[oracle@localhost 22291127]$ /home/oracle/app/oracle/product/12.1.0/dbhome_1/OPatch/opatch apply

Oracle Interim Patch Installer version 12.1.0.1.10

Copyright (c) 2016, Oracle Corporation.  All rights reserved.

Oracle Home       : /home/oracle/app/oracle/product/12.1.0/dbhome_1

Central Inventory : /home/oracle/app/oraInventory

   from           : /home/oracle/app/oracle/product/12.1.0/dbhome_1/oraInst.loc

OPatch version    : 12.1.0.1.10

OUI version       : 12.1.0.2.0

Log file location : /home/oracle/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-05-13_08-44-41AM_1.log

Verifying environment and performing prerequisite checks…

OPatch continues with these patches:   19769480  20299023  20831110  21359755  21948354  22291127

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

Provide your email address to be informed of security issues, install and

initiate Oracle Configuration Manager. Easier for you if you use your My

Oracle Support Email address/User Name.

Visit http://www.oracle.com/support/policies.html for details.

Email address/User Name:

You have not provided an email address for notification of security issues.

Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = ‘/home/oracle/app/oracle/product/12.1.0/dbhome_1’)

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files…

Applying sub-patch ‘19769480’ to OH ‘/home/oracle/app/oracle/product/12.1.0/dbhome_1’

Patching component oracle.rdbms.deconfig, 12.1.0.2.0…

Patching component oracle.xdk, 12.1.0.2.0…

Patching component oracle.tfa, 12.1.0.2.0…

Patching component oracle.rdbms.util, 12.1.0.2.0…

Patching component oracle.rdbms, 12.1.0.2.0…

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0…

Patching component oracle.xdk.parser.java, 12.1.0.2.0…

Patching component oracle.oraolap, 12.1.0.2.0…

Patching component oracle.xdk.rsf, 12.1.0.2.0…

Patching component oracle.rdbms.rsf, 12.1.0.2.0…

Patching component oracle.rdbms.rman, 12.1.0.2.0…

Patching component oracle.ldap.rsf, 12.1.0.2.0…

Patching component oracle.ldap.rsf.ic, 12.1.0.2.0…

Applying sub-patch ‘20299023’ to OH ‘/home/oracle/app/oracle/product/12.1.0/dbhome_1’

ApplySession: Optional component(s) [ oracle.has.crs, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.tfa, 12.1.0.2.0…

Patching component oracle.rdbms.deconfig, 12.1.0.2.0…

Patching component oracle.rdbms.rsf, 12.1.0.2.0…

Patching component oracle.rdbms, 12.1.0.2.0…

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0…

Patching component oracle.rdbms.rsf.ic, 12.1.0.2.0…

Patching component oracle.ldap.rsf, 12.1.0.2.0…

Patching component oracle.ldap.rsf.ic, 12.1.0.2.0…

Applying sub-patch ‘20831110’ to OH ‘/home/oracle/app/oracle/product/12.1.0/dbhome_1’

Patching component oracle.rdbms, 12.1.0.2.0…

Patching component oracle.oraolap.dbscripts, 12.1.0.2.0…

Patching component oracle.ldap.rsf, 12.1.0.2.0…

Patching component oracle.tfa, 12.1.0.2.0…

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0…

Patching component oracle.rdbms.rsf, 12.1.0.2.0…

Applying sub-patch ‘21359755’ to OH ‘/home/oracle/app/oracle/product/12.1.0/dbhome_1’

Patching component oracle.assistants.server, 12.1.0.2.0…

Patching component oracle.rdbms.rsf, 12.1.0.2.0…

Patching component oracle.rdbms, 12.1.0.2.0…

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0…

Patching component oracle.tfa, 12.1.0.2.0…

Applying sub-patch ‘21948354’ to OH ‘/home/oracle/app/oracle/product/12.1.0/dbhome_1’

Patching component oracle.rdbms.deconfig, 12.1.0.2.0…

Patching component oracle.xdk, 12.1.0.2.0…

Patching component oracle.tfa, 12.1.0.2.0…

Patching component oracle.rdbms, 12.1.0.2.0…

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0…

Patching component oracle.xdk.parser.java, 12.1.0.2.0…

Patching component oracle.rdbms.rsf, 12.1.0.2.0…

Patching component oracle.xdk.rsf, 12.1.0.2.0…

Applying sub-patch ‘22291127’ to OH ‘/home/oracle/app/oracle/product/12.1.0/dbhome_1’

ApplySession: Optional component(s) [ oracle.oid.client, 12.1.0.2.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.ldap.rsf, 12.1.0.2.0…

Patching component oracle.ldap.client, 12.1.0.2.0…

Patching component oracle.tfa, 12.1.0.2.0…

Patching component oracle.rdbms, 12.1.0.2.0…

Patching component oracle.rdbms.dbscripts, 12.1.0.2.0…

Patching component oracle.rdbms.rsf, 12.1.0.2.0…

Patching component oracle.oraolap, 12.1.0.2.0…

Patching component oracle.rdbms.rman, 12.1.0.2.0…

Patching component oracle.oraolap.dbscripts, 12.1.0.2.0…

Composite patch 22291127 successfully applied.

Log file location: /home/oracle/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-05-13_08-44-41AM_1.log

OPatch succeeded.

1. Post- Patch Installation

The datapatch utility will then run the necessary apply scripts to load the modified SQL files into the database. An entry will be added to the dba_registry_sqlpatch view reflecting the patch application. In the dba_registry_sqlpatch view, verify the Status for the APPLY is “SUCCESS”.

Startup the Database

sqlplus “/as sysdba”

startup

alter pluggable database all open;

[oracle@localhost 22291127]$ /home/oracle/app/oracle/product/12.1.0/dbhome_1/OPatch/datapatch -verbose

SQL Patching tool version 12.1.0.2.0 on Fri May 13 08:50:32 2016

Copyright (c) 2015, Oracle.  All rights reserved.

Log file for this invocation: /home/oracle/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_7427_2016_05_13_08_50_32/sqlpatch_invocation.log

Connecting to database…OK

Note:  Datapatch will only apply or rollback SQL fixes for PDBs

       that are in an open state, no patches will be applied to closed PDBs.

       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation

       (Doc ID 1585822.1)

Bootstrapping registry and package to current versions…done

Determining current state…done

Current state of SQL patches:

Bundle series PSU:

  ID 160419 in the binary registry and not installed in any PDB

Adding patches to installation queue and performing prereq checks…

Installation queue:

  For the following PDBs: CDB$ROOT PDB$SEED PDBDEV

    Nothing to roll back

    The following patches will be applied:

      22291127 (Database Patch Set Update : 12.1.0.2.160419 (22291127))

Installing patches…

Patch installation complete.  Total patches installed: 3

Validating logfiles…

Patch 22291127 apply (pdb CDB$ROOT): SUCCESS

  logfile: /home/oracle/app/oracle/cfgtoollogs/sqlpatch/22291127/19694308/22291127_apply_CDBDEV_CDBROOT_2016May13_08_52_30.log (no errors)

Patch 22291127 apply (pdb PDB$SEED): SUCCESS

  logfile: /home/oracle/app/oracle/cfgtoollogs/sqlpatch/22291127/19694308/22291127_apply_CDBDEV_PDBSEED_2016May13_08_52_51.log (no errors)

Patch 22291127 apply (pdb PDBDEV): SUCCESS

  logfile: /home/oracle/app/oracle/cfgtoollogs/sqlpatch/22291127/19694308/22291127_apply_CDBDEV_PDBDEV_2016May13_08_52_50.log (no errors)

SQL Patching tool complete on Fri May 13 08:53:04 2016

If you are using the Oracle Recovery Manager, the catalog needs to be upgraded. Enter the following command to upgrade it:

$ rman catalog username/password@alias

RMAN> UPGRADE CATALOG;

The datapatch will run the SQL load files in CDB and all PDB’s as well.

So that’s it. We have patched the Oracle 12c with CDB and PDB to the latest PSU.

Query to check sessions writing to flashcache on Exadata
select se.sid, sn.name, s.value, se.program from v$sesstat s natural join v$statname sn left join v$session se on (s.sid = se.sid) where sn.name in (‘physical write requests optimized’, ‘cell writes to flash cache’, ‘cell overwrites in flash cache’) and s.value <> 0 order by s.sid,name;

Configure 12c dataguard broker using the below command onPrimary database server

primeprd – > tnsnames for primary database

primedr-> tnsnames for standby database

/home/oracle::11g–>dgmgrl

DGMGRL> connect sys/********

DGMGRL> create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

DGMGRL> add database primedr as connect identifier is primedr maintained as physical;

Configuration “dg_primeprd” created with primary database “primeprd”

DGMGRL> add database primedr as connect identifier is primedr maintained as physical;
Error: ORA-16698: member has a LOG_ARCHIVE_DEST_n parameter with SERVICE attribute set

Failed.
DGMGRL> remove configuration;
Removed configuration
Set the log_archive_dest_2 settings from both the Primary and Standby databases to be nothing.

alter system set log_archive_dest_2=” scope=both sid=’*’;
Disable then Enable the broker parameter on both the Primary and Standby databases.

— Primary
alter system set dg_broker_start=false scope=both sid=’*’;
alter system set dg_broker_start=true scope=both sid=’*’;

— Standby
alter system set dg_broker_start=false scope=both sid=’*’;
alter system set dg_broker_start=true scope=both sid=’*’;

On the Primary database create the broker configuration for the Primary and Standby database and this time it should work fine with no issues since the log archive destination 2 setting is not set, this is the workaround/solution.

/home/oracle::11g–>dgmgrl

DGMGRL> connect sys/**********

DGMGRL> create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

DGMGRL> add database primedr as connect identifier is primedr maintained as physical;

DGMGRL> enable configuration;
If in Broker you get below error Error: ORA-16664: unable to receive the result from a database

DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd – Primary database
primedr – Physical standby database
Error: ORA-16664: unable to receive the result from a database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR (status updated 40 seconds ago)

Check in Broker Log File.

Mon Jul 04 14:41:46 2016
Error 1017 received logging on to the standby
————————————————————
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191

Failed to send message to site . Error code is ORA-01017.

Re-create password files in ASM in 12c like below, the correct format is pwd<dbuniquename>

Primary

srvctl config database -d primeprd

orapwd file=’+DATAC1/PRIMEPRD/pwdprimeprd’ entries=10 dbuniquename=primeprd password=*** force=y ignorecase=y
Standby

srvctl config database -d primedr
orapwd file=’+DATAC1/PRIMEDR/pwdprimedr’ entries=10 dbuniquename=primedr password=*** force=y ignorecase=y

DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd – Primary database
primedr – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 18 seconds ago)

— Try Switchover Now —
DGMGRL> switchover to primedr
Performing switchover NOW, please wait…
Error: ORA-16644: apply instance not available

Failed.
Unable to switchover, primary database is still “primeprd”

DGMGRL> edit database primedr set state=’APPLY-ON’ with apply instance=’primedr1′;

DGMGRL> show database verbose primedr

Database – primedr

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 6.00 KByte/s
Active Apply Rate: 755.00 KByte/s
Maximum Apply Rate: 827.00 KByte/s
Real Time Query: OFF
Instance(s):
primedr1 (apply instance)
primedr2
DGMGRL> switchover to primedr
Performing switchover NOW, please wait…
New primary database “primedr” is opening…
Oracle Clusterware is restarting database “primeprd” …
Switchover succeeded, new primary is “primedr”
DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr – Primary database
primeprd – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 40 seconds ago)
DGMGRL> edit database primeprd set state=’APPLY-ON’ with apply instance=’primeprd1′;

DGMGRL> show database verbose primeprd

Database – primeprd

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 307.00 KByte/s
Active Apply Rate: 0 Byte/s
Maximum Apply Rate: 0 Byte/s
Real Time Query: OFF
Instance(s):
primeprd1 (apply instance)
primeprd2

DGMGRL>
DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr – Primary database
primeprd – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 55 seconds ago)

Connect to DR site broker to intiate switchover

DGMGRL> switchover to primeprd ;
Performing switchover NOW, please wait…
Operation requires a connection to instance “primeprd1” on database “primeprd”
Connecting to instance “primeprd1″…
Connected as SYSDBA.
New primary database “primeprd” is opening…
Oracle Clusterware is restarting database “primedr” …
Switchover succeeded, new primary is “primeprd

DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd – Primary database
primedr – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 11 seconds ago

1 ) Set the environment variable also via srvctl @ oracle user.

export ORACLE_UNQNAME=’$ORACLE_HOME/bin/srvctl config database |grep -w ${ORACLE_SID%?}’

srvctl setenv database -d secdev -T “ORACLE_UNQNAME=secdev”

2) Create wallet directory on both nodes @ oracle user.

mkdir -p /u01/app/oracle/WALLETS/secdev

3) Configure sqlnet.ora as follows on both nodes @ oracle user.

If the databases share the same ORACLE_HOME, they also share the same sqlnet.ora file in $TNS_ADMIN. In order to access their individual wallet, the DIRECTORY entry for the ENCRYPTION_WALLET_LOCATION needs to point each database to its own wallet location:

NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
ENCRYPTION_WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/WALLETS/$ORACLE_UNQNAME/)))

The names of the subdirectories under /u01/app/oracle/WALLETS/  reflect the ORACLE_UNQNAME names of the individual databases. That’s why we created the directory ‘secdev’ in step 2. For each Database unique name, create one directory.

4) Create the wallet by using node1 login @ oracle user.

ALTER SYSTEM SET ENCRYPTION KEY IDENTIFIED BY “*******”;

5) Open wallet by using node1 login @ oracle user.

ALTER SYSTEM SET ENCRYPTION WALLET OPEN IDENTIFIED BY “welcome1”;

6) To configure auto login for wallet by using both nodes login @ oracle user.

orapki wallet create -wallet /u01/app/oracle/WALLETS/secdev -auto_login

7) copy the below files to node2 @ oracle user.

scp ewallet.p12  node2host:/u01/app/oracle/WALLETS/secdev
8) Change permissions on directory and files in both nodes @ oracle user.

cd /u01/app/oracle/WALLETS
chmod 700 secdev
cd secdev
chmod 600 ewallet.p12

9) After initially creating the encryption wallet (and optionally a (local) auto-open wallet), navigate to the directory that stores the Oracle Wallet and set the ‘immutable’ bit with: on both nodes @ root user.

# chattr +i ewallet.p12
# chattr +i cwallet.sso
10. You can have different wallets for different databases, all we have to do this create the directories for the unique name of databases and set the environment using srvctl utlity
ENCRYPTION_WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /orabin/app/oracle/WALLETS/$ORACLE_UNQNAME/)))

srvctl setenv database -d testdb -T “ORACLE_UNQNAME=testdb”

srvctl setenv database -d ftestdb -T “ORACLE_UNQNAME=ftestdb”

mkdir -p /u01/app/oracle/WALLETS/testdb

mkdir -p /u01/app/oracle/WALLETS/ftestdb

And for the other databases  proceed as above steps

Link to Oracle Whitepaper for best practises for TDE

http://www.oracle.com/technetwork/database/security/twp-transparent-data-encryption-bes-130696.pdf

Setup for the Environment
——————-

Exadata X4-2 Quater Rack n Primary and Standby Site
Grid Version : 12.1.0.2
DB Home Version : 12.1.0.2

Primary, 2 Node RAC, db_unique_name = primeprd
Instance 2 : primeprd1
Instance 2 : primeprd2
TNS entry: PRIMEPRD, but below entry in tnsnames.ora files on both node with DB Home user, most likely it will be Oracle user.

PRIMEPRD =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = primaryhost.domain.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primeprd)
)
)

Standby, 2 Node RAC, db_unique_name= primedr
Instance 2 : primedr1
Instance 2 : primedr2
TNS ENTRY: PRIMEDR, but below entry in tnsnames.ora files on both node with DB Home user, most likely it will be Oracle user.

PRIMEDR =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = standbyhost.domain.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primedr)
)
)

1. Enable Force Logging on the Database

 alter database force logging;

Database altered.

2.1 change the below parameter in Primary database.

ALTER SYSTEM SET db_recovery_file_dest='+RECOC1' scope=both sid='*';

System altered.

alter system set LOG_ARCHIVE_CONFIG='DG_CONFIG=(primeprd,primedr)' scope=both sid='*';

alter system set LOG_ARCHIVE_DEST_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=primeprd' scope=both sid='*';

alter system set LOG_ARCHIVE_DEST_2='SERVICE=primedr LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=primedr' scope=both sid='*';

alter system set log_archive_format='%t_%s_%r.arc' scope=spfile sid='*';

alter system set log_archive_max_processes=8 scope=both sid='*';

alter system set fal_server=primedr scope=both sid='*';

alter system set fal_client=primeprd scope=both sid='*';

alter system set standby_file_management=AUTO scope=both sid='*';

alter system set db_file_name_convert=('+DATAC1/primedr/', '+DATAC1/primeprd/', '+RECOC1/primedr/', '+RECOC1/primeprd/') scope=both sid='*';

alter system set log_file_name_convert=('+DATAC1/primedr/', '+DATAC1/primeprd/', '+RECOC1/primedr/', '+RECOC1/primeprd/') scope=both sid='*';


-- Query to check if parameters are set correctly --

set linesize 500 pages 0
col value for a90
col name for a50
select name, value
from v$parameter
where name in ('db_name','db_unique_name','log_archive_config', 'log_archive_dest_1','log_archive_dest_2',
'log_archive_dest_state_1','log_archive_dest_state_2', 'remote_login_passwordfile',
'log_archive_format','log_archive_max_processes','fal_server','db_file_name_convert',
'log_file_name_convert', 'standby_file_management');

select name, value
from v$parameter
where name in ('db_name','db_unique_name','log_archive_config', 'log_archive_dest_1','log_archive_dest_2',
'log_archive_dest_state_1','log_archive_dest_state_2', 'remote_login_passwordfile',
'log_archive_format','log_archive_max_processes','fal_server','db_file_name_convert',
'log_file_name_convert', 'standby_file_management');

3.1. verify the logfile using the below querys.

select group#,THREAD#,bytes/1024/1024,status,members from v$log;
select member from v$logfile;

3.2. Add the below logfile and standby logfile in primary database, make sure you have large redolog files. In this day and age, 100MB does not really cut it.

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 1 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 2 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 3 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 4 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 5 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 6 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 7 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 8 ('+DATAC1','+RECOC1') SIZE 1G;

 select * from v$standby_log;

-- Add Standby Redolog files, number of standby redo log files = Redo logfile Groups + 2 --

ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 9 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 10 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 11 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 12 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 13 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 14 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 15 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 16 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 17 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 18 ('+DATAC1','+RECOC1') SIZE 1G;

4. Take the rman backup using below command.

rman target / nocatalog
run
{
sql 'alter system archive log current';
backup as compressed backupset database plus archivelog format '/u03/rmanbkp/online/Primary_bkp_for_stndby_%U';
backup current controlfile for standby format '/u03/rmanbkp/online/stby.ctl';
sql 'alter system archive log current';
}

5.1. create the init file using the below command in primary database.

create pfile='/u03/rmanbkp/prime/pfile_for_standby.txt' from spfile;

— Primary PFILE —

primeprd1.__data_transfer_cache_size=0
primeprd2.__data_transfer_cache_size=0
primeprd1.__db_cache_size=7046430720
primeprd2.__db_cache_size=7147094016
primeprd1.__java_pool_size=67108864
primeprd2.__java_pool_size=67108864
primeprd1.__large_pool_size=301989888
primeprd2.__large_pool_size=301989888
primeprd2.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primeprd1.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primeprd1.__pga_aggregate_target=3221225472
primeprd2.__pga_aggregate_target=3221225472
primeprd1.__sga_target=9663676416
primeprd2.__sga_target=9663676416
primeprd1.__shared_io_pool_size=469762048
primeprd2.__shared_io_pool_size=469762048
primeprd1.__shared_pool_size=1644167168
primeprd2.__shared_pool_size=1577058304
primeprd1.__streams_pool_size=100663296
primeprd2.__streams_pool_size=67108864
*.audit_file_dest='/u03/app/oracle/admin/primeprd/adump'
*.audit_trail='db'
*.cluster_database=TRUE
*.compatible='12.1.0.2.0'
*.control_files='+DATAC1/PRIMEPRD/CONTROLFILE/current.660.914322763','+DATAC1/PRIMEPRD/CONTROLFILE/current.665.914322763'
*.cpu_count=16
*.db_block_size=8192
*.db_create_file_dest='+DATAC1'
*.db_domain=''
*.db_file_name_convert='+DATAC1/primedr/','+DATAC1/primeprd/','+RECOC1/primedr/','+RECOC1/primeprd/'
*.db_name='primeprd'
*.db_recovery_file_dest='+DATAC1'
*.db_recovery_file_dest_size=536870912000
*.db_writer_processes=6
*.ddl_lock_timeout=30
*.diagnostic_dest='/u03/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primeprdXDB)'
*.fal_client='primeprd'
*.fal_server='primedr'
primeprd2.instance_number=2
primeprd1.instance_number=1
*.log_archive_config='dg_config=(primeprd,primedr)'
*.log_archive_dest_1='LOCATION=+RECOC1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=primeprd'
*.log_archive_dest_2='service=primedr LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primedr'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='log%t_%s_%r.arc'
*.log_file_name_convert='+DATAC1/primedr/','+DATAC1/primeprd/','+RECOC1/primedr/','+RECOC1/primeprd/'
*.nls_length_semantics='CHAR'
*.open_cursors=2000
*.optimizer_adaptive_features=FALSE
*.pga_aggregate_target=3072m
*.processes=1500
*.remote_login_passwordfile='exclusive'
*.resource_manager_cpu_allocation=48
*.resource_manager_plan='DEFAULT'
*.session_cached_cursors=100
*.sessions=2272
*.sga_max_size=17179869184
*.sga_target=9216m
*.standby_file_management='AUTO'
*.temp_undo_enabled=TRUE
primeprd2.thread=2
primeprd1.thread=1
primeprd2.undo_tablespace='UNDOTBS2'
primeprd1.undo_tablespace='UNDOTBS1'

— DR PFILE —

primedr1.__data_transfer_cache_size=0
primedr2.__data_transfer_cache_size=0
primedr1.__db_cache_size=7046430720
primedr2.__db_cache_size=7147094016
primedr1.__java_pool_size=67108864
primedr2.__java_pool_size=67108864
primedr1.__large_pool_size=301989888
primedr2.__large_pool_size=301989888
primedr2.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primedr1.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primedr1.__pga_aggregate_target=3221225472
primedr2.__pga_aggregate_target=3221225472
primedr1.__sga_target=9663676416
primedr2.__sga_target=9663676416
primedr1.__shared_io_pool_size=469762048
primedr2.__shared_io_pool_size=469762048
primedr1.__shared_pool_size=1644167168
primedr2.__shared_pool_size=1577058304
primedr1.__streams_pool_size=100663296
primedr2.__streams_pool_size=67108864
*.audit_file_dest='/u03/app/oracle/admin/primeprd/adump' # Create on Both Nodes
*.audit_trail='db'
*.cluster_database=TRUE
*.compatible='12.1.0.2.0'
#*.control_files='+DATAC1/PRIMEPRD/CONTROLFILE/current.660.914322763','+DATAC1/PRIMEPRD/CONTROLFILE/current.665.914322763'
*.cpu_count=16
*.db_block_size=8192
*.db_create_file_dest='+DATAC1'
*.db_domain=''
*.db_file_name_convert='PRIMEPRD','PRIMEDR' ##### Changed for dataguard
*.db_unique_name='primedr' ##### Changed for dataguard
*.db_name='primeprd'
*.db_recovery_file_dest='+RECOC1'
*.db_recovery_file_dest_size=536870912000
*.db_writer_processes=6
*.ddl_lock_timeout=30
*.diagnostic_dest='/u03/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primedrXDB)' ##### Changed for dataguard
*.fal_client='primedr' ##### Changed for dataguard
*.fal_server='primeprd' ##### Changed for dataguard
primedr2.instance_number=2
primedr1.instance_number=1
*.log_archive_config='dg_config=(primeprd,primedr)'
*.log_archive_dest_1='LOCATION=+RECOC1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=primedr' ##### Changed for dataguard
*.log_archive_dest_2='service=primeprd LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primeprd' ##### Changed for dataguard
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='log%t_%s_%r.arc'
*.log_file_name_convert='PRIMEPRD','PRIMEDR' ##### Changed for dataguard
*.remote_listener='exadr-scan:1521' ##### Changed for dataguard
*.nls_length_semantics='CHAR'
*.open_cursors=2000
*.optimizer_adaptive_features=FALSE
*.pga_aggregate_target=3072m
*.processes=1500
*.remote_login_passwordfile='exclusive'
*.resource_manager_cpu_allocation=48
*.resource_manager_plan='DEFAULT'
*.session_cached_cursors=100
*.sessions=2272
*.sga_max_size=17179869184
*.sga_target=9216m
*.standby_file_management='AUTO'
*.temp_undo_enabled=TRUE
primedr2.thread=2
primedr1.thread=1
primedr2.undo_tablespace='UNDOTBS2'
primedr1.undo_tablespace='UNDOTBS1'

cp -ip pfile_for_standby.txt pfile_to_apply.ora

vi pfile_to_apply.ora

— Actual PFILE after modifying all parameters —

primedr1.__data_transfer_cache_size=0
primedr2.__data_transfer_cache_size=0
primedr1.__db_cache_size=7046430720
primedr2.__db_cache_size=7147094016
primedr1.__java_pool_size=67108864
primedr2.__java_pool_size=67108864
primedr1.__large_pool_size=301989888
primedr2.__large_pool_size=301989888
primedr2.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primedr1.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primedr1.__pga_aggregate_target=3221225472
primedr2.__pga_aggregate_target=3221225472
primedr1.__sga_target=9663676416
primedr2.__sga_target=9663676416
primedr1.__shared_io_pool_size=469762048
primedr2.__shared_io_pool_size=469762048
primedr1.__shared_pool_size=1644167168
primedr2.__shared_pool_size=1577058304
primedr1.__streams_pool_size=100663296
primedr2.__streams_pool_size=67108864
*.audit_file_dest='/u03/app/oracle/admin/primeprd/adump'
*.audit_trail='db'
*.cluster_database=TRUE
*.compatible='12.1.0.2.0'
#*.control_files='+DATAC1/PRIMEPRD/CONTROLFILE/current.660.914322763','+DATAC1/PRIMEPRD/CONTROLFILE/current.665.914322763'
*.cpu_count=16
*.db_block_size=8192
*.db_create_file_dest='+DATAC1'
*.db_domain=''
*.db_file_name_convert='PRIMEPRD','PRIMEDR'
*.db_unique_name='primedr'
*.db_name='primeprd'
*.db_recovery_file_dest='+RECOC1'
*.db_recovery_file_dest_size=536870912000
*.db_writer_processes=6
*.ddl_lock_timeout=30
*.diagnostic_dest='/u03/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primedrXDB)'
*.fal_client='primedr'
*.fal_server='primeprd'
primedr2.instance_number=2
primedr1.instance_number=1
*.log_archive_config='dg_config=(primeprd,primedr)'
*.log_archive_dest_1='LOCATION=+RECOC1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=primedr'
*.log_archive_dest_2='service=primeprd LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primeprd'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='log%t_%s_%r.arc'
*.log_file_name_convert='PRIMEPRD','PRIMEDR'
*.remote_listener='exadr-scan:1521'
*.nls_length_semantics='CHAR'
*.open_cursors=2000
*.optimizer_adaptive_features=FALSE
*.pga_aggregate_target=3072m
*.processes=1500
*.remote_login_passwordfile='exclusive'
*.resource_manager_cpu_allocation=48
*.resource_manager_plan='DEFAULT'
*.session_cached_cursors=100
*.sessions=2272
*.sga_max_size=17179869184
*.sga_target=9216m
*.standby_file_management='AUTO'
*.temp_undo_enabled=TRUE
primedr2.thread=2
primedr1.thread=1
primedr2.undo_tablespace='UNDOTBS2'
primedr1.undo_tablespace='UNDOTBS1'

6. Create same RMAN backup directory in DR and copy the backup Piece and init file from Production to DR server.

Also do not forget to create directory for adump in path ‘/u03/app/oracle/admin/primeprd/adump’

startup nomount pfile='/u03/rmanbkp/prime/pfile_to_apply.ora';

rman target sys/*****@primeprd auxiliary /

Recovery Manager: Release 12.1.0.2.0 - Production on Tue Jun 28 10:05:31 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.

connected to target database: PRIMEPRD (DBID=111111111111111)
connected to auxiliary database: PRIMEPRD (not mounted)

show all;

using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name PRIMEPRD are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE';
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/u03/rmanbkp/prime/autobackup_control_file_%F';
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
CONFIGURE DEVICE TYPE SBT_TAPE PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/u03/rmanbkp/prime';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u03/app/oracle/product/12.1.0.2/payapps/dbs/snapcf_primeprd1.f'; # default

 DUPLICATE TARGET DATABASE FOR STANDBY NOFILENAMECHECK;

9. start the MRP process using the below command in SQL prompt.

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;

10. In case if you get error while receiving logging onto standby use the below command and create password files on both nodes of production and dr

— For Oracle 12c, do not create password files with this method. Go to section below for Dataguard Broker Setup

orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimeprd1 force=y ignorecase=y
orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimeprd2 force=y ignorecase=y

orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimedr1 force=y ignorecase=y
orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimedr2 force=y ignorecase=y

select * from v$pwfile_users;

Check by logging in from sqlplus from both production site and DR site from both nodes

From DR both nodes

sqlplus sys/*****@primeprd as sysdba

From Prod Both nodes

sqlplus sys/******@primedr as sysdba

Stop and start recovery again in DR

alter database recover managed standby database cancel;

alter database recover managed standby database using current logfile disconnect from session;

select process,status from v$managed_standby;

11. Create the spfile and add the newly created control files

show parameter control_file

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time integer 7
control_files string +DATAC1/PRIMEDR/CONTROLFILE/current.636.915704919, +RECOC1/PRIMEDR/CONTROLFILE/current.15794.915704923

Add the controlfile string in the pfile you created earlier called pfile_to_apply.ora

cd /u03/rmanbkp/prime/
vi pfile_to_apply.ora

*.control_files=’+DATAC1/PRIMEDR/CONTROLFILE/current.636.915704919′,’+RECOC1/PRIMEDR/CONTROLFILE/current.15794.915704923′

shutdown immediate;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.

startup nomount pfile='/u03/rmanbkp/prime/pfile_to_apply.ora';

create spfile='+DATAC1/primedr/spfileprimedr.ora' from pfile='/u03/rmanbkp/prime/pfile_to_apply.ora';

shutdown immediate;

cd /u03/app/oracle/product/12.1.0.2/payapps/dbs/

vi initprimedr1.ora

–add only one line pointing to the SPFILE in ASM

spfile=’+DATAC1/primedr/spfileprimedr.ora’

startup mount pfile='/u03/app/oracle/product/12.1.0.2/payapps/dbs/initprimedr1.ora';

show parameter spfile

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATAC1/primedr/spfileprimedr.
ora

alter database recover managed standby database using current logfile disconnect from session;

12. Create cluster server using the below command.

srvctl add database -d primedr -o /u03/app/oracle/product/12.1.0.2/payapps -p +DATAC1/primedr/spfileprimedr.ora -r physical_standby -a DATAC1,RECOC1 -s MOUNT
srvctl add instance -d primedr -i primedr1 -n exadrdbadm01
srvctl add instance -d primedr -i primedr2 -n exadrdbadm02

srvctl add service -db onlndr -s onlnprod -preferred onlndr1,onlndr2
srvctl add service -db primedr -s primeprod -preferred primedr1,primedr2

srvctl config service -s onlnprod -db onlndr
srvctl config service -s primeprod -db primedr

srvctl stop database -d primedr

-- if gives error stop from sqlplus

srvctl start database -d primedr -o mount

— Start Recovery Again —

 alter database recover managed standby database using current logfile disconnect from session;

13.1. Add the below entries in both production nodes tnsnames.ora file as a oracle user. (Not required from ORacle 12c 12.1.0.2 onwards)

primeprd_DGMGRL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primeprd_DGMGRL)
)
)

13.2. Add the below entries in both DR nodes tnsnames.ora file as a oracle user.

primedr_DGMGRL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primedr_DGMGRL)
)
)

13.3. Add the below entries in production node1 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primeprd_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primeprd1)
)

13.4. Add the below entries in production node2 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primeprd_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primeprd2)
)

13.5. Add the below entries in DR node1 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primedr_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primedr1)
)

13.6. Add the below entries in DR node2 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primedr_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primedr2)
)

13.7. Using the below command to restart the listener.

$ srvctl stop listener
$ srvctl start listener

14.1. Set the below parameter in primary database.

alter system set dg_broker_config_file1 = '+DATAC1/PRIMEPRD/dr1primeprd.dat' scope=both sid='*';
alter system set dg_broker_config_file2 = '+DATAC1/PRIMEPRD/dr2primeprd.dat' scope=both sid='*';
alter system set dg_broker_start = true scope=both sid='*';

14.2. Set the below parameter in standby database.

alter system set dg_broker_config_file1 = '+DATAC1/PRIMEDR/dr1primedr.dat' scope=both sid='*';
alter system set dg_broker_config_file2 = '+DATAC1/PRIMEDR/dr2primedr.dat' scope=both sid='*';
alter system set dg_broker_start = true scope=both sid='*';

15.1. Configure the dataguard brocker using the below command from production database server

/home/oracle::11g- dgmgrl

connect sys/*****

create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

add database primedr as connect identifier is primedr maintained as physical;

Configuration "dg_primeprd" created with primary database "primeprd"

add database primedr as connect identifier is primedr maintained as physical;
Error: ORA-16698: member has a LOG_ARCHIVE_DEST_n parameter with SERVICE attribute set

Failed.

remove configuration;
Removed configuration

Set the log_archive_dest_2 settings from both the Primary and Standby databases to be nothing.

alter system set log_archive_dest_2='' scope=both sid='*';

Disable then Enable the broker parameter on both the Primary and Standby databases.

— Primary

alter system set dg_broker_start=false scope=both sid='*';
alter system set dg_broker_start=true scope=both sid='*';

— Standby

alter system set dg_broker_start=false scope=both sid='*';
alter system set dg_broker_start=true scope=both sid='*';

On the Primary database create the broker configuration for the Primary and Standby database and this time it should work fine with no issues since the log archive destination 2 setting is not set, this is the workaround/solution.

/home/oracle::11g-- dgmgrl

connect sys/*****

create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

add database primedr as connect identifier is primedr maintained as physical;

enable configuration;

Set back the log_archive_dest_2 on both production and dr

— In Production Database —

alter system set log_archive_dest_2='service=primedr LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primedr' scope=both sid='*';

— In DR Database —

alter system set log_archive_dest_2='service=primeprd LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primeprd' scope=both sid='*';

If in Broker you get below error

 show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd - Primary database
primedr - Physical standby database
Error: ORA-16664: unable to receive the result from a database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR (status updated 40 seconds ago)

Check in Broker Log File.

Mon Jul 04 14:41:46 2016
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191

Failed to send message to site . Error code is ORA-01017.

Then re-create password files in ASM in 12c like below, the correct format is pwd

Primary

srvctl config database -d primeprd
orapwd file='+DATAC1/PRIMEPRD/pwdprimeprd' entries=10 dbuniquename=primeprd password=****** force=y ignorecase=y

Standby

srvctl config database -d primedr
orapwd file='+DATAC1/PRIMEDR/pwdprimedr' entries=10 dbuniquename=primedr password=****** force=y ignorecase=y

Check configuration now and the error should have vanished and the status is success

show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd - Primary database
primedr - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 18 seconds ago)

— Try doing Switchover using DGMGRL Now —

 switchover to primedr
Performing switchover NOW, please wait...
Error: ORA-16644: apply instance not available

Failed.
Unable to switchover, primary database is still "primeprd"

edit database primedr set state='APPLY-ON' with apply instance='primedr1';

show database verbose primedr

Database - primedr

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 6.00 KByte/s
Active Apply Rate: 755.00 KByte/s
Maximum Apply Rate: 827.00 KByte/s
Real Time Query: OFF
Instance(s):
primedr1 (apply instance)
primedr2

switchover to primedr
Performing switchover NOW, please wait...
New primary database "primedr" is opening...
Oracle Clusterware is restarting database "primeprd" ...
Switchover succeeded, new primary is "primedr"

 show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr - Primary database
primeprd - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS (status updated 40 seconds ago)

edit database primeprd set state='APPLY-ON' with apply instance='primeprd1';

show database verbose primeprd

Database - primeprd

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 307.00 KByte/s
Active Apply Rate: 0 Byte/s
Maximum Apply Rate: 0 Byte/s
Real Time Query: OFF
Instance(s):
primeprd1 (apply instance)
primeprd2

DGMGRL

show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr - Primary database
primeprd - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 55 seconds ago)

Connect to DR site broker to intiate switchover

switchover to primeprd ;
Performing switchover NOW, please wait...
Operation requires a connection to instance "primeprd1" on database "primeprd"
Connecting to instance "primeprd1"...
Connected as SYSDBA.
New primary database "primeprd" is opening...
Oracle Clusterware is restarting database "primedr" ...
Switchover succeeded, new primary is "primeprd

show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd - Primary database
primedr - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 11 seconds ago