In my #OOW14 presentation about MAA and Multitenant, more precisely at slide #59, “PDB Creation from other PDB without ADG*”, I list a few commands that you can use to achieve a “correct” Pluggable Database clone in case you’re not using Active Data Guard.

What’s the problem with cloning a PDB in a MAA environment without ADG? If you’ve attended my session you should know the answer…

If you read the book “Data Guard Concepts and Administration 12c Release 1 (12.1)“, paragraph 3.5 Creating a PDB in a Primary Database, you’ll see that:

If you plan to create a PDB as a clone from a different PDB, then copy the data files that belong to the source PDB over to the standby database. (This step is not necessary in an Active Data Guard environment because the data files are copied automatically when the PDB is created on the standby database.)

But because there are good possibilities (99%?) that in a MAA environment you’re using ASM, this step is not so simple: you cannot copy the datafiles exactly where you want, it’s OMF, and the recovery process expects the files to be where the controlfile says they should be.

So, if you clone the PDB, the recovery process on the standby doesn’t find the datafiles at the correct location, thus the recovery process will stop and will not start until you fix manually. That’s why Oracle has implemented the new syntax “STANDBYS=NONE” that disables the recovery on the standby for a specific PDB: it lets you disable the recovery temporarily while the recovery process continues to apply logs on the remaining PDBs. (Note, however, that this feature is not intended as a generic solution for having PDBs not replicated. The recommended solution in this case is having two distinct CDBs, one protected by DG, the other not).

With ADG, when you clone the PDB on the primary, on the standby the ADG takes care of the following steps, no matter if on ASM or FS:

1. recover up to the point where the file# is registered in the controlfile
2. copy the datafiles from the source DB ON THE STANDBY DATABASE (so no copy over the network)
3. rename the datafile in the controlfile
4. continue with the recovery

If you don’t have ADG, and you’re on ASM, Oracle documentation says nothing with enough detail to let you solve the problem. So in August I’ve worked out the “easy” solution that I’ve also included in my slides (#59 and #60):

1. SQL> create pluggable database DEST from SRC standbys=none;
2. RMAN> backup as copy pluggable database DEST format ‘/tmp/dest%f.dbf';
3. $ scp  /tmp/dest*.dbf remote:/tmp
4. RMAN> catalog start with ‘/tmp/dest’
5. RMAN> set newnamefor pluggable database DEST to new;
6. RMAN> restore pluggable database DEST;
7. RMAN> switch pluggable database DEST to copy;
8. DGMGRL> edit database ‘STBY’ set state=’APPLY-OFF';
9. SQL> Alter pluggable database DEST enable recovery;
10. DGMGRL> edit database ‘STBY’ set state=’APPLY-ON';

Once at #OOW14, after endless conversations at the Demo Grounds, I’ve discovered that Oracle has worked out the very same solution requiring network transfer and that it has been documented in a new note.

Making Use of the STANDBYS=NONE Feature with Oracle Multitenant (Doc ID 1916648.1)

This note is very informative and I recommend to read it carefully!

What changes (better) in comparison with my first solution, is that Oracle suggests to use the new feature “restore from service”:

RMAN> run{
2> set newname for pluggable database DEST to new;
3> restore pluggable database DEST from service prim;
4> switch datafile all;
5> }

I’ve questioned the developers at the Demo Grounds about the necessity to use network transfer (I had the chance to speak directly with the developer who has written this piece of code!! :-)) and they said that they had worked out only this solution. So, if you have a huge PDB to clone, the network transfer from the primary to standby may impact severely your Data  Guard environment and/or your whole infrastructure, for the time of the transfer.

Of course, I have a complex, undocumented solution, I hope I will find the time to document it, so stay tuned if you’re curious!

Ok, if you’re reading this post, you may want to read also the previous one that explains something more about the problem.

Briefly said, if you have a CDB running on ASM in a MAA architecture and you do not have Active Data Guard, when you clone a PDB you have to “copy” the datafiles somehow on the standby. The only solution offered by Oracle (in a MOS Note, not in the documentation) is to restore the PDB from the primary to the standby site, thus transferring it over the network. But if you have a huge PDB this is a bad solution because it impacts your network connectivity. (Note: ending up with a huge PDB IMHO can only be caused by bad consolidation. I do not recommend to consolidate huge databases on Multitenant).

So I’ve worked out another solution, that still has many defects and is almost not viable, but it’s technically interesting because it permits to discover a little more about Multitenant and Data Guard.

The three options

At the primary site, the process is always the same: Oracle copies the datafiles of the source, and it modifies the headers so that they can be used by the new PDB (so it changes CON_ID, DBID, FILE#, and so on).

On the standby site, by opposite, it changes depending on the option you choose:

Option 1: Active Data Guard

If you have ADG, the ADG itself will take care of copying the datafile on the standby site, from the source standby pdb to the destination standby pdb. Once the copy is done, the MRP0 will continue the recovery. The modification of the header block of the destination PDB is done by the MRP0 immediately after the copy (at least this is what I understand).

Option 2: No Active Data Guard, but STANDBYS=none

In this case, the copy on the standby site doesn’t happen, and the recovery process just add the entry of the new datafiles in the controlfile, with status OFFLINE and name UNKNOWNxxx.  However, the source file cannot be copied anymore, because the MRP0 process will expect to have a copy of the destination datafile, not the source datafile. Also, any tentative of restore of the datafile 28 (in this example) will give an error because it does not belong to the destination PDB. So the only chance is to restore the destination PDB from the primary.

Option 3: No Active Data Guard, no STANDBYS=none

This is the case that I want to explain actually. Without the flag STANDBYS=none, the MRP0 process will expect to change the header of the new datafile, but because the file does not exist yet, the recovery process dies.
We can then copy it manually from the source standby pdb, and restart the recovery process, that will change the header. This process needs to be repeated for each datafile. (that’s why it’s not a viable solution, right now).

Let’s try it together:

The Environment

Primary

08:13:08 SYS@CDBATL_2> select db_unique_name, instance_name from v$database, gv$instance;

DB_UNIQUE_NAME                 INSTANCE_NAME
------------------------------ ----------------
CDBATL                         CDBATL_2
CDBATL                         CDBATL_1

Standby

07:35:56 SYS@CDBGVA_2> select db_unique_name, instance_name from v$database, gv$instance;

DB_UNIQUE_NAME                 INSTANCE_NAME
------------------------------ ----------------
CDBGVA                         CDBGVA_1
CDBGVA                         CDBGVA_2

The current user PDB (any resemblance to real people is purely coincidental  #haveUSeenMaaz):

08:14:31 SYS@CDBATL_2> select open_mode, name from gv$pdbs where name='MAAZ';

OPEN_MODE  NAME
---------- ------------------------------
OPEN       MAAZ
OPEN       MAAZ

Cloning the PDB on the primary

First, make sure that the source PDB is open read-only

08:45:54 SYS@CDBATL_2> alter pluggable database maaz close immediate instances=all;

Pluggable database altered.

08:46:20 SYS@CDBATL_2> alter pluggable database maaz open read only instances=all;

Pluggable database altered.

08:46:32 SYS@CDBATL_2> select open_mode, name from gv$pdbs where name='MAAZ' ;

OPEN_MODE  NAME
---------- ------------------------------
READ ONLY  MAAZ
READ ONLY  MAAZ

Then, clone the PDB on the primary without the clause STANDBYS=NONE:

08:46:41 SYS@CDBATL_2> create pluggable database LUDO from MAAZ;

Pluggable database created.

Review the clone on the Standby

At this point, on the standby the alert log show that the SYSTEM datafile is missing, and the recovery process stops.

Mon Dec 15 17:46:11 2014
Recovery created pluggable database LUDO
Mon Dec 15 17:46:11 2014
Errors in file /u01/app/oracle/diag/rdbms/cdbgva/CDBGVA_2/trace/CDBGVA_2_mrp0_16464.trc:
ORA-01565: error in identifying file '+DATA'
ORA-17503: ksfdopn:2 Failed to open file +DATA
ORA-15045: ASM file name '+DATA' is not in reference form
Recovery was unable to create the file as:
'+DATA'
MRP0: Background Media Recovery terminated with error 1274
Mon Dec 15 17:46:11 2014
Errors in file /u01/app/oracle/diag/rdbms/cdbgva/CDBGVA_2/trace/CDBGVA_2_mrp0_16464.trc:
ORA-01274: cannot add data file that was originally created as '+DATA/CDBATL/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/system.825.866396765'
Mon Dec 15 17:46:11 2014
Managed Standby Recovery not using Real Time Apply
Mon Dec 15 17:46:11 2014
Recovery interrupted!
Recovery stopped due to failure in applying recovery marker (opcode 17.34).
Datafiles are recovered to a consistent state at change 10433175 but controlfile could be ahead of datafiles.
Mon Dec 15 17:46:11 2014
Errors in file /u01/app/oracle/diag/rdbms/cdbgva/CDBGVA_2/trace/CDBGVA_2_mrp0_16464.trc:
ORA-01274: cannot add data file that was originally created as '+DATA/CDBATL/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/system.825.866396765'
Mon Dec 15 17:46:11 2014
MRP0: Background Media Recovery process shutdown (CDBGVA_2)

One remarkable thing, is that in the standby controlfile, ONLY THE SYSTEM DATAFILE exists:

18:02:50 SYS@CDBGVA_2> select con_id from v$pdbs where name='LUDO';

    CON_ID
----------
         4

18:03:10 SYS@CDBGVA_2> select name from v$datafile where con_id=4;

NAME
---------------------------------------------------------------------------
+DATA/CDBATL/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/system.825.866396765

We need to fix the datafiles one by one, but most of the steps can be done once for all the datafiles.

Copy the source PDB from the standby

What do we need to do? Well, the recovery process is stopped, so we can safely copy the datafiles of  the source PDB from the standby site because they have not moved yet. (meanwhile, we can put the primary source PDB back in read-write mode).

-- on primary
08:58:07 SYS@CDBATL_2> alter pluggable database maaz close immediate instances=all;

Pluggable database altered.

08:58:15 SYS@CDBATL_2> alter pluggable database maaz open read write instances=all;

Pluggable database altered.

Copy the datafiles:

## on the standby:
RMAN> backup as copy pluggable database MAAZ;

Starting backup at 15-DEC-14
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=58 instance=CDBGVA_2 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00029 name=+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/sysaux.463.857404625
output file name=+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/sysaux.863.866397043 tag=TAG20141215T175041 RECID=54 STAMP=866397046
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00028 name=+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.283.857404623
output file name=+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.864.866397049 tag=TAG20141215T175041 RECID=55 STAMP=866397051
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 15-DEC-14

Starting Control File and SPFILE Autobackup at 15-DEC-14
piece handle=+DATA/CDBGVA/AUTOBACKUP/2014_12_15/s_866396771.865.866397053 comment=NONE
Finished Control File and SPFILE Autobackup at 15-DEC-14

Do the magic

Now there’s the interesting part: we need to assign the datafile copies of the maaz PDB to LUDO.

Sadly, the OMF will create the copies on the bad location (it’s a copy, to they are created on the same location as the source PDB).

We cannot try to uncatalog and recatalog the copies, because they will ALWAYS be affected to the source PDB. Neither we can use RMAN because it will never associate the datafile copies to the new PDB. We need to rename the files manually.

RMAN> list datafilecopy all;

List of Datafile Copies
=======================

Key File S Completion Time Ckp SCN Ckp Time
------- ---- - --------------- ---------- ---------------
55 28 A 15-DEC-14 10295232 14-DEC-14
 Name: +DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.864.86639709
 Tag: TAG20141215T175041

54 29 A 15-DEC-14 10295232 14-DEC-14
 Name: +DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/sysaux.863.86639703
 Tag: TAG20141215T175041


RMAN> select name, guid from v$pdbs;

NAME       GUID
---------- --------------------------------
PDB$SEED   FFBCECBB503D606BE043334EA8C019B7
MAAZ       0243BF7B39D4440AE053334EA8C0E471
LUDO       0A4A0048D5321597E053334EA8C0E40A

It’s better to uncatalog the datafile copies before, so we keep the catalog clean:

RMAN> change datafilecopy '+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.864.866397049' uncatalog;

uncataloged datafile copy
datafile copy file name=+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.864.866397049 RECID=55 STAMP=866397051
Uncataloged 1 objects


RMAN> change datafilecopy '+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/sysaux.863.866397043' uncatalog;

uncataloged datafile copy
datafile copy file name=+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/sysaux.863.866397043 RECID=54 STAMP=866397046
Uncataloged 1 objects

Then, because we cannot rename files on a standby database with standby file management set to AUTO, we need to put it temporarily to MANUAL.

10:24:21 SYS@CDBGVA_2> alter database rename file '+DATA/CDBATL/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/system.825.866396765' to '+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.864.866397049';
alter database rename file '+DATA/CDBATL/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/system.825.866396765' to '+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.864.866397049'
*
ERROR at line 1:
ORA-01275: Operation RENAME is not allowed if standby file management is automatic.

10:27:49 SYS@CDBGVA_2> select name, ispdb_modifiable from v$parameter where name like 'standby%';

NAME                                                         ISPDB
------------------------------------------------------------ -----
standby_archive_dest                                         FALSE
standby_file_management                                      FALSE

standby_file_management is not PDB modifiable, so we need to do it for the whole CDB.

10:31:42 SYS@CDBGVA_2> alter system set standby_file_management=manual;

System altered.

18:05:04 SYS@CDBGVA_2> alter database rename file '+DATA/CDBATL/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/system.825.866396765' to '+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.864.866397049';

Database altered.

then we need to set back the standby_file_management=auto or the recover will not start:

10:34:24 SYS@CDBGVA_2> alter system set standby_file_management=auto;
System altered.

We can now restart the recovery.

The recovery process will:
– change the new datafile by modifying the header for the new PDB
– create the entry for the second datafile in the controlfile
– crash again because the datafile is missing

18:11:30 SYS@CDBGVA_2> alter database recover managed standby database;
alter database recover managed standby database
*
ERROR at line 1:
ORA-00283: recovery session canceled due to errors
ORA-01111: name for data file 61 is unknown - rename to correct file
ORA-01110: data file 61: '/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/UNNAMED00061'
ORA-01157: cannot identify/lock data file 61 - see DBWR trace file
ORA-01111: name for data file 61 is unknown - rename to correct file
ORA-01110: data file 61: '/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/UNNAMED00061'


18:11:33 SYS@CDBGVA_2> select name from v$datafile where con_id=4;

NAME
---------------------------------------------------------------------------
+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.864.866397049
/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/UNNAMED00061

We already have the SYSAUX datafile, right? So we can alter the name again:

18:14:21 SYS@CDBGVA_2> alter system set standby_file_management=manual;

System altered.

18:14:29 SYS@CDBGVA_2> alter database rename file '/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/UNNAMED00061' to '+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/sysaux.863.866397043';

Database altered.

18:14:31 SYS@CDBGVA_2> alter system set standby_file_management=auto;

System altered.

18:14:35 SYS@CDBGVA_2> alter database recover managed standby database;

This time all the datafiles have been copied (no user datafile for this example) and the recovery process will continue!! so we can hit ^C and start it in background.

18:14:35 SYS@CDBGVA_2> alter database recover managed standby database;
alter database recover managed standby database
*
ERROR at line 1:
ORA-16043: Redo apply has been canceled.
ORA-01013: user requested cancel of current operation

 

18:18:10 SYS@CDBGVA_2> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

Database altered.

18:18:19 SYS@CDBGVA_2>

The Data Guard configuration reflects the success of this operation.

Do we miss anything?

Of course, we do!! The datafile names of the new PDB reside in the wrong ASM path. We need to fix them!

18:23:07 SYS@CDBGVA_2> alter database recover managed standby database cancel;

Database altered.

RMAN> backup as copy pluggable database ludo;

Starting backup at 15-DEC-14
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=60 instance=CDBGVA_2 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00061 name=+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/sysaux.863.866397043
output file name=+DATA/CDBGVA/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/sysaux.866.866398933 tag=TAG20141215T182213 RECID=56 STAMP=866398937
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00060 name=+DATA/CDBGVA/0243BF7B39D4440AE053334EA8C0E471/DATAFILE/system.864.866397049
output file name=+DATA/CDBGVA/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/system.867.866398941 tag=TAG20141215T182213 RECID=57 STAMP=866398943
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 15-DEC-14

Starting Control File and SPFILE Autobackup at 15-DEC-14
piece handle=+DATA/CDBGVA/AUTOBACKUP/2014_12_15/s_866398689.868.866398945 comment=NONE
Finished Control File and SPFILE Autobackup at 15-DEC-14

RMAN> switch pluggable database ludo to copy;

using target database control file instead of recovery catalog
datafile 60 switched to datafile copy "+DATA/CDBGVA/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/system.867.866398941"
datafile 61 switched to datafile copy "+DATA/CDBGVA/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/sysaux.866.866398933"

18:23:54 SYS@CDBGVA_2> select name from v$datafile where con_id=4;

NAME
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+DATA/CDBGVA/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/system.867.866398941
+DATA/CDBGVA/0A4A0048D5321597E053334EA8C0E40A/DATAFILE/sysaux.866.866398933

I know there’s no practical use of this procedure, but it helps a lot in understanding how Multitenant has been implemented.

I expect some improvements in 12.2!!

Cheers

—

Ludo

After almost 1 year, I’ve decided to publish these articles on my Slideshare account. You may have already seen them in the IOUG Collaborate 14 conference content or in the SOUG Newsletter 2014/4. Nothing really new, but I hope you’ll still enjoy them.

Cheers

—

Ludo

I don’t like to publish small code snippets, but I’ve just rewritten one of my most used SQL scripts for SQL Server that gets the details about the last backup for every database (for both backup database and backup log).

It now makes use of with () and rank() over() to make it much easier to read and modify.

So I think it’s worth to share it.

with
db as (
	select [Instance] = @@SERVERNAME,
		[Database]  = name,
	    [RecoveryMode]   = DATABASEPROPERTYEX(name, 'Recovery'),
	    [CreationTime]   = crdate,
	    [Status]         = DATABASEPROPERTYEX(name, 'Status')
		from master..sysdatabases
		where name!='tempdb'
),
lastfull as	(
	select * from (
	 select [Database]     = s.database_name,
		--[Type]   = s.type,
		[LastFullDate] = convert(varchar, s.backup_finish_date, 120),
		[LastFullSize]   = s.backup_size,
		[LastFullDevice] = f.physical_device_name,
        [LastFullDevTyp] = f.device_type,
		[Nrank] = rank() over (partition by s.database_name order by s.backup_finish_date desc)
	from msdb.dbo.backupset s, msdb.dbo.backupmediafamily f
		where 
         s.media_set_id=f.media_set_id
		and s.type='D'
		-- and f.device_type = 7 -- only backup devices
	) f
	where nrank=1
),
lastlog as (
	select * from (
	 select [Database]     = s.database_name,
		--[Type]   = s.type,
		[LastLogDate] = convert(varchar, s.backup_finish_date, 120),
		[LastLogSize]   = s.backup_size,
		[LastLogDevice] = f.physical_device_name,
        [LastLogDevTyp] = f.device_type,
		[Nrank] = rank() over (partition by s.database_name order by s.backup_finish_date desc)
	 from msdb.dbo.backupset s, msdb.dbo.backupmediafamily f
		where 
         s.media_set_id=f.media_set_id
		and s.type='L'
		-- and f.device_type = 7 -- only backup devices
	) l
	where nrank=1
)
select db.[Instance],db.[Database], db.[RecoveryMode], db.[CreationTime], db.[Status],
	lastfull.[LastFullDate], lastfull.[LastFullSize], 
        lastfull.[LastFullDevice], lastfull.[LastFullDevTyp],
	lastlog.[LastLogDate], lastlog.[LastLogSize], lastlog.[LastLogDevice], lastlog.[LastLogDevTyp]
from db
	left outer join lastfull
		on (db.[Database]=lastfull.[Database])
	left outer join lastlog
		on (db.[Database]=lastlog.[Database])

As you can see, modify it to include for example incremental backups should be very easy.

Cheers
—
Ludo

Very recently I had to configure a customer’s RAC private interconnect from bonding to HAIP to get benefit of both NICs.

So I would like to recap here what the hypothetic steps would be if you need to do the same.

In this example I’ll switch from a single-NIC interconnect (eth1) rather than from a bond configuration, so if you are familiar with the RAC Attack! environment you can try to put everything in place on your own.

First, you need to plan the new network configuration in advance, keeping in mind that there are a couple of important restrictions:

1. Your interconnect interface naming must be uniform on all nodes in the cluster. The interconnect uses the interface name in its configuration and it doesn’t support different names on different hosts
2. You must bind the different private interconnect interfaces in different subnets (see Note: 1481481.1 – 11gR2 CSS Terminates/Node Eviction After Unplugging one Network Cable in Redundant Interconnect Environment if you need an explanation)

Implementation 

The RAC Attack book uses one interface per node for the interconnect (eth1, using network 172.16.100.0)

To make things a little more complex, we’ll not use the eth1 in the new HAIP configuration, so we’ll test also the deletion of the old interface.

What you need to do is add two new interfaces (host only in your virtualbox) and configure them as eth2 and eth3, e.g. in networks 172.16.101.0 and 172.16.102.0)

eth2      Link encap:Ethernet  HWaddr 08:00:27:32:76:DD
          inet addr:172.16.101.51  Bcast:172.16.101.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe32:76dd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:29 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2044 (1.9 KiB)  TX bytes:1714 (1.6 KiB)

eth3      Link encap:Ethernet  HWaddr 08:00:27:2E:05:4B
          inet addr:172.16.102.61  Bcast:172.16.102.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe2e:54b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1140 (1.1 KiB)  TX bytes:720 (720.0 b)

modify /var/named/racattack in order to use the new addresses (RAC doesn’t care about logical names, it’s just for our convenience):

collabn1 A 192.168.78.51
collabn1-vip A 192.168.78.61
collabn1-priv A 172.16.100.51
collabn1-priv1 A 172.16.101.51
collabn1-priv2 A 172.16.102.61
collabn2 A 192.168.78.52
collabn2-vip A 192.168.78.62
collabn2-priv A 172.16.100.52
collabn2-priv1 A 172.16.101.52
collabn2-priv2 A 172.16.102.62

add also the reverse lookup in  in-addr.arpa:

51.101.16.172 PTR collabn1-priv1.racattack.
52.102.16.172 PTR collabn1-priv2.racattack.
61.101.16.172 PTR collabn2-priv1.racattack.
62.102.16.172 PTR collabn2-priv2.racattack.

restart  named on the first node and check that both nodes can ping all the names correctly:

[root@collabn1 named]# ping collabn2-priv1
PING collabn2-priv1.racattack (172.16.101.52) 56(84) bytes of data.
64 bytes from 172.16.101.52: icmp_seq=1 ttl=64 time=1.27 ms
64 bytes from 172.16.101.52: icmp_seq=2 ttl=64 time=0.396 ms
^C
--- collabn2-priv1.racattack ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1293ms
rtt min/avg/max/mdev = 0.396/0.835/1.275/0.440 ms
[root@collabn1 named]# ping collabn2-priv2
PING collabn2-priv2.racattack (172.16.102.62) 56(84) bytes of data.
64 bytes from 172.16.102.62: icmp_seq=1 ttl=64 time=0.924 ms
64 bytes from 172.16.102.62: icmp_seq=2 ttl=64 time=0.251 ms
^C
--- collabn2-priv2.racattack ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1480ms
rtt min/avg/max/mdev = 0.251/0.587/0.924/0.337 ms
[root@collabn1 named]# ping collabn1-priv2
PING collabn1-priv2.racattack (172.16.102.61) 56(84) bytes of data.
64 bytes from 172.16.102.61: icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from 172.16.102.61: icmp_seq=2 ttl=64 time=0.032 ms
^C
--- collabn1-priv2.racattack ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1240ms
rtt min/avg/max/mdev = 0.019/0.025/0.032/0.008 ms
[root@collabn1 named]# ping collabn1-priv1
PING collabn1-priv1.racattack (172.16.101.51) 56(84) bytes of data.
64 bytes from 172.16.101.51: icmp_seq=1 ttl=64 time=0.017 ms
64 bytes from 172.16.101.51: icmp_seq=2 ttl=64 time=0.060 ms
^C
--- collabn1-priv1.racattack ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1224ms
rtt min/avg/max/mdev = 0.017/0.038/0.060/0.022 ms

check the nodes that compose the cluster:

[root@collabn1 network-scripts]# olsnodes -s
collabn1 Active
collabn2 Active

on all nodes, make a copy of the gpnp profile.xml (just in case, the oifcfg tool does the copy automatically)

$ cd $GRID_HOME/gpnp/`hostname`/profiles/peer/
$ cp -p profile.xml profile.xml.bk

List the available networks:

[root@collabn1 bin]# ./oifcfg iflist -p -n
eth0 192.168.78.0 PRIVATE 255.255.255.0
eth1 172.16.100.0 PRIVATE 255.255.255.0
eth1 169.254.0.0 UNKNOWN 255.255.0.0
eth2 172.16.101.0 PRIVATE 255.255.255.0
eth3 172.16.102.0 PRIVATE 255.255.255.0

Get the current ip configuration for the interconnect:

[root@collabn1 bin]# ./oifcfg getif
eth0 192.168.78.0 global public
eth1 172.16.100.0 global cluster_interconnect

one one node only, set the new interconnect interfaces:

[root@collabn1 network-scripts]# oifcfg setif -global eth2/172.16.101.0:cluster_interconnect
[root@collabn1 network-scripts]# oifcfg setif -global eth3/172.16.102.0:cluster_interconnect
[root@collabn1 network-scripts]# oifcfg getif
eth0 192.168.78.0 global public
eth1 172.16.100.0 global cluster_interconnect
eth2 172.16.101.0 global cluster_interconnect
eth3 172.16.102.0 global cluster_interconnect

check that the other nodes has received the new configuration:

[root@collabn2 bin]# ./oifcfg getif
eth0 192.168.78.0 global public
eth1 172.16.100.0 global cluster_interconnect
eth2 172.16.101.0 global cluster_interconnect
eth3 172.16.102.0 global cluster_interconnect

Before deleting the old interface, it would be sensible to stop your cluster resources (in some cases, one of the nodes may be evicted), in any case the cluster must be restarted completely in order to get the new interfaces working.

Note: having three interfaces in a HAIP interconnect is perfectly working, HAIP works from 2 to 4 interfaces. I’m showing how to delete eth1 just for information!!

[root@collabn1 network-scripts]# oifcfg delif -global eth1/172.16.100.0
[root@collabn1 network-scripts]# oifcfg getif
eth0 192.168.78.0 global public
eth2 172.16.101.0 global cluster_interconnect
eth3 172.16.102.0 global cluster_interconnect

on all nodes, shutdown the CRS:

[root@collabn1 network-scripts]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'collabn1'
...

Now you can disable the old interface:

[root@collabn1 network-scripts]# ifdown eth1

and modify the parameter ONBOOT=no inside the configuration script of eth1 interface.

Start the cluster again:

[root@collabn1 network-scripts]# crsctl start crs

And check that the resources are up & running:

# crscst stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE collabn1
ONLINE ONLINE collabn2
ora.LISTENER.lsnr
ONLINE ONLINE collabn1
ONLINE ONLINE collabn2
ora.asm
ONLINE ONLINE collabn1 Started
ONLINE ONLINE collabn2 Started
ora.gsd
OFFLINE OFFLINE collabn1
OFFLINE OFFLINE collabn2
ora.net1.network
ONLINE ONLINE collabn1
ONLINE ONLINE collabn2
ora.ons
ONLINE ONLINE collabn1
ONLINE ONLINE collabn2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE collabn2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE collabn1
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE collabn1
ora.collabn1.vip
1 ONLINE ONLINE collabn1
ora.collabn2.vip
1 ONLINE ONLINE collabn2
ora.cvu
1 ONLINE ONLINE collabn1
ora.oc4j
1 ONLINE ONLINE collabn1
ora.orcl.db
1 ONLINE ONLINE collabn1 Open
2 ONLINE ONLINE collabn2 Open
ora.scan1.vip
1 ONLINE ONLINE collabn2
ora.scan2.vip
1 ONLINE ONLINE collabn1
ora.scan3.vip
1 ONLINE ONLINE collabn1

 Testing the high availability

Disconnect cable from one of the two interfaces (virtually if you’re in virtualbox )

Pay attention at the NO-CARRIER status (in eth2 in this example):

# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:07:33:94 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 08:00:27:7f:b4:88 brd ff:ff:ff:ff:ff:ff
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 08:00:27:51:1d:78 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:39:86:f2 brd ff:ff:ff:ff:ff:ff

check that the CRS is still up & running:

# crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE collabn1
ONLINE ONLINE collabn2
ora.LISTENER.lsnr
ONLINE ONLINE collabn1
ONLINE ONLINE collabn2
ora.asm
ONLINE ONLINE collabn1 Started
ONLINE ONLINE collabn2 Started
ora.gsd
OFFLINE OFFLINE collabn1
OFFLINE OFFLINE collabn2
ora.net1.network
ONLINE ONLINE collabn1
ONLINE ONLINE collabn2
ora.ons
ONLINE ONLINE collabn1
ONLINE ONLINE collabn2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE collabn2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE collabn1
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE collabn1
ora.collabn1.vip
1 ONLINE ONLINE collabn1
ora.collabn2.vip
1 ONLINE ONLINE collabn2
ora.cvu
1 ONLINE ONLINE collabn1
ora.oc4j
1 ONLINE ONLINE collabn1
ora.orcl.db
1 ONLINE ONLINE collabn1 Open
2 ONLINE ONLINE collabn2 Open
ora.scan1.vip
1 ONLINE ONLINE collabn2
ora.scan2.vip
1 ONLINE ONLINE collabn1
ora.scan3.vip
1 ONLINE ONLINE collabn1

The virtual interface eth2:1 as failed over on the second interface as eth3:2

eth3:1    Link encap:Ethernet  HWaddr 08:00:27:39:86:F2
          inet addr:169.254.185.134  Bcast:169.254.255.255  Mask:255.255.128.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth3:2    Link encap:Ethernet  HWaddr 08:00:27:39:86:F2
          inet addr:169.254.104.52  Bcast:169.254.127.255  Mask:255.255.128.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

After the cable is reconnected, the virtual interface is back on eth2:

eth2:1 Link encap:Ethernet HWaddr 08:00:27:51:1D:78
inet addr:169.254.104.52 Bcast:169.254.127.255 Mask:255.255.128.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

Further information

For this post I’ve used a RAC version 11.2, but RAC 12c use the very same procedure.

You can discover more here about HAIP:

http://docs.oracle.com/cd/E11882_01/server.112/e10803/config_cw.htm#HABPT5279 

And here about how to set it (beside this post!):

https://docs.oracle.com/cd/E11882_01/rac.112/e41959/admin.htm#CWADD90980

https://docs.oracle.com/cd/E11882_01/rac.112/e41959/oifcfg.htm#BCGGEFEI

Cheers

—

Ludo

In a little more than a couple of weeks, the great Collaborate conference will start again.

My agenda will be quite packed again, as speaker, panelist and workshop organizer:

RAC Attack! 12c

This technical workshop and networking event (never forget it’s a project created several years ago thanks to an intuition of Jeremy Schneider), confirms to be one of the best, long-living projects in the Oracle Community. It certainly boosted my Community involvement up to becoming an Oracle ACE. This year I’m the coordinator of the organization of the workshop, it’s a double satisfaction and it will certainly be a lot of fun again. Did I said that it’s already full booked? I’ve already blogged about it (and about what the lucky participants will get) here.

Oracle RAC, Data Guard, and Pluggable Databases: When MAA Meets Oracle Multitenant 

One of my favorite presentations, I’ve presented it already at OOW14 and UKOUG Tech14, but it’s still a very new topic for most people, even the most experienced DBAs. You’ll learn how Multitenant, RAC and Data Guard work together. Expect colorful architecture schemas and a live demo!  You can read more about it in this post.

Panel: Nothing to BLOG About – Think Again

My friend Michael Abbey (Pythian) invited me to participate in his panel about blogging. It’s my first time as panelist, so I’m very excited!

IOUG RAC SIG Meeting

Missing this great networking event is not an option! I’m organizing this session as RAC SIG board member (Thanks to the IOUG for this opportunity!). We’ll focus on Real Application Clusters role in the private cloud and infrastructure optimization. We’ll have many special guests, including Oracle RAC PM Markus Michalewicz, Oracle QoS PM Mark Scardina and Oracle ASM PM James Williams.

Can you ever miss it???

A good Trivadis representative!!

This year I’m not going to Las Vegas alone. My Trivadis colleague Markus Flechtner , one of the most expert RAC technologists I have the chance to know, will also come and present a session about RAC diagnostics:

615: RAC Clinics- Starring Dr. ORACHK, Dr CHM and Dr. TFA

Mon. April 13| 9:15 AM – 10:15 AM | Room Palm D

If you speak German you can follow his nice blog: http://oracle.markusflechtner.de/

Looking forward to meet you there

—

Ludovico

Did I say in some of my previous posts that I love RAC Attack? I love it more during Collaborate conference, because doing it as a pre-conference workshop is just the right way the get people involved and go straight to the goal: learning while having fun together.

We had way less people than expected but it still has been a great success!

The t-shirts have been great for coloring the room: as soon as people finished the installation of the first Linux VM, they’ve got one t-shirt.

Look at the room at the beginning of the workshop:

after a few hours, the room looked better! New ninjas, red stack, happy participants

We had a very special guest today. Mr. RAC PM in person has tried and validated our installation instructions 

We got pizza again, but because of restrictions at the convention center, it has been a beer-free afternoon

Thank you anyway to the OTN for sponsoring almost everything!!

Looking forward to organize the next RAC Attack, Thank you guys!!

—

Ludo

Cloud. What a wonderful word. Wonderful and gray.
If you are involved in the Oracle Community, blogs and conferences, you certainly care about it and have perhaps your own conception of it or ideas about how to implement it.

My Collaborate 2015 RAC SIG experience

During the last Collaborate Conference, I’ve “tried” to animate the traditional RAC SIG Round-Table  with this topic:

In the last few years, cloud computing and infrastructure optimization have been the leading topics that guided the IT innovation. What’s the role of Oracle RAC in this context?

During this meeting leading RAC specialists, product managers, RAC SIG representatives and RAC Attack Ninjas will come together and discuss with you about the new Oracle RAC 12c features for the private cloud and the manageability of RAC environments.

Join us for the great discussion. This is your chance to have a great networking session!

Because it’s the RAC SIG meeting, most of the participants DO HAVE a RAC environment to manage, and are looking for best practices and ideas to improve it, or maybe they want to share their experiences.

I’ve started the session by asking how many people are currently operating a private cloud and how many would like to implement it.

With my biggest surprise (so big that I felt immediately uncomfortable), except one single person, nobody raised the hand.

What?

I’ve spent a very bad minute, I was almost speechless. I was actually asking myself: “is my conception of private cloud wrong?”. Then my good friend Yury came in help and we started the discussion about the RAC features that enable private cloud capabilities. During those 30 minutes, almost no users intervened. Then Oracle Product Managers (RAC, ASM, QoS, Cloud) started explaining their point of view, and I suddenly realized that

when talking about Private Cloud, there is a huge gap between the Oracle Private Cloud implementation best practices and the small customers skills and budgets.

When Oracle product managers talk about Private Cloud, they target big companies and advice to plan the infrastructure using:

• Exadata
• Full-pack of options for a total of 131k per CPU:
  • Enterprise Edition (47.5k)
  • Multitenant (17.5k)
  • Real Application Clusters (23k)
  • Partitioning (11.5k)
  • Diagnostic Pack (7.5k)
  • Tuning Pack (5k)
  • Lifecycle Management Pack (12k)
  • Cloud Management Pack (7.5k)
• Flex Cluster
• Policy Managed Databases
• Quality of Services Management
• Rapid Home provisioning
• Enterprise Manager and DBaaS Self Service portal

The CapEx needed for such a stack is definitely a show stopper for most small-medium companies. And it’s not only about the cost. When I gave my presentation about Policy Managed Databases at Collaborate in 2014, and later about Multitenant and MAA at Open World, it was clear that “almost” nobody (let’s say less than 5%, just to give an idea) uses these new technologies. Many of them are new and, in some cases, not stable. Notably, Multitenant and QoS are not working together as of now. Qos will work with the new resource manager at PDB level only in release 12.2 (and still not guaranteed).

For the average company (or the average DBA), there is more than enough to be scared about, so private cloud is not seen as easy to implement.

So there’s no private cloud solution for SMBs?

It really depends on what you want to achieve, and at which level.

Based on my experience at Trivadis, I can say that you can achieve Private Cloud for less. Much less.

What a Private Cloud should guarantee? According to its NIST definition, five four things:

1. On-demand self-service.
2. Broad network access.
3. Resource pooling.
4. Rapid elasticity.
5. Measured service.

Number 5 is a clear field of EM, and AWR Warehouse new feature may be of great help, for free  (but still, you can do a lot on your own with Statspack and some scripting if you are crazy enough to do it without Diagnostic pack).

Numbers 3 and 4 are a peculiarity of RAC, and they are included in the EE+RAC license. By leveraging OVM, there are very good opportunities of savings if the initial sizing of the solution is a problem. With OVM you can start as small as you want.

Number 1 depends on standards and automation already in place at your company. Generally speaking, nowadays scripting automatic provisioning with DBCA and APEX is very simple. If you’re not comfortable with coding, tools like the Trivadis Toolbox make this task easier. Moreover, nobody said that the self-service provisioning must be done through a web interface by the final user. It might be (and usually is) triggered by an event, like the creation of a service request, so you can keep web development outside of your cloud.

Putting all together

You can create a basic Private Cloud that fits perfectly your needs without spending or changing too much in your RAC environment.

Automation doesn’t mean cost, you can do it on your own and keep it simple. If you need an advice, ideas or some help, just drop me an email (myfirstname.mylastname@trivadis.com), it would be great to discuss about your need for private cloud!

Things can be less complex than what we often think. Our imagination is the new limit!

—

Ludovico

The new Oracle 12c optimizer adaptive features are just great and work well out of the box in most cases.

Recently, however,  I’ve experienced my very first problem with SQL Plan Directives migrating a database to 12c, so I would like to share it.

Disclaimer 1: this is a specific problem that I found on ONE system. My solution may not fit with your environment, don’t use it if you are not sure about what you’re doing!

Disclaimer 2: despite I had this problem with a single SPD, I like adaptive features and I encourage to use them!!

Problem: a query takes a sub-second in 11gR2, in 12c it takes 12 seconds or more.

--11gR2
SQL> select * from APPUSER.V_TAB_PROP where TAB_ID = 842300;

...

48 rows selected.

Elapsed: 00:00:00.71

 
--12c
SQL> select * from APPUSER.V_TAB_PROP where TAB_ID = 842300;

...

48 rows selected.

Elapsed: 00:00:12.74

V_TAB_PROP is a very simple view. It just selects a central table “TAB” and then takes different properties by joining  a property table “TAB_PROP”.

To do that, it does 11 joins on the same property table.

create view ... as
select ...
from TAB li
left join
(select v.TAB_PROP_ID, v.PROP_VAL as c89 from  TAB_PROP v where v.PROP_ID = 89) v89 on li.TAB_PROP_ID = v89.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c88 from  TAB_PROP v where v.PROP_ID = 88) v88 on li.TAB_PROP_ID = v88.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c90 from  TAB_PROP v where v.PROP_ID = 90) v90 on li.TAB_PROP_ID = v90.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c82 from  TAB_PROP v where v.PROP_ID = 82) v82 on li.TAB_PROP_ID = v82.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c84 from  TAB_PROP v where v.PROP_ID = 84) v84 on li.TAB_PROP_ID = v84.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c93 from  TAB_PROP v where v.PROP_ID = 93) v93 on li.TAB_PROP_ID = v93.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c79 from  TAB_PROP v where v.PROP_ID = 79) v79 on li.TAB_PROP_ID = v79.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c81 from  TAB_PROP v where v.PROP_ID = 81) v81 on li.TAB_PROP_ID = v81.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c96 from  TAB_PROP v where v.PROP_ID = 96) v96 on li.TAB_PROP_ID = v96.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c95 from  TAB_PROP v where v.PROP_ID = 95) v95 on li.TAB_PROP_ID = v95.TAB_PROP_ID
left join                          
(select v.TAB_PROP_ID, v.PROP_VAL as c94 from  TAB_PROP v where v.PROP_ID = 94) v94 on li.TAB_PROP_ID = v94.TAB_PROP_ID
);

On the property table, TAB_PROP_ID and PROP_ID are unique (they compose the pk), so nested loops and index unique scans are the best way to get this data.
The table is 1500Mb big and the index 1000Mb.

This was the plan in 11g:

----------------------------------------------------------------------------------------------------------
| Id  | Operation                              | Name            | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                       |                 |       |       |  3401 (100)|          |
|   1 |  NESTED LOOPS OUTER                    |                 |  1009 |   218K|  3401   (1)| 00:00:41 |
|   2 |   NESTED LOOPS OUTER                   |                 |   615 |   123K|  2171   (1)| 00:00:27 |
|   3 |    NESTED LOOPS OUTER                  |                 |   390 | 73320 |  1391   (1)| 00:00:17 |
|   4 |     NESTED LOOPS OUTER                 |                 |   248 | 42408 |   894   (0)| 00:00:11 |
|   5 |      NESTED LOOPS OUTER                |                 |   160 | 24640 |   574   (0)| 00:00:07 |
|   6 |       NESTED LOOPS OUTER               |                 |   104 | 14248 |   366   (0)| 00:00:05 |
|   7 |        NESTED LOOPS OUTER              |                 |    68 |  8160 |   230   (0)| 00:00:03 |
|   8 |         NESTED LOOPS OUTER             |                 |    44 |  4532 |   142   (0)| 00:00:02 |
|   9 |          NESTED LOOPS OUTER            |                 |    29 |  2494 |    84   (0)| 00:00:02 |
|  10 |           NESTED LOOPS OUTER           |                 |    19 |  1311 |    46   (0)| 00:00:01 |
|  11 |            NESTED LOOPS OUTER          |                 |    13 |   676 |    20   (0)| 00:00:01 |
|  12 |             TABLE ACCESS BY INDEX ROWID| TAB             |     8 |   280 |     4   (0)| 00:00:01 |
|* 13 |              INDEX RANGE SCAN          | FK_TAB_PROP     |     8 |       |     3   (0)| 00:00:01 |
|  14 |             TABLE ACCESS BY INDEX ROWID| TAB_PROP        |     1 |    17 |     2   (0)| 00:00:01 |
|* 15 |              INDEX UNIQUE SCAN         | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  16 |            TABLE ACCESS BY INDEX ROWID | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 17 |             INDEX UNIQUE SCAN          | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  18 |           TABLE ACCESS BY INDEX ROWID  | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 19 |            INDEX UNIQUE SCAN           | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  20 |          TABLE ACCESS BY INDEX ROWID   | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 21 |           INDEX UNIQUE SCAN            | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  22 |         TABLE ACCESS BY INDEX ROWID    | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 23 |          INDEX UNIQUE SCAN             | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  24 |        TABLE ACCESS BY INDEX ROWID     | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 25 |         INDEX UNIQUE SCAN              | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  26 |       TABLE ACCESS BY INDEX ROWID      | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 27 |        INDEX UNIQUE SCAN               | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  28 |      TABLE ACCESS BY INDEX ROWID       | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 29 |       INDEX UNIQUE SCAN                | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  30 |     TABLE ACCESS BY INDEX ROWID        | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 31 |      INDEX UNIQUE SCAN                 | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  32 |    TABLE ACCESS BY INDEX ROWID         | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 33 |     INDEX UNIQUE SCAN                  | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
|  34 |   TABLE ACCESS BY INDEX ROWID          | TAB_PROP        |     2 |    34 |     2   (0)| 00:00:01 |
|* 35 |    INDEX UNIQUE SCAN                   | PK_TAB_PROP     |     1 |       |     1   (0)| 00:00:01 |
----------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
  13 - access("LI"."TAB_ID"=842300)
  15 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=94)
  17 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=93)
  19 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=79)
  21 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=96)
  23 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=84)
  25 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=95)
  27 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=82)
  29 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=81)
  31 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=88)
  33 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=89)
  35 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=90)

In 12c, the plan switches to adaptive, and half of the joins are converted to hash joins / full table scans:

---------------------------------------------------------------------------------------------------------------------
|   Id  | Operation                                                    | Name            | Starts | E-Rows | A-Rows |
---------------------------------------------------------------------------------------------------------------------
|     0 | SELECT STATEMENT                                             |                 |      1 |        |     48 |
|  *  1 |  HASH JOIN RIGHT OUTER                                       |                 |      1 |    829K|     48 |
|  *  2 |   TABLE ACCESS FULL                                          | TAB_PROP        |      1 |   2486K|   2486K|
|  *  3 |   HASH JOIN OUTER                                            |                 |      1 |    539K|     48 |
|-    4 |    NESTED LOOPS OUTER                                        |                 |      1 |    539K|     48 |
|-    5 |     STATISTICS COLLECTOR                                     |                 |      1 |        |     48 |
|  *  6 |      HASH JOIN OUTER                                         |                 |      1 |    350K|     48 |
|-    7 |       NESTED LOOPS OUTER                                     |                 |      1 |    350K|     48 |
|-    8 |        STATISTICS COLLECTOR                                  |                 |      1 |        |     48 |
|  *  9 |         HASH JOIN OUTER                                      |                 |      1 |    228K|     48 |
|-   10 |          NESTED LOOPS OUTER                                  |                 |      1 |    228K|     48 |
|-   11 |           STATISTICS COLLECTOR                               |                 |      1 |        |     48 |
|  * 12 |            HASH JOIN OUTER                                   |                 |      1 |    148K|     48 |
|-   13 |             NESTED LOOPS OUTER                               |                 |      1 |    148K|     48 |
|-   14 |              STATISTICS COLLECTOR                            |                 |      1 |        |     48 |
|  * 15 |               HASH JOIN OUTER                                |                 |      1 |  96510 |     48 |
|-   16 |                NESTED LOOPS OUTER                            |                 |      1 |  96510 |     48 |
|-   17 |                 STATISTICS COLLECTOR                         |                 |      1 |        |     48 |
|  * 18 |                  HASH JOIN OUTER                             |                 |      1 |  62771 |     48 |
|-   19 |                   NESTED LOOPS OUTER                         |                 |      1 |  62771 |     48 |
|-   20 |                    STATISTICS COLLECTOR                      |                 |      1 |        |     48 |
|- * 21 |                     HASH JOIN OUTER                          |                 |      1 |  40827 |     48 |
|    22 |                      NESTED LOOPS OUTER                      |                 |      1 |  40827 |     48 |
|-   23 |                       STATISTICS COLLECTOR                   |                 |      1 |        |     48 |
|- * 24 |                        HASH JOIN OUTER                       |                 |      1 |  26554 |     48 |
|    25 |                         NESTED LOOPS OUTER                   |                 |      1 |  26554 |     48 |
|-   26 |                          STATISTICS COLLECTOR                |                 |      1 |        |     48 |
|- * 27 |                           HASH JOIN OUTER                    |                 |      1 |  17271 |     48 |
|    28 |                            NESTED LOOPS OUTER                |                 |      1 |  17271 |     48 |
|-   29 |                             STATISTICS COLLECTOR             |                 |      1 |        |     48 |
|- * 30 |                              HASH JOIN OUTER                 |                 |      1 |  11305 |     48 |
|    31 |                               NESTED LOOPS OUTER             |                 |      1 |  11305 |     48 |
|-   32 |                                STATISTICS COLLECTOR          |                 |      1 |        |     48 |
|    33 | BATCHED                         TABLE ACCESS BY INDEX ROWID  | TAB             |      1 |      9 |     48 |
|  * 34 |                                  INDEX RANGE SCAN            | FK_TAB_PROP     |      1 |      9 |     48 |
|    35 |                                TABLE ACCESS BY INDEX ROWID   | TAB_PROP        |     48 |   1326 |     48 |
|  * 36 |                                 INDEX UNIQUE SCAN            | PK_TAB_PROP     |     48 |      1 |     48 |
|- * 37 |                               TABLE ACCESS FULL              | TAB_PROP        |      0 |   1326 |      0 |
|    38 |                             TABLE ACCESS BY INDEX ROWID      | TAB_PROP        |     48 |      2 |     48 |
|  * 39 |                              INDEX UNIQUE SCAN               | PK_TAB_PROP     |     48 |      1 |     48 |
|- * 40 |                            TABLE ACCESS FULL                 | TAB_PROP        |      0 |      2 |      0 |
|    41 |                          TABLE ACCESS BY INDEX ROWID         | TAB_PROP        |     48 |      2 |     48 |
|  * 42 |                           INDEX UNIQUE SCAN                  | PK_TAB_PROP     |     48 |      1 |     48 |
|- * 43 |                         TABLE ACCESS FULL                    | TAB_PROP        |      0 |      2 |      0 |
|    44 |                       TABLE ACCESS BY INDEX ROWID            | TAB_PROP        |     48 |      2 |     48 |
|  * 45 |                        INDEX UNIQUE SCAN                     | PK_TAB_PROP     |     48 |      1 |     48 |
|- * 46 |                      TABLE ACCESS FULL                       | TAB_PROP        |      0 |      2 |      0 |
|-   47 |                    TABLE ACCESS BY INDEX ROWID               | TAB_PROP        |      0 |      2 |      0 |
|- * 48 |                     INDEX UNIQUE SCAN                        | PK_TAB_PROP     |      0 |        |      0 |
|  * 49 |                   TABLE ACCESS FULL                          | TAB_PROP        |      1 |   2486K|   2486K|
|-   50 |                 TABLE ACCESS BY INDEX ROWID                  | TAB_PROP        |      0 |      2 |      0 |
|- * 51 |                  INDEX UNIQUE SCAN                           | PK_TAB_PROP     |      0 |        |      0 |
|  * 52 |                TABLE ACCESS FULL                             | TAB_PROP        |      1 |   2486K|   2486K|
|-   53 |              TABLE ACCESS BY INDEX ROWID                     | TAB_PROP        |      0 |      2 |      0 |
|- * 54 |               INDEX UNIQUE SCAN                              | PK_TAB_PROP     |      0 |        |      0 |
|  * 55 |             TABLE ACCESS FULL                                | TAB_PROP        |      1 |   2486K|   2486K|
|-   56 |           TABLE ACCESS BY INDEX ROWID                        | TAB_PROP        |      0 |      2 |      0 |
|- * 57 |            INDEX UNIQUE SCAN                                 | PK_TAB_PROP     |      0 |        |      0 |
|  * 58 |          TABLE ACCESS FULL                                   | TAB_PROP        |      1 |   2486K|   2486K|
|-   59 |        TABLE ACCESS BY INDEX ROWID                           | TAB_PROP        |      0 |      2 |      0 |
|- * 60 |         INDEX UNIQUE SCAN                                    | PK_TAB_PROP     |      0 |        |      0 |
|  * 61 |       TABLE ACCESS FULL                                      | TAB_PROP        |      1 |   2486K|   2486K|
|-   62 |     TABLE ACCESS BY INDEX ROWID                              | TAB_PROP        |      0 |      2 |      0 |
|- * 63 |      INDEX UNIQUE SCAN                                       | PK_TAB_PROP     |      0 |        |      0 |
|  * 64 |    TABLE ACCESS FULL                                         | TAB_PROP        |      1 |   2486K|   2486K|
---------------------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   1 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
   2 - filter("V"."PROP_ID"=84)
   3 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
   6 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
   9 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
  12 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
  15 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
  18 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
  21 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
  24 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
  27 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
  30 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID")
  34 - access("LI"."TAB_ID"=842300)
  36 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=94)
  37 - filter("V"."PROP_ID"=94)
  39 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=89)
  40 - filter("V"."PROP_ID"=89)
  42 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=93)
  43 - filter("V"."PROP_ID"=93)
  45 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=90)
  46 - filter("V"."PROP_ID"=90)
  48 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=79)
  49 - filter("V"."PROP_ID"=79)
  51 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=81)
  52 - filter("V"."PROP_ID"=81)
  54 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=96)
  55 - filter("V"."PROP_ID"=96)
  57 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=95)
  58 - filter("V"."PROP_ID"=95)
  60 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=88)
  61 - filter("V"."PROP_ID"=88)
  63 - access("LI"."TAB_PROP_ID"="V"."TAB_PROP_ID" AND "V"."PROP_ID"=82)
  64 - filter("V"."PROP_ID"=82)
 
Note
-----
   - dynamic statistics used: dynamic sampling (level=2)
   - this is an adaptive plan (rows marked '-' are inactive)
   - 1 Sql Plan Directive used for this statement

However, the inflection point is never reached. The execution keeps the default plan that has half of the joins HJ and the other half NL.

The problem in this case is the SQL Directive. Why?

There are to many distinct values for TAB_ID and the data is very skewed.

-- without adaptive features
SQL> alter session set optimizer_adaptive_features=false ;
 
Session altered.
 
SQL> select * from APPUSER.V_TAB_PROP where TAB_ID = 842300;
...
48 rows selected.

Elapsed: 00:00:00.23


-- with adaptive features
SQL> alter session set optimizer_adaptive_features=true;
 
Session altered.
SQL> select * from APPUSER.V_TAB_PROP where TAB_ID = 842300;
...
48 rows selected.
 
Elapsed: 00:00:13.84

The histogram on that column is OK and it always leads to the correct plan (with the adaptive features disabled).
But there are still some “minor” misestimates, and the optimizer sometimes decides to create a SQL Plan directive:

SQL> select DIRECTIVE_ID, TYPE, ENABLED, REASON, NOTES from dba_sql_plan_directives where directive_id in (select directive_id from dba_sql_plan_dir_objects where object_name='TAB_PROP');

        DIRECTIVE_ID TYPE             ENA REASON                             
NOTES
-------------------- ---------------- --- ------------------------------------ --------------------------------------------------------------------------------
5347794880142580861 DYNAMIC_SAMPLING YES JOIN CARDINALITY MISESTIMATE        
<spd_note><internal_state>PERMANENT</internal_state><redundant>NO</redundant><spd_text>{F(APPUSER.TAB) - F(APPUSER.TAB_PROP)}</spd_text></spd_note>
 
5473412518742433352 DYNAMIC_SAMPLING YES JOIN CARDINALITY MISESTIMATE        
<spd_note><internal_state>HAS_STATS</internal_state><redundant>NO</redundant><spd_text>{(APPUSER.TAB) - F(APPUSER.TAB) - F(APPUSER.TAB_PROP)}</spd_text></spd_note>
 
14420228120434685523 DYNAMIC_SAMPLING YES JOIN CARDINALITY MISESTIMATE        
<spd_note><internal_state>HAS_STATS</internal_state><redundant>NO</redundant><spd_text>{F(APPUSER.CHAMP) - (APPUSER.TAB) - F(APPUSER.TAB) - F(APPUSER.TAB_PROP)}</spd_text></spd_note>

The Directive instructs the optimizer to do a dynamic sampling, but with a such big and skewed table this is not ok, so the Dynamic sampling result is worse than using the histogram. I can check it by simplifying the query to just one join:

-- with dynamic sampling/sql plan directive:
-------------------------------------------------------------------------------------------
| Id  | Operation                            | Name            | Starts | E-Rows | A-Rows |
-------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                     |                 |      1 |        |     48 |
|   1 |  NESTED LOOPS OUTER                  |                 |      1 |  11305 |     48 |
|   2 |   TABLE ACCESS BY INDEX ROWID BATCHED| TAB             |      1 |      9 |     48 |
|*  3 |    INDEX RANGE SCAN                  | FK_TAB_PROP     |      1 |      9 |     48 |
|   4 |   TABLE ACCESS BY INDEX ROWID        | TAB_PROP        |     48 |   1326 |     48 |
|*  5 |    INDEX UNIQUE SCAN                 | PK_TAB_PROP     |     48 |      1 |     48 |
-------------------------------------------------------------------------------------------
 
-- without dynamic sampling
-------------------------------------------------------------------------------------------
| Id  | Operation                            | Name            | Starts | E-Rows | A-Rows |
-------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                     |                 |      1 |        |     48 |
|   1 |  NESTED LOOPS OUTER                  |                 |      1 |     13 |     48 |
|   2 |   TABLE ACCESS BY INDEX ROWID BATCHED| TAB             |      1 |      9 |     48 |
|*  3 |    INDEX RANGE SCAN                  | FK_TAB_PROP     |      1 |      9 |     48 |
|   4 |   TABLE ACCESS BY INDEX ROWID        | TAB_PROP        |     48 |      2 |     48 |
|*  5 |    INDEX UNIQUE SCAN                 | PK_TAB_PROP     |     48 |      1 |     48 |
-------------------------------------------------------------------------------------------

What’s the fix?

I’ve tried to drop the directive first, but it reappears as soon as there are new misestimates.
The best solution in my case has been to disable the directive, an operation that can be done easily with the DBMS_SPD package:

BEGIN
  FOR rec in (select d.directive_id as did from dba_sql_plan_directives d join dba_sql_plan_dir_objects o on
    (d.directive_id=o.directive_id) where o.owner='APPUSER' and o.object_name in ('TAB','TAB_PROP'))
  LOOP
    DBMS_SPD.ALTER_SQL_PLAN_DIRECTIVE ( rec.did, 'ENABLED','NO');
  END LOOP;
END;
/

I did this on a QAS environment.
Because the production system is not migrated to 12c yet, it’s wise to import these disabled directives in production before the optimizer creates and enables them.

-- export from the source
SET SERVEROUTPUT ON
DECLARE
  my_list  DBMS_SPD.OBJECTTAB := DBMS_SPD.ObjectTab();
  dir_cnt  NUMBER;
BEGIN
  DBMS_SPD.CREATE_STGTAB_DIRECTIVE  (table_name => 'TAB_PROP_DIRECTIVES', table_owner=> 'SYSTEM' );
  my_list.extend(2);
 
  -- TAB table
  my_list(1).owner := 'APPUSER';
  my_list(1).object_name := 'TAB';
  my_list(1).object_type := 'TABLE';
  -- TAB_PROP table
  my_list(2).owner := 'APPUSER';
  my_list(2).object_name := 'TAB_PROP';
  my_list(2).object_type := 'TABLE';
 
  dir_cnt :=
  dir_cnt :=
   DBMS_SPD.PACK_STGTAB_DIRECTIVE(table_name => 'TAB_PROP_DIRECTIVES', table_owner=> 'SYSTEM', obj_list => my_list);
   DBMS_OUTPUT.PUT_LINE('dir_cnt = ' || dir_cnt);
END;
/

expdp directory=data_pump_dir dumpfile=TAB_PROP_DIRECTIVES.dmp logfile=expdp_VAL_LIG_DIRECTIVES.log tables=system.TAB_PROP_DIRECTIVES

-- import into the destination
impdp directory=data_pump_dir dumpfile=TAB_PROP_DIRECTIVES.dmp logfile=impdp_VAL_LIG_DIRECTIVES.log

SELECT DBMS_SPD.UNPACK_STGTAB_DIRECTIVE(table_name => 'TAB_PROP_DIRECTIVES', table_owner=> 'SYSTEM') FROM DUAL;

Off course, the directives can’t be created for objects that do not exist, the import  has to be done after the objects migrate to the 12c version.

Because the SQL Plan Directives are tied to specific objects and not specific queries, they can fix many statements at once, but in case like this one, they can compromise several statements!

Monitoring the creation of new directives is an important task as it may indicate misestimates/lack of statistics on one side or execution plan changes on the other one.

If you are an Oracle customer who has several database versions running, you have to deal with scripts that become more and more complex to maintain. Depending on the version or the edition of your database, you may want to run different pieces of code. This forces you to get programmatically more information about your database version and edition (e.g., in order to run a statspack or AWR report if your software is either Enterprise or Standard).
The most common way to get information about the software is connecting to the database and getting it through a couple of selects. But what if you don’t have any running databases?
The ORACLE_HOME inventory has such information, and you can get it with a short shell function:

function ohversion ()
{
    ORACLE_VERSION=`grep "<PATCH NAME=\"oracle.server\"" $ORACLE_HOME/inventory/ContentsXML/comps.xml 2>/dev/null | tr ' ' '\n' | grep ^VER= | awk -F\" '{print $2}'`;
    if [ -z "$ORACLE_VERSION" ]; then
        ORACLE_VERSION=`grep "<COMP NAME=\"oracle.server\"" $ORACLE_HOME/inventory/ContentsXML/comps.xml 2>/dev/null | tr ' ' '\n' | grep ^VER= | awk -F\" '{print $2}'`;
    fi;
    if [ -z "$ORACLE_VERSION" ]; then
        echo "OH not set";
    fi;
    ORACLE_MAJOR=`echo $ORACLE_VERSION |  cut -d . -f 1`;
    case $ORACLE_MAJOR in
        11 | 12)
            EDITION=`grep "oracle_install_db_InstallType" $ORACLE_HOME/inventory/globalvariables/oracle.server/globalvariables.xml 2>/dev/null | tr ' ' '\n' | grep VALUE | awk -F\" '{print $2}'`
        ;;
        10)
            EDITION=`grep "s_serverInstallType" $ORACLE_HOME/inventory/Components21/oracle.server/*/context.xml 2>/dev/null | tr ' ' '\n' | grep VALUE | awk -F\" '{print $2}'`
        ;;
        *)

        ;;
    esac;
    export ORACLE_VERSION EDITION;
    echo $ORACLE_VERSION $EDITION
}

The snippet searches for a patchset entry in comps.xml to get the patch version rather than the base version (for releases prior to 11gR2 where out-of-place patching occurs). If a patchset cannot be found, it looks for the base version. Depending on the major release, the information about the edition is either in globalvariables.xml (11g, 12c) or in context.xml (10g).
When you call this “ohversion” function, you get both the Oracle version and the edition of your current ORACLE_HOME.
If you’re using the bash as user shell, you may want to take one step forward and  include this information in a much fancier bash prompt than the prompt by default:

function ora_prompt ()
{
    PSERR=$?;
    colylw='\033[0;33m';
    colcyn='\033[0;36m';
    colured='\033[4;31m';
    colugrn='\033[4;32m';
    coluylw='\033[4;33m';
    colrst='\033[0m';
    PS1="\n# [ \u@\h:${colcyn}$PWD${colrst} [\\t] [${colylw}\$(ohversion) SID=${coluylw}${ORACLE_SID:-\"not set\"}${colrst}] \$( if [[ \$PSERR -eq 0 ]]; then echo \"${colugrn}0${colrst}\" ; else echo \"${colured}\$PSERR${colrst}\";fi) ] #\\n# "
}

export PROMPT_COMMAND=ora_prompt

Although this prompt may seem long, it has several advantages that save you a lot of typing:
• The newline character inside the prompt let’s you start typing commands on an almost empty line so you don’t have to worry about how long your command is.
• The full username@host:path can be copied and pasted quickly for scp commands.
• The time inside the square brackets is helpful to track timings.
• The indication of the current environment (version, edition, SID) lets you know which environment you’re working on.
• The leading number is the exit code of the last command ($?). It’s green when the exit code is zero and red for all other exit codes.
• Hash characters before and after the prompt mitigate the risk of copying and pasting the wrong line by mistake inside your session.

Note: this post originally appeared on IOUG Tips & Best Practices Booklet 9th edition.

Version française ici.

After a great success in 2014, RAC Attack! comes back to Geneva!
Set up an Oracle Real Application Clusters 12c environment on your laptop, try advanced configurations or simply take the opportunity to discuss about Oracle technology with the best experts in Suisse Romande!
Experienced volunteers (ninjas) will help you  address any related issues and guide you through the setup process.

Where? Trivadis office, Chemin Château-Bloch 11, CH1219 Geneva

When? Thursday September 17th, 2015, from 17h00 onwards

Cost? It is a FREE event! It is a community based, informal and enjoyable workshop. You just need to bring your own laptop and your desire to have fun!

Confirmed Ninjas:
Ludovico Caldara – Oracle ACE, RAC SIG Chair & co-auteur RAC Attack
Eric Grancher – Membre OAK Table & Senior DBA
Jacques Kostic – OCM 11g & Senior Consultant chez Trivadis

Limited places! Reserve your seat and T-shirt now!

Agenda:
17.00 – Welcome and T-shirt distribution
17.30 – RAC Attack 12c part I
19.30 – Pizza and Beers! (sponsored by Trivadis)
20.00 – RAC Attack 12c part II
22.00 – Group photo and wrap-up!!

Still undecided? Look at what we did last year!

To subscribe, simply fill-in the form below! I will send you.a confirmation as soon as possible. First come, first serve.

Fields marked with an * are required

Enter your details below:

First Name *

Last Name *

Email *

Company

Company Role

T-Shirt preferred size

• XXXL
• XXL
• XL
• L
• M
• S
• XS

Requirements *

To participate in the workshop, you need to bring your own laptop.

Required specification:
a) any 64 bit OS that supports Oracle Virtual Box
b) 8GB RAM, 50GB free HDD space
c) an administrative account on the laptop

Check if you've read and understood the requirements:

I’ve read the recent good post from my friend Rene on Pythian’s Blog about how to troubleshoot when user ulimits are different from what specified in limits.conf:

Quick Tip : Oracle User Ulimit Doesn’t Reflect Value on /etc/security/limits.conf

I would like to add my 2 cents:

Once you fix the problem, you may want to check (any maybe monitor) when an instance is running with a wrong value (and maybe encounter the famous Error message: Linux-x86_64 Error: 23: Too many open files in system).

for pmonspid in `ps -eaf | grep [p]mon | awk '{print $2}'` ; do ps -f -p $pmonspid  ; grep "open files" /proc/$pmonspid/limits ; done

This single line gives you an overview of all your instances at once:

$ for pmonspid in `ps -eaf | grep [p]mon | awk '{print $2}'` ; do ps -f -p $pmonspid  ; grep "open files" /proc/$pmonspid/limits ; done
UID        PID  PPID  C STIME TTY     TIME CMD           
oracle     545     1  0 Mar18 ?   00:08:27 ora_pmon_orcl1
Max open files            1024            1024           files <<< 1024!!
UID        PID  PPID  C STIME TTY     TIME CMD           
oracle    1294     1  0 Apr20 ?   00:00:09 ora_pmon_orcl2
Max open files            1024            1024           files <<< 1024!!
UID        PID  PPID  C STIME TTY     TIME CMD           
oracle    9917     1  0 Jan26 ?   00:08:17 ora_pmon_orcl3
Max open files            1024            1024           files <<< 1024!!
UID        PID  PPID  C STIME TTY     TIME CMD           
oracle   11286     1  0 Jan26 ?   00:07:35 ora_pmon_orcl4
Max open files            1024            1024           files <<< 1024!!
UID        PID  PPID  C STIME TTY     TIME CMD           
oracle   11647     1  0 Mar04 ?   00:04:36 ora_pmon_orcl5
Max open files            65536           65536          files
UID        PID  PPID  C STIME TTY     TIME CMD           
oracle   11836     1  0 Jan26 ?   00:07:55 ora_pmon_orcl6
Max open files            1024            1024           files <<< 1024!!
UID        PID  PPID  C STIME TTY     TIME CMD           
oracle   14183     1  0 Feb06 ?   00:07:13 ora_pmon_orcl7
Max open files            65536           65536          files
UID        PID  PPID  C STIME TTY     TIME CMD           
oracle   16023     1  0 Feb27 ?   00:05:20 ora_pmon_orcl8
Max open files            65536           65536          files
UID        PID  PPID  C STIME TTY     TIME CMD           
oracle   18756     1  0 Mar20 ?   00:03:24 ora_pmon_orcl9
Max open files            65536           65536          files

If you find any wrong values, plan a restart before you encounter any error during peak hours!

—

Ludo

If you use Block Change Tracking on your production database and try to duplicate it, you there are good possibilities that you will encounter this error:

ORA-19755: could not open change tracking file
ORA-19750: change tracking file: ‘/u02/oradata/ORCL/ORCL_bct.dbf’
ORA-27037: unable to obtain file status

The problem is caused by the block change tracking file entry that exists in the target controlfile, but Oracle can’t find the file because the directory structure on the auxiliary server changes.

After the restore and recovery of the auxiliary database, the duplicate process tries to open the DB but the bct file doesn’t exist and the error is thrown.

If you do a quick google search you will find several workarounds:

• disable the block change tracking after you get the error and manually open the auxiliary instance (this prevent the possibility to get the duplicate outcome from the rman return code)
• disable the BCT on the target before running the duplicate (this forces your incremental backups to read all your target database!)
• Richard Harrison proposed another workaround, you can read more about it here.

There is another workaround that I like more (and that you can also find as comment in Richard’s post):

• Disable the Block Change Tracking on the auxiliary while it’s doing the restore/recovery (in mounted status)

(This solutions isn’t coming from me, but as far as I know, the father of this solution is a colleague working at Trivadis.)

You can easily fork a process before running the duplicate command that:

• loops and checks the auxiliary instance status
• run the disable as soon as the auxiliary is mounted

I’ve worked out this script that does the job:

#!/bin/ksh

if [ $# -ne 1 ] ; then
        echo "Usage: $0 \$ORACLE_SID"
        exit 1
fi
ORACLE_SID=$1
export ORACLE_SID

ORACLE_HOME=`cat /etc/oratab | grep ^$ORACLE_SID | awk -F":" '{print $2}'`
export ORACLE_HOME


disable_trk () {
        echo "DISABLE BLOCK CHANGE TRACKING"
        sqlplus -s / as sysdba << EOF
        set echo on
        col FILENAME for a30
        ALTER DATABASE DISABLE BLOCK CHANGE TRACKING;
        select * from v\$block_change_tracking;
EOF
}

echo "Checking block change tracking for $ORACLE_SID"

PATH=${ORACLE_HOME}/bin:$PATH; export PATH

## for how much time do you want to run the check?
INTERVAL=30
MAXCOUNT=30

# loop until the database status is either MOUNTED or OPEN
COUNT=1
while [ $COUNT -le $MAXCOUNT ] ; do
        STATUS=`sqlplus -s / as sysdba <<EOF
                whenever sqlerror exit SQL.SQLCODE;
                set head off feed off
                select status from v\\$instance;
EOF
`
        if [ $? -eq 0 ] ; then
                if [ $STATUS != "MOUNTED" ] && [ $STATUS != "OPEN" ] ; then
                        echo "$COUNT : Still not mounted"
                        sleep $INTERVAL
                        COUNT=$(($COUNT+1))
                        continue;
                else
                        echo
                        echo "If there is an error, BCT is enabled but Oracle can't find the file (ORA-27037)"
                        echo "This is normal, I'll procede with the disable to avoid the error at the end of the duplicate."
                        sqlplus -s / as sysdba <<EOF 2>/dev/null
                        whenever sqlerror exit SQL.SQLCODE;
                        set head off feed off
                        select status from v\$block_change_tracking where status='ENABLED';
EOF
                        # if i get an error, BCT is enabled but Oracle can't find the file (ORA-27037)
                        if [ $? -eq 0 ] ; then
                                echo "BCT already disabled"
                        else
                                echo "Got the error. Let's disable the BCT!"
                                disable_trk
                        fi
                        break;
                fi
        else
                echo "$COUNT : Still not mounted"
                sleep $INTERVAL
                COUNT=$(($COUNT+1))
                continue;
        fi
done

Run it  just before the duplicate! e.g.

/u01/app/oracle/local/bin/disable_trk_dup.sh $ORACLE_SID &
rman target=... auxiliary=...

HTH

—

Ludovico

Today I’ve upgraded EM12c for a customer from the second-last version (12.1.0.3) to the last one (12.1.0.5) and the EM Repository from 11.2.0.3 to 12.1.0.2.

The upgrade path was not very easy: EM 12.1.0.3 is not compatible with a repository 12.1.0.2 and EM 12.1.0.5 requires a mandatory patch for the repository if 11.2.0.3 (or an upgrade to 11.2.0.4).

So I’ve done:

• upgrade of the repository from 11.2.0.3 (in Data Guard configuration) to 11.2.0.4
• upgrade of the EM from 12.1.0.3 to 12.1.0.5
• upgrade of the repository from 11.2.0.4 to 12.1.0.2 (in Data Guard configuration), from Solaris to Linux

In my case, I was particularly concerned about my customer’s EM topology:

• two OMS in load balancing
• console secured with a custom SSL certificate
• a good amount of targets (more than 800 total targets, more than 500 targets with status)
• a lot of jobs and custom reports
• a big, shared central software library
• many other small customizations: auth, groups, metrics, templates…

I will not bother with the actual execution steps, every installation may differ, I strongly recommend to read the upgrade documentation (I know, it’s HUGE ).

Just to resume, the upgrade guide is here: https://docs.oracle.com/cd/E24628_01/upgrade.121/e22625/toc.htm

in my case I had to read carefully the chapters 3, 4, 5, 6 and appendixes G and K.

By following every step carefully, I had no problems at all and at the end everything was working correctly: all the targets up, the load balancing working in SSL as expected, the jobs restarted and ran successfully…

It has been incredible to see how many operations the OUI has done without raising a single error!!

Ok, it’s not just a Click Next Next Next Next installation, there are a lot of steps to do manually before and afterwards, but still… very good impression.

It took a little more than one hour to upgrade the first OMS (this also upgrades the EM repository) and a little less than 20 minutes to upgrade the second one.

Let a couple of hours for checking everything before, staging the binaries, taking backups/snapshots, creating restore points… and one hours more for upgrading the central agents and cleansing the old installations.

About upgrading/moving the repository, check this good post by Maaz Anjum: MIGRATE ENTERPRISE MANAGER 12.1.0.4.0 TO A PDB FROM A NON-CDB, even if you don’t plan to do it, it’s worth a read.

HTH

—

Ludo

After a recent upgrade to Enterprise Manager 12c 12.1.0.5, we noticed that the few Oracle Databases in release 10g were no more connectable:

On MOS there are a few notes about the error: ORA-00604: error occurred at recursive SQL level 1 ORA-01882: timezone region not found, the most relevant being Doc ID 1513536.1 and Doc ID 1934470.1.

The problem is due to the time zone table on the target database that doesn’t contain the timezone requested by the client. But who’s the client? In our case, all the target agents were correctly set to Europe/Zurich, but the timezone table of the target database contained it:

SQL> select version from v$timezone_file;

   VERSION
----------
         4

SQL> SELECT TZname, TzAbbrev FROM V$TIMEZONE_NAMES where TZname like 'Europe/Zurich';

TZNAME             TZABBREV
------------------ ---------------------
Europe/Zurich      LMT
Europe/Zurich      BMT
Europe/Zurich      CET
Europe/Zurich      CEST

So what was causing the problem?

The upgrade process of the OMSes from 12.1.0.3 to 12.1.0.5, without the presence of a specific TZ environment variable, set the OMS timezone to Europe/Vaduz. I figured it out after searching deep and large,  inside the WLS product properties:

$ pwd
/u01/app/oracle/Middleware_12105/wlserver_10.3
$ cat .product.properties
...
USER_TIMEZONE=Europe/Vaduz
...

Indeed, that timezone was not present in the timezone table version 4:

SQL> sELECT TZname, TzAbbrev FROM V$TIMEZONE_NAMES where TZname like 'Europe/Vaduz';

no rows selected

SQL> alter session set time_zone='Europe/Zurich';

Session altered.

SQL> alter session set time_zone='Europe/Vaduz';
ERROR:
ORA-01882: timezone region  not found

After setting explicitly the TZ to Europe/Zurich on the OMS servers and restarting the OMSes, everything was fine again.

HTH

—

Ludo

Disclaimer (wow, most of my recent posts start with a disclaimer, I guess it’s bad): this post explains an UNSUPPORTED workaround for an error enforced by recent Oracle security checks. You should never use it in production! Forewarned is forearmed.

Before Oracle Database 11.2.0.4, it was possible to create a database link using the following syntax:

create database link XX connect to YY 
identified by values 'DEA2G0D1A57B0071057A11DA7A' using 'ZZZ';

It was possible to get the password hash by either selecting dbms_metadata.get_ddl for the database link or by querying directly the link$ table.

Starting with Oracle 11.2.0.4, Oracle is enforcing a check that prevents to use such syntax. Every newly created database link must have the password explicitly set.

This is clearly stated in the MOS note:

ORA-02153: Invalid VALUES Password String When Creating a Database Link Using BY VALUES With Obfuscated Password After Upgrade To 11.2.0.4 (Doc ID 1905221.1)

This is seen as a security enhancement. In my opinion, it forces also to specify clear text passwords somewhere in the scripts that create the db links. (You do not create the db links by hand in sql*plus every time you need one.  Do you?)

The only exception is when using the expdp/impdp. If you expdp a schema, the dumpfile contains the password hash and the statement needed to recreate the database link (… identified by values ‘:1′), but Oracle only allows impdp to use such statement.

So, simple workaround, just create the database links on a dev/staging environment, export them using expdp and then provide your dba the dumpfile so he/she can import it and create the dblinks. Right? Not always.

There is one case where you really need of the old syntax.

• You don’t know the password

AND

• You MUST change the database link name.

As you may know, there are no ways to change a database link name (even through impdp, there is no remap_dblink or anything like that).

E.g., you need to keep the db link and intend to use it for a check BUT you want to prevent the application from using it with the old name.

Because I believe that no problems exist that cannot be solved by my Trivadis’ colleagues, I’ve checked internally. A colleague came out with a dead simple (and unsupported) solution:

Insert/update sys.link$, flush the shared_pool.

SQL> select * from dba_db_links;

OWNER                          DB_LINK              USERNAME                       HOST       CREATED
------------------------------ -------------------- ------------------------------ ---------- ---------
SCOTT                          REMOTEDB             SCOTT                           remotedb   10-APR-15

SQL> select * from sys.link$;
    OWNER# NAME                 CTIME     HOST       USERID     PASSWORD                             FLAG AUTHUSR
---------- -------------------- --------- ---------- ---------- ------------------------------ ---------- ------------------------------
AUTHPWD
------------------------------
PASSWORDX
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
AUTHPWDX
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
         0 REMOTEDB             10-APR-15 remotedb   SCOTT                                              2

061D009E40A5981668DFEE1C710CF68E20B1A4DEE898857B2C3C458C3DEA042675E6CC98CC8D7B72C2F21314D94872D32882BECDE0594B3A525E342B8958BDF37ACE0DE3CE0A4D153AF41EEAF8391A9D84924521C45BA79FF2A2
CEA78709E3BD7775DB9B79A2B4D2F742472B7B5733E142CBCBA2A73511B81F3840611737351

SQL> insert into sys.link$ (OWNER#, NAME, CTIME, HOST, USERID, PASSWORD, FLAG, AUTHUSR, AUTHPWD, PASSWORDX, AUTHPWDX)
  2  select OWNER#, 'NEWDBLINK', CTIME, HOST, USERID, PASSWORD, FLAG, AUTHUSR, AUTHPWD, PASSWORDX, AUTHPWDX 
  3  from sys.link$ where name='REMOTEDB';

1 row created.

SQL> commit;

Commit complete.

SQL> alter system flush shared_pool;

System altered.

SQL> select * from dba_db_links;

OWNER                          DB_LINK              USERNAME                       HOST       CREATED
------------------------------ -------------------- ------------------------------ ---------- ---------
SCOTT                          REMOTEDB             SCOTT                          remotedb   10-APR-15
SCOTT                          NEWDBLINK            SCOTT                          remotedb   10-APR-15

Remember, use it at your own risk (or don’t use it at all)

HTH

—

Ludovico

Last week I have hosted the second Swiss RAC Attack workshop at Trivadis offices in Geneva. It has been a great success, with 21 total participants: 5 Ninjas, 4 alumni and 14 people actively installing or playing with RAC 12c on their laptops.

Last year I was suprised by a participant coming fron Nanterre. This year two people came directly from Moscow, just for the workshop!

We’ve got good pizza and special beer: Chimay , Vedett, Duvel, Andechs…

Last but not least, our friend Marc Fielding was visiting Switzerland last week, so he took the opportunity to join us and make the workshop even more interesting! 

Looking forward to organize it again in one year! Thank you guys

—

Ludovico

Losing the Disk Group that contains OCR and voting files has always been a challenge. It requires you to take regular backups of OCR, spfile and diskgroup metadata.

Since Oracle 12cR1, there are a few additional components you must take care of:

– The ASM password file (if you have Flex ASM it can be quite critical)

– The Grid Infrastructure Management Repository

Why ASM password file is important? Well, you can read this good blog post form my colleague Robert Bialek: http://blog.trivadis.com/b/robertbialek/archive/2014/10/26/are-you-using-oracle-12c-flex-asm-if-yes-do-you-have-asm-password-file-backup.aspx

So the problem here, is not whether you should back them up or not, but how you can restore them quickly.

Assumptions: you back up regularly:

ASM parameter  file:

SQL> create pfile='/backup/spfileASM.ora' from spfile;

File created.

Oracle Cluster Registry:

grid@tvdrach01:~/ [+ASM1] sudo $ORACLE_HOME/bin/ocrconfig -manualbackup
tvdrach03 2015/09/21 14:30:39 /u01/app/grid/12.1.0.2/cdata/tvdrac-cluster/backup_20150921_143039.ocr 0

ASM Diskgroup Metadata:

ASMCMD [+] > md_backup GRID.dg -G GRID
Disk group metadata to be backed up: GRID
Current alias directory path: _MGMTDB/DATAFILE
Current alias directory path: _MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE
Current alias directory path: tvdrac-cluster
Current alias directory path: _MGMTDB/FD9AC0F7C36E4438E043B6A9E80A24D5/DATAFILE
Current alias directory path: _MGMTDB/FD9AC0F7C36E4438E043B6A9E80A24D5
Current alias directory path: ASM/PASSWORD
Current alias directory path: _MGMTDB/TEMPFILE
Current alias directory path: tvdrac-cluster/ASMPARAMETERFILE
Current alias directory path: _MGMTDB/20BC39F0F36C18F4E0533358A8C058F7/TEMPFILE
Current alias directory path: _MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815
Current alias directory path: _MGMTDB/20BC2691871B0B14E0533358A8C01AC6
Current alias directory path: _MGMTDB/ONLINELOG
Current alias directory path: _MGMTDB
Current alias directory path: ASM
Current alias directory path: tvdrac-cluster/OCRFILE
Current alias directory path: _MGMTDB/20BC39F0F36C18F4E0533358A8C058F7
Current alias directory path: _MGMTDB/20BC2691871B0B14E0533358A8C01AC6/TEMPFILE
Current alias directory path: _MGMTDB/CONTROLFILE
Current alias directory path: _MGMTDB/PARAMETERFILE

ASM password file:

ASMCMD [+GRID] > pwcopy +GRID/orapwASM /backup/
copying +GRID/orapwASM -> /backup/orapwASM

What about the GIMR?

According to the MOS Note: FAQ: 12c Grid Infrastructure Management Repository (GIMR) (Doc ID 1568402.1), there is no such need for the moment.

Weird, huh? The -MGMTDB itself contains for the moment just the Cluster Health Monitor repository, but expect to see its important increasing with the next versions of Oracle Grid Infrastructure.

If you REALLY want to back it up (even if not fundamental, it is not a bad idea, after all), you can do it.

The -MGMTDB is in noarchivelog by default. You need to either put it in archivelog mode (and set a recovery area, etc etc) or back it up while it is mounted.

Because the Cluster Health Monitor (ora.crf)  depends on it, you have to stop it beforehand:

grid@tvdrach01:~/ [-MGMTDB] crsctl stop resource ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'tvdrach01'
CRS-2677: Stop of 'ora.crf' on 'tvdrach01' succeeded

Then you can operate with -MGMTDB:

grid@tvdrach01:~/ [-MGMTDB] srvctl stop mgmtdb -stopoption IMMEDIATE
grid@tvdrach01:~/ [-MGMTDB] srvctl start mgmtdb -startoption MOUNT

grid@tvdrach01:~/ [-MGMTDB]

grid@tvdrach02:~/ [-MGMTDB] rman

Recovery Manager: Release 12.1.0.2.0 - Production on Sun Sep 27 17:59:55 2015

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target /

connected to target database: _MGMTDB (DBID=1095800268, not open)

RMAN> backup as compressed backupset database format '+DATA';

Starting backup at 27-SEP-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=24 device type=DISK
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00011 name=+GRID/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/sysmgmtdata.269.891526555
input datafile file number=00007 name=+GRID/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/system.270.891526555
input datafile file number=00008 name=+GRID/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/sysaux.271.891526555
input datafile file number=00010 name=+GRID/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/sysgridhomedata.272.891526555
input datafile file number=00012 name=+GRID/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/sysmgmtdatadb.273.891526555
input datafile file number=00009 name=+GRID/_MGMTDB/FD9B43BF6A646F8CE043B6A9E80A2815/DATAFILE/users.274.891526555
channel ORA_DISK_1: starting piece 1 at 27-SEP-15
channel ORA_DISK_1: finished piece 1 at 27-SEP-15
piece handle=+DATA/_MGMTDB/20BC39F0F36C18F4E0533358A8C058F7/BACKUPSET/2015_09_27/nnndf0_tag20150927t180016_0.256.891540019 tag=TAG20150927T180016 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=+GRID/_MGMTDB/DATAFILE/system.258.891526155
input datafile file number=00003 name=+GRID/_MGMTDB/DATAFILE/sysaux.257.891526135
input datafile file number=00004 name=+GRID/_MGMTDB/DATAFILE/undotbs1.259.891526181
channel ORA_DISK_1: starting piece 1 at 27-SEP-15
channel ORA_DISK_1: finished piece 1 at 27-SEP-15
piece handle=+DATA/_MGMTDB/BACKUPSET/2015_09_27/nnndf0_tag20150927t180016_0.257.891540043 tag=TAG20150927T180016 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00005 name=+GRID/_MGMTDB/FD9AC0F7C36E4438E043B6A9E80A24D5/DATAFILE/system.265.891526233
input datafile file number=00006 name=+GRID/_MGMTDB/FD9AC0F7C36E4438E043B6A9E80A24D5/DATAFILE/sysaux.266.891526233
channel ORA_DISK_1: starting piece 1 at 27-SEP-15
channel ORA_DISK_1: finished piece 1 at 27-SEP-15
piece handle=+DATA/_MGMTDB/20BC2691871B0B14E0533358A8C01AC6/BACKUPSET/2015_09_27/nnndf0_tag20150927t180016_0.258.891540069 tag=TAG20150927T180016 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
Finished backup at 27-SEP-15

Starting Control File and SPFILE Autobackup at 27-SEP-15
piece handle=/u01/app/grid/12.1.0.2/dbs/c-1095800268-20150927-00 comment=NONE
Finished Control File and SPFILE Autobackup at 27-SEP-15

RMAN> alter database open;

Statement processed

RMAN>

Now, imagine that you loose the GRID diskgroup (nowadays, with the ASM Filter Driver, it’s more complex to corrupt a device by mistake, but let’s assume that you do it):

root@tvdrach01:~/ [-MGMTDB] dd if=/dev/zero of=/dev/asm-disk1 bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 0.360653 s, 372 MB/s

The cluster will not start anymore, you need to disable the crs, reboot and start it in exclusive mode:

root@tvdrach01:~/ [-MGMTDB] crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'tvdrach01'
CRS-2672: Attempting to start 'ora.mdnsd' on 'tvdrach01'
CRS-2676: Start of 'ora.mdnsd' on 'tvdrach01' succeeded
CRS-2676: Start of 'ora.evmd' on 'tvdrach01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'tvdrach01'
CRS-2676: Start of 'ora.gpnpd' on 'tvdrach01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'tvdrach01'
CRS-2672: Attempting to start 'ora.gipcd' on 'tvdrach01'
CRS-2676: Start of 'ora.cssdmonitor' on 'tvdrach01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'tvdrach01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'tvdrach01'
CRS-2672: Attempting to start 'ora.diskmon' on 'tvdrach01'
CRS-2676: Start of 'ora.diskmon' on 'tvdrach01' succeeded
CRS-2676: Start of 'ora.cssd' on 'tvdrach01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'tvdrach01'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'tvdrach01'
CRS-2672: Attempting to start 'ora.ctssd' on 'tvdrach01'
CRS-2676: Start of 'ora.ctssd' on 'tvdrach01' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'tvdrach01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'tvdrach01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'tvdrach01'
CRS-2676: Start of 'ora.asm' on 'tvdrach01' succeeded
root@tvdrach01:~/ [-MGMTDB]

Then you can recreate the GRID disk group and restore everything inside it:

SQL> alter system set asm_diskstring='/dev/asm*';

System altered.

SQL> create diskgroup GRID  external redundancy disk '/dev/asm-disk1' attribute 'COMPATIBLE.ADVM'='12.1.0.0.0', 'COMPATIBLE.ASM'='12.1.0.0.0';

Diskgroup created.

SQL> create spfile='+GRID' from pfile='/backup/spfileASM.ora';

File created.

SQL> 

root@tvdrach01:~/ [+ASM1] ocrconfig -restore /u01/app/grid/12.1.0.2/cdata/tvdrac-cluster/backup_20150927_174702.ocr
root@tvdrach01:~/ [+ASM1]

grid@tvdrach01:~/ [+ASM1] crsctl replace votedisk '+GRID'
Successful addition of voting disk a375f4bdb7854f8fbf7a92cd880fba60.
Successfully replaced voting disk group with +GRID.
CRS-4266: Voting file(s) successfully replaced


root@tvdrach01:~/ [+ASM1]  crsctl stop crs -f
...
root@tvdrach01:~/ [+ASM1]  crsctl start crs
...


ASMCMD [+] >  pwcopy --asm /backup/orapwASM +GRID/orapwASM
copying /backup/orapwASM -> +GRID/orapwASM

Finally, the last missing component: the GIMR.

You can recreate it or restore it (if you backed it up at some point in time).

Let’s see how to recreate it:

grid@tvdrach03:~/ [-MGMTDB] srvctl disable mgmtdb
grid@tvdrach03:~/ [-MGMTDB] srvctl remove mgmtdb
Remove the database _mgmtdb? (y/[n]) y
grid@tvdrach01:~/ [+ASM1] dbca -silent -createDatabase -sid -MGMTDB \
> -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc \
> -gdbName _mgmtdb -storageType ASM -diskGroupName +GRID \
> -datafileJarLocation $ORACLE_HOME/assistants/dbca/templates -characterset AL32UTF8 \
> -autoGeneratePasswords -skipUserTemplateCheck
Cleaning up failed steps
5% complete
Registering database with Oracle Grid Infrastructure
11% complete
Copying database files
12% complete
14% complete
21% complete
27% complete
34% complete
41% complete
44% complete
Creating and starting Oracle instance
46% complete
51% complete
52% complete
53% complete
58% complete
62% complete
63% complete
66% complete
Completing Database Creation
70% complete
80% complete
90% complete
100% complete
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.
grid@tvdrach01:~/ [+ASM1] dbca -silent -createPluggableDatabase -sourceDB -MGMTDB \
>  -pdbName tvdrac_cluster -createPDBFrom RMANBACKUP \
>  -PDBBackUpfile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb \
>  -PDBMetadataFile $ORACLE_HOME/assistants/dbca/templates/mgmtseed_pdb.xml \
>  -createAsClone true -internalSkipGIHomeCheck
Creating Pluggable Database
Creating Pluggable Database
4% complete
12% complete
21% complete
38% complete
55% complete
O-GRINF Grid Infrastructure Disaster Recovery
Page 21
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/_mgmtdb/tvdrac_cluster/_mgmtdb.log" for further details.
grid@tvdrach01:~/ [+ASM1] srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node tvdrach01

grid@tvdrach01:~/ [+ASM1] sudo $ORACLE_HOME/bin/crsctl modify res ora.crf -attr ENABLED=1 -init
grid@tvdrach01:~/ [+ASM1] crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'tvdrach01'
CRS-2676: Start of 'ora.crf' on 'tvdrach01' succeeded
grid@tvdrach01:~/ [+ASM1]

Conclusion

Recovering from a lost Disk Group / Cluster is not rocket science. Just practice it every now and then. If you do not have a test RAC, you can build your lab on your laptop using the RAC Attack instructions. If you want to test all the scenarios, the RAC SIG webcast: Oracle 11g Clusterware failure scenarios with practical demonstrations by Kamran Agayev is the best starting point, IMHO. Just keep in mind that Flex ASM and the GIMR add more complexity.

HTH

—

Ludovico

This quick post is mainly for myself… I will certainly use it for reference in the future.

Debugging problems due to adaptive dynamic sampling and in general adaptive features sometimes needs to get historical data about, e.g., parse time.

In order to get this information you may need to query the view DBA_HIST_SYS_TIME_MODEL (take care, it needs Diagnostic Pack license!)

You can use this query as an example.

with h as (
select s.snap_id, s.BEGIN_INTERVAL_TIME,
        --s.END_INTERVAL_TIME,
        g.STAT_ID,
        g.stat_name,
        nvl(
          decode(
            greatest(
              VALUE,
              nvl(lag(VALUE) over (partition by s.dbid, s.instance_number, g.stat_name order by s.snap_id),0)
             ),
            VALUE,
            VALUE - lag(VALUE)
               over (partition by s.dbid,
                                    s.instance_number,
                                    g.stat_name
                    order by s.snap_id
                ),
            VALUE
           ),
           0
        ) VALUE
from DBA_HIST_SNAPSHOT s,
    DBA_HIST_SYS_TIME_MODEL g,
    v$instance i
where s.SNAP_ID=g.SNAP_ID
and s.BEGIN_INTERVAL_TIME >=
    trunc(to_timestamp(nvl('&startdate',to_char(sysdate,'YYYYMMDD')),'YYYYMMDD'))
and s.BEGIN_INTERVAL_TIME < =
    trunc(to_timestamp(nvl('&enddate',to_char(sysdate,'YYYYMMDD')),'YYYYMMDD')+1)
and s.instance_number=i.instance_number
and s.instance_number=g.instance_number
)
select p.begin_interval_time, p.value as "parse time elapsed", t.value as "DB time",
round(p.value/t.value,2)*100 as "parse pct", par.value as opt_adapt_feat
from h p, h t , dba_hist_parameter par
where p.snap_id=t.snap_id
and p.snap_id=par.snap_id
and p.stat_name='parse time elapsed'
and t.stat_name='DB time'
and par.parameter_name='optimizer_adaptive_features'
and t.value>0
order by p.begin_interval_time
/

In this specific example, it shows the “parse time elapsed”, the “DB time” and the percentage parse/dbtime, along with the value of the parameter “optimizer_adaptive_features“. You can use it to check if changing the parameters related to adaptive dynamic sampling improves or not the parse time.

The output will be something like this:

BEGIN_INTERVAL_TIME    	  parse time elapsed     DB time  parse pct OPT_ADAPT_FEAT
-------------- ---------- ------------------ ----------- ---------- ----------
23-OCT-15 03.00.36.569 AM       3235792   	57030479      	5.67 TRUE
23-OCT-15 03.30.38.712 AM       3438093   	60262996       	5.71 TRUE
23-OCT-15 04.00.40.709 AM       4622998   	69813760       	6.62 TRUE
23-OCT-15 04.30.42.776 AM       4590463   	56441202       	8.13 TRUE
23-OCT-15 05.00.44.735 AM      13772357        113741371      	12.11 TRUE
23-OCT-15 05.30.46.722 AM       3448944   	49807800       	6.92 TRUE
23-OCT-15 06.00.48.664 AM       4792886   	54235691       	8.84 TRUE
23-OCT-15 06.30.50.713 AM       8527305   	58775613      	14.51 TRUE
23-OCT-15 07.00.52.667 AM       8518273   	75248056      	11.32 TRUE
23-OCT-15 07.30.54.622 AM       9800048  	17381081       1.07 TRUE
23-OCT-15 08.00.56.609 AM       6986551       1629027583      .43 TRUE
23-OCT-15 08.30.58.568 AM       8414695       2493025822      .34 TRUE
23-OCT-15 09.00.00.457 AM      13648260       2412333113      .57 TRUE
23-OCT-15 09.30.02.384 AM      15186610       4635080356      .33 TRUE
23-OCT-15 10.00.04.298 AM      23465769  	39080849       3.17 FALSE
23-OCT-15 10.30.06.421 AM      12152991       2654461964      .46 FALSE
23-OCT-15 11.00.08.444 AM      24901111        549936076       4.53 FALSE
23-OCT-15 11.30.10.485 AM       8080236        354568317       2.28 FALSE
23-OCT-15 12.00.12.453 PM       4291839   	91028268       	4.71 FALSE
23-OCT-15 12.30.14.430 PM       3675163        177312397       2.07 FALSE
23-OCT-15 01.00.16.468 PM       9184841        231138367       3.97 FALSE
23-OCT-15 01.30.18.438 PM       8132397        162607229       5 FALSE
23-OCT-15 02.00.20.707 PM      13375709        210251458       6.36 FALSE
23-OCT-15 02.30.23.740 PM      10116413        285114368       3.55 FALSE
23-OCT-15 03.00.25.699 PM       8067777        123864339       6.51 FALSE
23-OCT-15 03.30.27.641 PM       5787931        110621767       5.23 FALSE

HTH

—

Ludo

Yesterday I’ve got a weird problem with Adaptive Cursor Sharing. I’m not sure yet about the issue, but it seems to be related to cursor sharing histograms. Hopefully one day I will blog about what I’ve learnt from this experience.

To better monitor the problem on that specific query, I’ve prepared this script (tested on 12.1.0.2):

COLUMN Shareable HEADING 'S|H|A|R|E|A|B|L|E'
COLUMN "Bind-Aware" HEADING 'B|I|N|D| |A|W|A|R|E'
COLUMN Sensitive HEADING 'S|E|N|S|I|T|I|V|E'
COLUMN Reoptimizable HEADING 'R|E|O|P|T|I|M|I|Z|A|B|L|E'
BREAK on child_number on Execs on "Gets/Exec" on "Ela/Exec" on "Sensitive" on "Shareable" on "Bind-Aware" on bucket0 on bucket1 on bucket2 on cnt on "Reoptimizable" on is_resolved_adaptive_plan

select * from (select *
  from (
select 
s.child_number,
  s.plan_hash_value,
  executions as Execs, 
  round(buffer_gets/executions) as "Gets/Exec",
  round(elapsed_time/executions) as "Ela/Exec",
  is_bind_sensitive as "Sensitive",
  is_shareable as "Shareable",
  is_bind_aware as "Bind-Aware",
  to_char(h.bucket_id) as bucket, h.count as cnt,
  is_reoptimizable as "Reoptimizable",
  is_resolved_adaptive_plan,
  "UNBOUND_CURSOR",  "SQL_TYPE_MISMATCH",  "OPTIMIZER_MISMATCH",
  "OUTLINE_MISMATCH", "STATS_ROW_MISMATCH", "LITERAL_MISMATCH",
  "FORCE_HARD_PARSE", "EXPLAIN_PLAN_CURSOR", "BUFFERED_DML_MISMATCH",
  "PDML_ENV_MISMATCH", "INST_DRTLD_MISMATCH", "SLAVE_QC_MISMATCH",
  "TYPECHECK_MISMATCH", "AUTH_CHECK_MISMATCH", "BIND_MISMATCH",
  "DESCRIBE_MISMATCH", "LANGUAGE_MISMATCH", "TRANSLATION_MISMATCH",
  "BIND_EQUIV_FAILURE", "INSUFF_PRIVS", "INSUFF_PRIVS_REM",
  "REMOTE_TRANS_MISMATCH", "LOGMINER_SESSION_MISMATCH", "INCOMP_LTRL_MISMATCH",
  "OVERLAP_TIME_MISMATCH", "EDITION_MISMATCH", "MV_QUERY_GEN_MISMATCH",
  "USER_BIND_PEEK_MISMATCH", "TYPCHK_DEP_MISMATCH", "NO_TRIGGER_MISMATCH",
  "FLASHBACK_CURSOR", "ANYDATA_TRANSFORMATION", "PDDL_ENV_MISMATCH",
  "TOP_LEVEL_RPI_CURSOR", "DIFFERENT_LONG_LENGTH", "LOGICAL_STANDBY_APPLY",
  "DIFF_CALL_DURN", "BIND_UACS_DIFF", "PLSQL_CMP_SWITCHS_DIFF",
  "CURSOR_PARTS_MISMATCH", "STB_OBJECT_MISMATCH", "CROSSEDITION_TRIGGER_MISMATCH",
  "PQ_SLAVE_MISMATCH", "TOP_LEVEL_DDL_MISMATCH", "MULTI_PX_MISMATCH",
  "BIND_PEEKED_PQ_MISMATCH", "MV_REWRITE_MISMATCH", "ROLL_INVALID_MISMATCH",
  "OPTIMIZER_MODE_MISMATCH", "PX_MISMATCH", "MV_STALEOBJ_MISMATCH",
  "FLASHBACK_TABLE_MISMATCH", "LITREP_COMP_MISMATCH", "PLSQL_DEBUG",
  "LOAD_OPTIMIZER_STATS", "ACL_MISMATCH", "FLASHBACK_ARCHIVE_MISMATCH",
  "LOCK_USER_SCHEMA_FAILED", "REMOTE_MAPPING_MISMATCH", "LOAD_RUNTIME_HEAP_FAILED",
  "HASH_MATCH_FAILED", "PURGED_CURSOR", "BIND_LENGTH_UPGRADEABLE",
  "USE_FEEDBACK_STATS"
from v$sql s
  join v$sql_cs_histogram h
    on (s.sql_id=h.sql_id and
	s.child_number=h.child_number and
	s.con_id=h.con_id
	)
  join v$sql_shared_cursor shc
    on (shc.sql_id=h.sql_id and 
	shc.child_number=h.child_number and
	s.con_id=shc.con_id
	)
	where s.sql_id='&sql_id'
)
pivot (sum(cnt) for (bucket) IN ('0' AS Bucket0,'1' AS Bucket1,'2' AS Bucket2))
)
unpivot (result FOR reason_type IN ("UNBOUND_CURSOR",
  "SQL_TYPE_MISMATCH", "OPTIMIZER_MISMATCH",
  "OUTLINE_MISMATCH", "STATS_ROW_MISMATCH", "LITERAL_MISMATCH",
  "FORCE_HARD_PARSE", "EXPLAIN_PLAN_CURSOR", "BUFFERED_DML_MISMATCH",
  "PDML_ENV_MISMATCH", "INST_DRTLD_MISMATCH", "SLAVE_QC_MISMATCH",
  "TYPECHECK_MISMATCH", "AUTH_CHECK_MISMATCH", "BIND_MISMATCH",
  "DESCRIBE_MISMATCH", "LANGUAGE_MISMATCH", "TRANSLATION_MISMATCH",
  "BIND_EQUIV_FAILURE", "INSUFF_PRIVS", "INSUFF_PRIVS_REM",
  "REMOTE_TRANS_MISMATCH", "LOGMINER_SESSION_MISMATCH", "INCOMP_LTRL_MISMATCH",
  "OVERLAP_TIME_MISMATCH", "EDITION_MISMATCH", "MV_QUERY_GEN_MISMATCH",
  "USER_BIND_PEEK_MISMATCH", "TYPCHK_DEP_MISMATCH", "NO_TRIGGER_MISMATCH",
  "FLASHBACK_CURSOR", "ANYDATA_TRANSFORMATION", "PDDL_ENV_MISMATCH",
  "TOP_LEVEL_RPI_CURSOR", "DIFFERENT_LONG_LENGTH", "LOGICAL_STANDBY_APPLY",
  "DIFF_CALL_DURN", "BIND_UACS_DIFF", "PLSQL_CMP_SWITCHS_DIFF",
  "CURSOR_PARTS_MISMATCH", "STB_OBJECT_MISMATCH", "CROSSEDITION_TRIGGER_MISMATCH",
  "PQ_SLAVE_MISMATCH", "TOP_LEVEL_DDL_MISMATCH", "MULTI_PX_MISMATCH",
  "BIND_PEEKED_PQ_MISMATCH", "MV_REWRITE_MISMATCH", "ROLL_INVALID_MISMATCH",
  "OPTIMIZER_MODE_MISMATCH", "PX_MISMATCH", "MV_STALEOBJ_MISMATCH",
  "FLASHBACK_TABLE_MISMATCH", "LITREP_COMP_MISMATCH", "PLSQL_DEBUG",
  "LOAD_OPTIMIZER_STATS", "ACL_MISMATCH", "FLASHBACK_ARCHIVE_MISMATCH",
  "LOCK_USER_SCHEMA_FAILED", "REMOTE_MAPPING_MISMATCH", "LOAD_RUNTIME_HEAP_FAILED",
  "HASH_MATCH_FAILED", "PURGED_CURSOR", "BIND_LENGTH_UPGRADEABLE",
  "USE_FEEDBACK_STATS"))
where result='Y'
order by child_number;

The result is something similar (in my case it has 26 child cursors):

R
                                                                    E
                                                                    O
                                                                  B P
                                                              S S I T
                                                              E H N I
                                                              N A D M
                                                              S R   I
                                                              I E A Z
                                                              T A W A
                                                              I B A B
                                                              V L R L
CHILD_NUMBER PLAN_HASH_VALUE      EXECS  Gets/Exec   Ela/Exec E E E E I    BUCKET0    BUCKET1    BUCKET2 REASON_TYPE                   R
------------ --------------- ---------- ---------- ---------- - - - - - ---------- ---------- ---------- ----------------------------- -
           0      2293695281        455       2466      14464 Y Y Y N            0        455          0 ROLL_INVALID_MISMATCH         Y
                  2293695281                                                                             BIND_EQUIV_FAILURE            Y
           1      1690560038         99      13943     103012 Y Y Y N            0         99          0 ROLL_INVALID_MISMATCH         Y
                  1690560038                                                                             BIND_EQUIV_FAILURE            Y
           2      3815006743        541      43090     230245 Y Y Y N            0        541          0 BIND_EQUIV_FAILURE            Y
                  3815006743                                                                             ROLL_INVALID_MISMATCH         Y
           3      1483632464        251       4111      18940 Y Y Y N           49        202          0 ROLL_INVALID_MISMATCH         Y
                  1483632464                                                                             BIND_EQUIV_FAILURE            Y
           4      3815006743       1152      42632     220730 Y Y Y N            0       1000          0 BIND_EQUIV_FAILURE            Y
                  3815006743                                                                             ROLL_INVALID_MISMATCH         Y
           5      3922835573        150      39252     184176 Y Y Y N            0        150          0 ROLL_INVALID_MISMATCH         Y
                  3922835573                                                                             BIND_EQUIV_FAILURE            Y
           6       767857637          3       4731     124707 Y Y Y N            0          3          0 ROLL_INVALID_MISMATCH         Y
                   767857637                                                                             BIND_EQUIV_FAILURE            Y
           7       767857637         11       4739      71119 Y Y Y N            0         11          0 BIND_EQUIV_FAILURE            Y
           8      2800467281          1        307     249727 Y Y Y N            0          1          0 BIND_EQUIV_FAILURE            Y
           9      3123241890        536       2982      14428 Y Y Y N            6        530          0 ROLL_INVALID_MISMATCH         Y
                  3123241890                                                                             BIND_EQUIV_FAILURE            Y
          10      3125518635         17        315      16492 Y Y Y N           16          1          0 ROLL_INVALID_MISMATCH         Y
                  3125518635                                                                             BIND_EQUIV_FAILURE            Y
          11      2184442252        130       4686      40188 Y Y Y N            0        130          0 ROLL_INVALID_MISMATCH         Y
                  2184442252                                                                             BIND_EQUIV_FAILURE            Y
          12      3815006743        553      42765     231391 Y Y Y N            0        553          0 ROLL_INVALID_MISMATCH         Y
                  3815006743                                                                             BIND_EQUIV_FAILURE            Y
          13      1166983254         47      14193     111256 Y Y Y N            0         47          0 BIND_EQUIV_FAILURE            Y
                  1166983254                                                                             ROLL_INVALID_MISMATCH         Y
          14      2307602173          2         38      45922 Y Y Y N            2          0          0 BIND_EQUIV_FAILURE            Y
                  2307602173                                                                             ROLL_INVALID_MISMATCH         Y
          15       767857637         11       4304      59617 Y Y Y N            0         11          0 BIND_EQUIV_FAILURE            Y
                   767857637                                                                             ROLL_INVALID_MISMATCH         Y
          16      3108045525          2      34591     176749 Y N N N            1          1          0 ROLL_INVALID_MISMATCH         Y
                  3108045525                                                                             LOAD_OPTIMIZER_STATS          Y
                  3108045525                                                                             BIND_EQUIV_FAILURE            Y
          17      3108045525          6       1794      33335 Y Y Y N            4          2          0 BIND_EQUIV_FAILURE            Y
                  3108045525                                                                             ROLL_INVALID_MISMATCH         Y
          18      2440443365        470       2009      13361 Y Y Y N            0        470          0 ROLL_INVALID_MISMATCH         Y
                  2440443365                                                                             BIND_EQUIV_FAILURE            Y
          19      4079924956         15       2032      19773 Y Y Y N            8          7          0 ROLL_INVALID_MISMATCH         Y
                  4079924956                                                                             BIND_EQUIV_FAILURE            Y
          20       777919270         32       2675      18260 Y Y Y N           11         21          0 BIND_EQUIV_FAILURE            Y
                   777919270                                                                             ROLL_INVALID_MISMATCH         Y
          21      1428146033         63      13929     111116 Y Y Y N            0         63          0 ROLL_INVALID_MISMATCH         Y
                  1428146033                                                                             BIND_EQUIV_FAILURE            Y
          22      3815006743        218      43673     234642 Y Y Y N            0        218          0 BIND_EQUIV_FAILURE            Y
                  3815006743                                                                             ROLL_INVALID_MISMATCH         Y
          23       277802667          1         62      99268 Y Y Y N            1          0          0 BIND_EQUIV_FAILURE            Y
                   277802667                                                                             ROLL_INVALID_MISMATCH         Y
          24      3898025231          3       2364     111231 Y Y Y N            0          3          0 BIND_EQUIV_FAILURE            Y
                  3898025231                                                                             ROLL_INVALID_MISMATCH         Y
          25       767857637          2       6495     169363 Y Y Y N            0          2          0 ROLL_INVALID_MISMATCH         Y
                   767857637                                                                             BIND_EQUIV_FAILURE            Y
          26      3690167092        100       2998      20138 Y Y Y N            0        100          0 BIND_EQUIV_FAILURE            Y
                  3690167092                                                                             ROLL_INVALID_MISMATCH         Y

It’s a quick way to get the relevant information in a single result.

Off course, if you need deeper details, you should consider something more powerful like SQLd360 from Mauro Pagano.

Credits: I’ve got the unpivot idea (and copied that part of the code) from this post by Timur Akhmadeev.

—

Ludo