Moving 11gR2 RAC to New Set of ASM Diskgroups

This post is for a situation that require moving the RAC components from current set of ASM disk groups to new set of ASM disk group. Situation could arise in the form of moving to higher redundancy disk group than current ASM disk group redundancy or migrating to a new SAN. The steps required to move are actually a collection of previous posts and will be referenced throughout. The post uses setup created with11gR2 RAC installation on RHEL6.
The current setup has 3 ASM diskgroups CLUSTER_DG (Quorum disk group), DATA and FLASH. The new diskgroups are named NEWCLUSTERDG, NEWDATA and NEWFLASH. If the asm_diskstring is different for example if old asm_diskstring is /dev/emcpower* and new asm_diskstring is /dev/mapper/mpath* then asm_diskstring must be edited to include both (old and new) asm_diskstring until all relevant files are moved out of the old set of ASM diskgroup. Once completed asm_diskstring could be edited to include only the currently valid set of disks. If ASMLib is used then this step could be omitted as asm_diskstring would not change (ORCL:* for all). This step is not shown and it is assumed that ASM disk groups could be created without any issue.
The main steps include

1. Moving cluster related files that are in ASM to new ASM diskgroup
2. Moving database related files to new ASM diskgroup

1. Moving cluster related files that are in ASM to new ASM diskgroup
This could be further itemized as follows

1.1. Moving OCR to new ASM diskgroup
1.2. Moving Vote disks to new ASM diskgroup
1.3. Moving server side ASM SPfile to new ASM diskgroup

Same diskgroup will be used for all these files types (OCR,Vote and ASM SPfile) and this disk group is created as a quorum diskgroup.

create diskgroup newclusterdg quorum 
failgroup fail1 disk '/dev/sdg1' 
failgroup fail2 disk '/dev/sdh1' 
failgroup fail3 disk '/dev/sdi1' 
attribute 'compatible.asm'='11.2';

SQL> select name,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
CLUSTER_DG                     MOUNTED
DATA                           MOUNTED
FLASH                          MOUNTED
NEWCLUSTERDG                   MOUNTED   <-- used for moving the cluster files

Once created the diskgroup is only mounted on the node it's created.

crsctl status resource ora.NEWCLUSTERDG.dg
NAME=ora.NEWCLUSTERDG.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE           , OFFLINE
STATE=ONLINE on rhel6m1, OFFLINE

It must be mounted on other nodes as well before continuing with the cluster file moving. Run the following on each node to mount the new diskgroup

 alter diskgroup NEWCLUSTERDG mount;

or mount it accross the cluster with

# crsctl start resource ora.NEWCLUSTERDG.dg
CRS-2672: Attempting to start 'ora.NEWCLUSTERDG.dg' on 'rhel6m2'
CRS-2676: Start of 'ora.NEWCLUSTERDG.dg' on 'rhel6m2' succeeded

crsctl status resource ora.NEWCLUSTERDG.dg
NAME=ora.NEWCLUSTERDG.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE           , ONLINE
STATE=ONLINE on rhel6m1, ONLINE on rhel6m2

or by instance using srvctl

srvctl start diskgroup  -g newclusterdg -n rhel6m2

srvctl status diskgroup  -g newclusterdg -n "rhel6m1,rhel6m2" -a
Disk Group data is running on rhel6m1,rhel6m2
Disk Group data is enabled on rhel6m1,rhel6m2

1.1. Moving OCR to new ASM diskgroup
OCR could be moved to new ASM disk either using the add/delete method or replace option. However when there's only one OCR file (no OCR mirror) then replace option cannot be used. OCR add/delete must be run as root.

ocrcheck  (# current configuration)
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3328
         Available space (kbytes) :     258792
         ID                       : 1487892601
         Device/File Name         : +CLUSTER_DG
                                    Device/File integrity check succeeded

# ocrconfig -replace +CLUSTER_DG -replacement +NEWCLUSTERDG
PROT-28: Cannot delete or replace the only configured Oracle Cluster Registry location

First the new ASM diskgroup must be added and this will appear as the OCRMirror. Afterwards OCR location in the old diskgroup is removed (this will make the ocrmirror to become the ocr).

# ocrconfig -add +NEWCLUSTERDG

# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3344
         Available space (kbytes) :     258776
         ID                       : 1487892601
         Device/File Name         : +CLUSTER_DG
                                    Device/File integrity check succeeded
         Device/File Name         : +NEWCLUSTERDG
                                    Device/File integrity check succeeded


# ocrconfig -delete +CLUSTER_DG

# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3344
         Available space (kbytes) :     258776
         ID                       : 1487892601
         Device/File Name         : +NEWCLUSTERDG
                                    Device/File integrity check succeeded

This concludes the moving of OCR to new ASM diskgroup.
Related Post: Migrating OCR to ASM in 11gR2 Clusterware
Related metalink note : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]

1.2. Moving Vote disks to new ASM diskgroup
The vote disk could be moved with the replace option using crsctl. Unlike previous versions (10.2, 11.1) in 11.2 the vote disk replace could be run as grid user. Current vote disk configuration

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2c90c4f4a0fb4faebfb264244a836b96 (/dev/sdb1) [CLUSTER_DG]
 2. ONLINE   1d7890a346284fa4bf0e278b2f3b1d3d (/dev/sdc1) [CLUSTER_DG]
 3. ONLINE   433f41571bf44fcebfe636a97d94777a (/dev/sdd1) [CLUSTER_DG]
Located 3 voting disk(s).

Run replace command

$ crsctl replace votedisk +NEWCLUSTERDG
Successful addition of voting disk 58e2e7ab4cdc4fc1bf98b745437a5082.
Successful addition of voting disk 1fb2f493ec644ffdbfaca24bbbf41864.
Successful addition of voting disk 74048ba14f1b4f8dbf677abb76f929c1.
Successful deletion of voting disk 2c90c4f4a0fb4faebfb264244a836b96.
Successful deletion of voting disk 1d7890a346284fa4bf0e278b2f3b1d3d.
Successful deletion of voting disk 433f41571bf44fcebfe636a97d94777a.
Successfully replaced voting disk group with +NEWCLUSTERDG.
CRS-4266: Voting file(s) successfully replaced

New vote disk configuration

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   58e2e7ab4cdc4fc1bf98b745437a5082 (/dev/sdg1) [NEWCLUSTERDG]
 2. ONLINE   1fb2f493ec644ffdbfaca24bbbf41864 (/dev/sdh1) [NEWCLUSTERDG]
 3. ONLINE   74048ba14f1b4f8dbf677abb76f929c1 (/dev/sdi1) [NEWCLUSTERDG]
Located 3 voting disk(s).

That concludes the moving of vote disk to new ASM diskgroup
Related Post: Migrating Voting Disk to ASM in 11gR2 Clusterware
Related metalink note : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]

1.3. Moving server side ASM SPfile to new ASM diskgroup
In 11gR2 RAC and Standalone (Restart) the ASM SPfile reside in the ASM disk group itself. Current location of the ASM Spfile could be obtained in number of ways. Show parameter command (running it on ASM instance)

SQL> show parameter spfile;

NAME      TYPE        VALUE
--------- ----------- ------------------------------------------------------------------
spfile    string      +CLUSTER_DG/rhel6m-cluster/asmparameterfile/registry.253.785350531

It's also visible in the ASM alert log

Machine:        x86_64
Using parameter settings in server-side spfile +CLUSTER_DG/rhel6m-cluster/asmparameterfile/registry.253.785350531
System parameters with non-default values

and with spget on ASMCMD

ASMCMD> spget
+CLUSTER_DG/rhel6m-cluster/asmparameterfile/registry.253.785350531

To move the ASM SPfile to new ASM diskgroup create ASM pfile from the current spfile

SQL> create pfile='/home/grid/asmpfile.ora' from spfile;

Create ASM SPfile in the new ASM diskgroup location using the earlier created pfile

SQL> create spfile='+newclusterdg' from pfile='/home/grid/asmpfile.ora';

New SPfile location will be logged on the ASM alert log

NOTE: updated gpnp profile ASM SPFILE to +NEWCLUSTERDG/rhel6m-cluster/asmparameterfile/registry.253.806083655

This commands update the gpnp profile and during the next restart ASM will use the new SPFile. This could be verified using the ASM alert log or asmcmd spget command. To explicitly set the SPFile location use ASMCMD's spset command (this step is not necessary)

ASMCMD> spset +NEWCLUSTERDG/rhel6m-cluster/asmparameterfile/registry.253.806083655
Using parameter settings in server-side spfile +NEWCLUSTERDG/rhel6m-cluster/asmparameterfile/registry.253.806083655

This concludes the moving of ASM SPfile to new ASM diskgroup
Related metalink note : How to move ASM spfile to a different disk group [ID 1082943.1]

The ASM diskgroup that stores cluster files could be dropped now. However since copies of various files (OCR, ASMSPfile) still reside in it trying to drop without content clause would throw an error

SQL> drop diskgroup cluster_dg;
drop diskgroup cluster_dg
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15053: diskgroup "CLUSTER_DG" contains existing files

It is also important that before dropping the diskgroup is dismounted from all other ASM instances in the cluster, if not an error will be thrown

SQL> drop diskgroup cluster_dg including contents;
drop diskgroup cluster_dg including contents
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15073: diskgroup CLUSTER_DG is mounted by another ASM instance

SQL> alter diskgroup cluster_dg dismount;  <-- run on all ASM instances except one

At times trying to dismount could throw the following error

SQL> alter diskgroup newclusterdg dismount;
alter diskgroup newclusterdg dismount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15027: active use of diskgroup "CLUSTER_DG" precludes its dismount

This could be due to file reference still opened even though they are not used. In such cases restart of OHAS on the nodes where dismount fails could resolve this issue.
Once dismounted drop the ASM diskgroup.

SQL> drop diskgroup cluster_dg including contents;
Diskgroup dropped.

2. Moving database related files to new ASM diskgroup

This step could be further itemized as

2.1. Moving control files to new ASM diskgroup
2.2. Moving SPfile to new ASM diskgroup
2.3. Moving data files to new ASM diskgroup
2.4. Moving temp files to new ASM diskgroup
2.5. Moving online redo log files to new ASM diskgroup
2.6. Moving backup files to new ASM diskgroup

Create new disk groups used for data files and fast recovery area and mount them across all the ASM instances

SQL> create diskgroup newdata external redundancy disk '/dev/sdj1';
Diskgroup created.

SQL> create diskgroup newflash external redundancy disk '/dev/sdk1';
Diskgroup created.

SQL> alter diskgroup newdata mount;
Diskgroup altered.

SQL> alter diskgroup newflash mount;
Diskgroup altered.

Metalink note How To Move The Database To Different Diskgroup (Change Diskgroup Redundancy) [ID 438580.1] shows how to move a database from one diskgroup to another using backup database as copy. This is also a valid option to move the database from asm-to-asm. In this post steps are shown to move each individual file type related to the database from one asm disk to another.

2.1. Moving control files to new ASM diskgroup
Currently the system has two control files in two diskgroups

SQL> show parameter control

NAME          TYPE   VALUE
------------- ------ ------------------------------
control_files string +DATA/std11g2/controlfile/current.256.785694741,
                     +FLASH/std11g2/controlfile/current.256.785694743

Before cloning the control files create a pfile from the current SPFile. This pfile will also be used for moving the SPFile as well.

SQL> create pfile='/home/oracle/pfile.ora' from spfile;
File created.

Update the control file locations on the SPFile by specifying the new ASM diskgroups.

SQL> alter system set control_files='+NEWDATA','+NEWFLASH' scope=spfile sid='*';
System altered.

Stop the database and start one instance in nomount mode.

srvctl stop database -d std11g2
srvctl start instance -d std11g2 -i std11g22 -o nomount

Restore control files to new location using the current control file.

rman target /

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Jan 30 16:56:16 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
connected to target database: STD11G2 (not mounted)

RMAN> restore controlfile from '+DATA/std11g2/controlfile/current.256.785694741';

Starting restore at 30-JAN-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=141 instance=std11g22 device type=DISK

channel ORA_DISK_1: copied control file copy
output file name=+NEWDATA/std11g2/controlfile/current.256.806086607
output file name=+NEWFLASH/std11g2/controlfile/current.256.806086609
Finished restore at 30-JAN-13

Mount and open the database and verify the new control file locations

SQL> alter database mount;
Database altered.

SQL> alter database open;
Database altered.


SQL> show parameter control

NAME          TYPE   VALUE
------------- ------ ------------------------------
control_files string +NEWDATA/std11g2/controlfile/current.256.806086607,
                     +NEWFLASH/std11g2/controlfile/current.256.806086609

This concludes the moving of control files to new ASM disk groups.
Related Post : Cloning/Duplicating controlfile between ASM diskgroups
Related metalink note : How to duplicate a controlfile when ASM is involved [ID 345180.1]
Useful metalink note : How To Move The Database To Different Diskgroup (Change Diskgroup Redundancy) [ID 438580.1]

2.2. Moving SPfile to new ASM diskgroup
If the earlier created pfile is used for creating the spfile in this section modify the control_files parameter value in the pfile to reflect the new locations, otherwise create a new pfile from the spfile which will have the new control files location. Create a spfile in the new asm disk using the pfile (again it's important that control file locations reflect the new values. Modify pfile before creating spfile)

SQL> create spfile='+newdata' FROM PFILE='/home/oracle/pfile.ora';
File created.

Replacing the SPFile involves creating a alias to new spfile and updating the database configuration. Loging to ASM using ASMCMD (as grid user) and create the spfile alias

ASMCMD> mkalias +NEWDATA/STD11G2/PARAMETERFILE/spfile.257.806088197 spfilestd11g2.ora
ASMCMD> ls -l
Type           Redund  Striped  Time             Sys  Name
                                                 Y    CONTROLFILE/
                                                 Y    PARAMETERFILE/
                                                 N    spfilestd11g2.ora => +NEWDATA/STD11G2/PARAMETERFILE/spfile.257.806088197

srvctl config database -d std11g2
Database unique name: std11g2
Database name: std11g2
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/std11g2/spfilestd11g2.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: std11g2
Database instances: std11g21,std11g22
Disk Groups: DATA,FLASH,NEWDATA,NEWFLASH
Mount point paths:
Services: myservice
Type: RAC
Database is administrator managed

$ srvctl modify database -d std11g2 -p +NEWDATA/std11g2/spfilestd11g2.ora

$ srvctl config database -d std11g2
Database unique name: std11g2
Database name: std11g2
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +NEWDATA/std11g2/spfilestd11g2.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: std11g2
Database instances: std11g21,std11g22
Disk Groups: DATA,FLASH,NEWDATA,NEWFLASH
Mount point paths:
Services: myservice
Type: RAC
Database is administrator managed

Modify the pfiles that give reference to the SPFile in $ORACLE_HOME/dbs

cat /opt/app/oracle/product/11.2.0/dbhome_1/dbs/initstd11g22.ora

SPFILE='+NEWDATA/std11g2/spfilestd11g2.ora'

Restart the database and SPFile parameter will reflect the new SPFile location.

SQL> show parameter spfile

NAME     TYPE         VALUE 
-------- -----------  --------------------------------
spfile    string      +DATA/std11g2/spfilestd11g2.ora

After DB restart

SQL> show parameter spfile

NAME     TYPE         VALUE 
-------- -----------  --------------------------------
spfile    string      +NEWDATA/std11g2/spfilestd11g2.ora

This concludes the moving of SPFile to new ASM diskgroup.
Related Post : Replacing spfile in ASM
Related Post : Oracle 11gR2 RAC SPfile Issue

2.3. Moving data files to new ASM diskgroup
Moving data files from one ASM disk group to another involves creating a backup of the data file as a copy in the new asm diskgroup and switching to the data file copy. Following SQL could be used to generate the backup as copy rman commands for all the data files

SQL> select 'backup as copy datafile '||file#||' format ''+NEWDATA'';' from v$datafile;

Run a RMAN session using the commands generated above (output is truncated)

RMAN> run{
2> backup as copy datafile 1 format '+NEWDATA';
3> backup as copy datafile 2 format '+NEWDATA';
4> backup as copy datafile 3 format '+NEWDATA';
5> backup as copy datafile 4 format '+NEWDATA';
6> backup as copy datafile 5 format '+NEWDATA';
7> backup as copy datafile 6 format '+NEWDATA';
8> backup as copy datafile 7 format '+NEWDATA';
9> }

Starting backup at 31-JAN-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=17 instance=std11g21 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA/std11g2/datafile/system.259.785694749
output file name=+NEWDATA/std11g2/datafile/system.258.806151043 tag=TAG20130131T105042 RECID=2 STAMP=806151061
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:26
Finished backup at 31-JAN-13
...
...

Shutdown all database instances and start one instance in mount mode

srvctl stop database -d std11g2
srvctl start instance -d std11g2 -i std11g21 -o mount

Use this instance to run the switch to copy command. Following SQL could be used to generate the switch command for all the data files

SQL> select 'switch datafile '||file#||' to copy;' from v$datafile;

Run the switch command from a RMAN session (output is truncated)

switch datafile 1 to copy;
switch datafile 2 to copy;
switch datafile 3 to copy;
switch datafile 4 to copy;
switch datafile 5 to copy;
switch datafile 6 to copy;
switch datafile 7 to copy;

RMAN>datafile 1 switched to datafile copy "+NEWDATA/std11g2/datafile/system.258.806151043"

RMAN>datafile 2 switched to datafile copy "+NEWDATA/std11g2/datafile/sysaux.259.806151073"
...
...

Recover each datafile. Following SQL could be used to generate the recover commands

select 'recover datafile '||file#||';' from v$datafile;

Run a RMAN session to recover the datafiles (output is truncated)

recover datafile 1;
recover datafile 2;
recover datafile 3;
recover datafile 4;
recover datafile 5;
recover datafile 6;
recover datafile 7;

RMAN>
Starting recover at 31-JAN-13
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 31-JAN-13
...
...

Once the recovery is completed open the database and bring up the other instances

SQL> alter database open;

Take a full backup of the database and delete the data file copies

RMAN> delete datafilecopy all;

using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=19 instance=std11g21 device type=DISK
List of Datafile Copies
=======================

Key     File S Completion Time Ckp SCN    Ckp Time
------- ---- - --------------- ---------- ---------------
9       1    A 31-JAN-13       36408130   31-JAN-13
        Name: +DATA/std11g2/datafile/system.259.785694749

10      2    A 31-JAN-13       36408130   31-JAN-13
        Name: +DATA/std11g2/datafile/sysaux.260.785694765

11      3    A 31-JAN-13       36408130   31-JAN-13
        Name: +DATA/std11g2/datafile/undotbs1.261.785694779

12      4    A 31-JAN-13       36408130   31-JAN-13
        Name: +DATA/std11g2/datafile/undotbs2.263.785694797

13      5    A 31-JAN-13       36408130   31-JAN-13
        Name: +DATA/std11g2/datafile/users.264.785694801

14      6    A 31-JAN-13       36408130   31-JAN-13
        Name: +DATA/std11g2/datafile/example.268.785857285

15      7    A 31-JAN-13       36408130   31-JAN-13
        Name: +DATA/std11g2/datafile/test.269.785857289


Do you really want to delete the above objects (enter YES or NO)? yes
deleted datafile copy
datafile copy file name=+DATA/std11g2/datafile/system.259.785694749 RECID=9 STAMP=806151523
deleted datafile copy
datafile copy file name=+DATA/std11g2/datafile/sysaux.260.785694765 RECID=10 STAMP=806151524
deleted datafile copy
...
...

This concludes the moving of data files to new ASM diskgroup.
Related Post : Moving Datafiles Between ASM Disk Groups
Related Post : Moving Datafiles to new location
Useful metalink note : How to move a datafile from ASM to the file system [ID 390416.1]

2.4. Moving temp files to new ASM diskgroup
To move the temp file, add a new temp file to the temporary tablespace by giving the new asm diskgroup as the location. Once the new temp file is in place remove the existing temp file.

SQL> alter tablespace temp add tempfile '+newdata(tempfile)';
Tablespace altered.

SQL> alter tablespace temp drop tempfile '+DATA/std11g2/tempfile/temp.262.785694783';
Tablespace altered.

This concludes the moving of temp file to new asm diskgroup.

2.5. Moving online redo log files to new ASM diskgroup
Moving online diskgroups could be done in two ways. One is to drop the current online redo log groups and add new online redo log groups in the new disks groups. Second option is to add new log files with new ASM diskgroup locations into the existing diskgroups and then removing log files that reside in the current (about to be removed) location. The second option is shown below. Current log files

SQL> select group#,member from v$logfile;

    GROUP# MEMBER
---------- --------------------------------------------------
         1 +DATA/std11g2/onlinelog/group_1.257.785694743
         1 +FLASH/std11g2/onlinelog/group_1.257.785694745
         2 +DATA/std11g2/onlinelog/group_2.258.785694747
         2 +FLASH/std11g2/onlinelog/group_2.258.785694747
         3 +DATA/std11g2/onlinelog/group_3.265.785696921
         3 +FLASH/std11g2/onlinelog/group_3.259.785696923
         4 +DATA/std11g2/onlinelog/group_4.266.785696925
         4 +FLASH/std11g2/onlinelog/group_4.260.785696927

Add log file members using the new ASM diskgroups

SQL> alter database add logfile member '+newdata','+newflash' to group 1;

SQL> select group#,member from v$logfile order by 1;

    GROUP# MEMBER
---------- --------------------------------------------------
         1 +DATA/std11g2/onlinelog/group_1.257.785694743
         1 +FLASH/std11g2/onlinelog/group_1.257.785694745
 1 +NEWDATA/std11g2/onlinelog/group_1.266.806152613
         1 +NEWFLASH/std11g2/onlinelog/group_1.257.806152615
         2 +FLASH/std11g2/onlinelog/group_2.258.785694747
         2 +DATA/std11g2/onlinelog/group_2.258.785694747
         3 +DATA/std11g2/onlinelog/group_3.265.785696921
         3 +FLASH/std11g2/onlinelog/group_3.259.785696923
         4 +FLASH/std11g2/onlinelog/group_4.260.785696927
         4 +DATA/std11g2/onlinelog/group_4.266.785696925

Group 1 now has 4 log files. Add log files to all other groups as well. Switch through all the groups so that newly created log files members also get written to by executing "archive current logs"

SQL> alter system archive log current;

This has to be done until all the log files groups have done a switch and an archive. If not trying to remove the second log file member will result in an error.

SQL> alter database drop logfile member '+DATA/std11g2/onlinelog/group_1.257.785694743';
Database altered.   <--- first member removed without an issue

SQL> alter database drop logfile member '+FLASH/std11g2/onlinelog/group_1.257.785694745';
alter database drop logfile member '+FLASH/std11g2/onlinelog/group_1.257.785694745' <--- trying remove second member gives an error.
*
ERROR at line 1:
ORA-00362: member is required to form a valid logfile in group 1
ORA-01517: log member: '+FLASH/std11g2/onlinelog/group_1.257.785694745'

This is because newly added log files are missing the redo records of the existing log files. If both of them were dropped prior to archive recovery will not find required redo records therefore a full log switch must be performed for all the log file groups before issuing the drop redo command. Once removed only redo log files with the new location will remain

SQL> select group#,member from v$logfile order by 1;

    GROUP# MEMBER
---------- --------------------------------------------------
         1 +NEWFLASH/std11g2/onlinelog/group_1.257.806152615
         1 +NEWDATA/std11g2/onlinelog/group_1.266.806152613
         2 +NEWFLASH/std11g2/onlinelog/group_2.258.806152855
         2 +NEWDATA/std11g2/onlinelog/group_2.267.806152853
         3 +NEWDATA/std11g2/onlinelog/group_3.268.806152859
         3 +NEWFLASH/std11g2/onlinelog/group_3.259.806152859
         4 +NEWDATA/std11g2/onlinelog/group_4.269.806152863
         4 +NEWFLASH/std11g2/onlinelog/group_4.260.806152863

This concludes the moving of online redo log files to new location.
Useful metalink notes : How To Move The Database To Different Diskgroup (Change Diskgroup Redundancy) [ID 438580.1]
Useful metalink notes : How to Change The Location Of Redo Log Files [ID 395062.1]

Related to data files, online redo files there are several database parameters that must be changed. Change db_recovery_file_dest (used as an archive log location among other things)and db_create_file_dest (default datafile location)

NAME      TYPE     VALUE
----------------------- -------- -------
db_recovery_file_dest   string   +FLASH
db_create_file_dest     string   +DATA

SQL> alter system set db_recovery_file_dest='+NEWFLASH' scope=both sid='*';
System altered.

SQL> alter system set db_create_file_dest='+newdata' scope=both sid='*';
System altered.

NAME      TYPE     VALUE
----------------------- -------- -------
db_recovery_file_dest   string   +NEWFLASH
db_create_file_dest     string   +NEWDATA

Change the db_create_online_log_dest* location as well

SQL> alter system set db_create_online_log_dest_1='+newdata' SCOPE=BOTH SID='*';
System altered.

SQL> alter system set db_create_online_log_dest_2='+newflash' SCOPE=BOTH SID='*';
System altered.

Archive the current log files and verify they archive to new location (if only recovery file dest is used for archive logs)

SQL> alter system archive log current;
SQL> select max(sequence#),thread# from v$archived_log  group by thread#;

MAX(SEQUENCE#)    THREAD#
-------------- ----------
           200          1
           151          2

SQL> select name from v$archived_log where (sequence#=200 and thread#=1) or (sequence#=151 and thread#=2);

NAME
----------------------------------------------------------------------
+NEWFLASH/std11g2/archivelog/2013_01_31/thread_1_seq_200.261.806153191
+NEWFLASH/std11g2/archivelog/2013_01_31/thread_2_seq_151.262.806153193

Create a tablespace without specifying the datafile clause and see if db create file dest uses new location

SQL> create tablespace abc;
Tablespace created.

SQL> select tablespace_name,file_name from dba_data_files where tablespace_name='ABC';

TABLESPACE_NAME                FILE_NAME
------------------------------ --------------------------------------------------
ABC                            +NEWDATA/std11g2/datafile/abc.270.806153661

Query following views v$controlfile,v$datafile,v$tempfile,v$logfile (also v$block_change_tracking if block tracking is enabled) to verify that no file exists in the ASM diskgroup about to be dropped. Querying ASM views show what type of files remain in the ASM diskgroups

SQL> select ag.name,af.type from v$asm_diskgroup ag,v$asm_file af where ag.group_number=af.group_number and af.group_number=2;

NAME  TYPE
----- ---------------
DATA  CONTROLFILE
DATA  PARAMETERFILE

Since control file and SPFile both moved out this ASM diskgroup could be dropped. Unmount the diskgroup from all nodes except one and drop it from there.

SQL> alter diskgroup data dismount;
Diskgroup altered.

SQL> drop diskgroup data including contents;
Diskgroup dropped.

2.6. Moving backup files to new ASM diskgroup
In this setup the only backup related parameter was the db_recovery_file_dest parameter which was modified in the 2.5 to reflect the new ASM diskgroup name. Therefore subsequent backups will use this new ASM diskgroup. If the existing backups could be discarded then there's no more steps required and the old ASM diskgroup could be dropped. However if the existing backups need to be moved to the new ASM diskgroup location then further steps are needed.
To clear any archive logs that still remains in the ASM diskgroup that will be dropped, backup and delete the archive logs with

backup archivelog all delete input;

Find out the file types that are in the backup related disk group (in this case flash).

SQL> select distinct af.type,ag.name from v$asm_diskgroup ag,v$asm_file af where ag.group_number=af.group_number and af.group_number=3;

NAME                 TYPE
-------------------- --------------------
FLASH                CONTROLFILE
FLASH                BACKUPSET
FLASH                AUTOBACKUP

Using ASMCMD create a new directory structure in the new ASM diskgroup similar to existing backup files directory structure.

ASMCMD> cd +newflash/std11g2/
ASMCMD> mkdir backupset
ASMCMD> cd backupset
ASMCMD> mkdir 2013_01_31

Remember to copy the files without file number of incarnation. (Explained in ASMCMD cp command fails with ORA-15046 [ID 452158.1])

ASMCMD> cp +flash/std11g2/backupset/2013_01_31/annnf0_TAG20130131T114145_0.266.806154115 +newflash/std11g2/backupset/2013_01_31/annnf0_TAG20130131T114145
copying +flash/std11g2/backupset/2013_01_31/annnf0_TAG20130131T114145_0.266.806154115 -> +newflash/std11g2/backupset/2013_01_31/annnf0_TAG20130131T114145
ASMCMD> ls +newflash/std11g2/backupset/2013_01_31
annnf0_TAG20130131T114145
...
ASMCMD> cp +flash/std11g2/autobackup/2013_01_31/s_806158183.279.806158185 +newflash/std11g2/autobackup/2013_01_31/s_806158183

Once all the backup files are copied over catalog the recovery area

RMAN> catalog recovery area;

using target database control file instead of recovery catalog
searching for all files in the recovery area

List of Files Unknown to the Database
=====================================
File Name: +newflash/STD11G2/backupset/2013_01_31/annnf0_TAG20130131T114145
File Name: +newflash/STD11G2/AUTOBACKUP/2013_01_31/s_806158183

Do you really want to catalog the above files (enter YES or NO)? yes
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: +newflash/STD11G2/backupset/2013_01_31/annnf0_TAG20130131T114145
File Name: +newflash/STD11G2/AUTOBACKUP/2013_01_31/s_806158183

List of files in Recovery Area not managed by the database
==========================================================
File Name: +NEWFLASH/std11g2/controlfile/current.256.806086609
  RMAN-07527: Reason: File was not created using DB_RECOVERY_FILE_DEST initialization parameter
File Name: +NEWFLASH/std11g2/onlinelog/group_1.257.806152615
  RMAN-07527: Reason: File was not created using DB_RECOVERY_FILE_DEST initialization parameter
File Name: +NEWFLASH/std11g2/onlinelog/group_2.258.806152855
  RMAN-07527: Reason: File was not created using DB_RECOVERY_FILE_DEST initialization parameter
File Name: +NEWFLASH/std11g2/onlinelog/group_3.259.806152859
  RMAN-07527: Reason: File was not created using DB_RECOVERY_FILE_DEST initialization parameter
File Name: +NEWFLASH/std11g2/onlinelog/group_4.260.806152863
  RMAN-07527: Reason: File was not created using DB_RECOVERY_FILE_DEST initialization parameter

number of files not managed by recovery area is 5, totaling 217.84MB

Message "RMAN-07527: Reason: File was not created using DB_RECOVERY_FILE_DEST initialization parameter" has been explained in Ora-19816 raised in alert log after creating new log group in FRA [ID 1100416.1]
Delete backup files in the ASM diskgroup to be removed with

RMAN> delete backup COMPLETED BEFORE 'sysdate -2';

or using backup file specific tag.

RMAN> delete backup tag 'TAG20130131T114145';

It is also possible to drop the backup related files along with the disk group if removing backups files one at a time is not convenient. Verify the types of files in the ASM diskgroup all have been migrated to new ASM diskgroups

SQL> select distinct af.type,ag.name from v$asm_diskgroup ag,v$asm_file af where ag.group_number=af.group_number and af.group_number=3;

TYPE            NAME
--------------- ------------------------------
AUTOBACKUP      FLASH
BACKUPSET       FLASH
CONTROLFILE     FLASH

In this case controflile was migrated and backup files were copied over and remaining files will be dropped along with the disk group. Dismount the disk group being dropped from all ASM instances except one.

SQL> alter diskgroup flash dismount;
Diskgroup altered.

Drop the ASM diskgroup from the ASM instance where it is still mounted

SQL> drop diskgroup flash including contents;
Diskgroup dropped.

With this step the entire RAC is now running on the new set of ASM diskgroups

SQL> select name from v$asm_diskgroup;

NAME
-------------
NEWCLUSTERDG
NEWDATA
NEWFLASH