Resizing Grid Disks
Procedure
This section demonstrates the resizing procedure in detail.Â
1. Ensure prerequisites are metÂ
- FREE_MB > REQUIRED_MIRROR_FREE_MB in V$ASM_DISKGROUP for the disk group that will shrink (typically RECO); ideally, FREE_MB > 2 * REQUIRED_MIRROR_FREE_MB [USED_MB may be more realistic, need to check] to cut number of rebalances in half by doing and ADD/DROP operation simultaneously for each cell, except the first and last cells. On the first cell you will do a DROP disk operation and on the last one you will do an ADD disk operation.Â
- If the RECO disk group is configured as high redundancy, then you can do this procedure on a Half Rack or greater. You will not be able to do this procedure on a Quarter Rack because ASM will not allow you to drop a failure group such that only one copy of the data remains.Â
- All disk groups are backed up and you have tested the recovery strategy.Â
- The load on the system will not inhibit rebalancing activity at the desired power level.Â
- All DCLI commands should use the celladmin user on the cells as a best practice. This may require establishing equivalency to the celladmin user from the oracle user on DB nodes.Â
- Run the Exachk script ( MOS note 1070954.1) one or two weeks before doing this procedure to ensure the system is healthy.Â
2. Calculate RECO and DATA disk group and grid disk sizing changesÂ
In this step we assume that you have properly estimated the space needed for objects to store in RECO (without considering redundancy), e.g., number of archivelogs you want to store, etc. The system in the examples below was a quarter-rack installation containing three cells for a total of 36 grid disks in each of the DATA_A and RECO_A disk groups. We are using DATA_A and RECO_A disk groups instead of the customary DATA and RECO disk groups to make our testing flexible and not interfere with the other disk groups (this also demonstrates that the procedure will work for any two disk groups).Â
This value is known as USABLE_MB and is expressed in megabytes.
We will first obtain information about the current disk group sizes:
a. Query 1: Find the amount of free space in the disk groups:Â
select group_number, name, type,total_mb,free_mb,required_mirror_free_mb,usable_file_mb,(total_mb-free_mb) used_mb
from v$asm_diskgroup
order by group_number;
 Â
GROUP_NUMBER NAMEÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â TYPEÂ Â Â Â TOTAL_MBÂ Â Â FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MBÂ Â Â USED_MB
———— —————————— —— ———- ———- ———————– ————– ———-
…
          4 DATA_A_DM01                 NORMAL   6659712   6577052                2219904       2178574     82660
          5 RECO_A_DM01                 NORMAL  10742976  10721816                3580992       3570412     21160Â
For our example here, we will choose to resize DATA_A_DM01 and RECO_A_DM01. Other disk groups won’t be affected and hence aren’t shown.Â
We need to determine the number of cells and disks that are being used; this is accomplished by the following queries:
b. Query 2: Find miscellaneous information about the ASM disks:Â
SQL> SELECT dg.name, count(1) “Num Disks”
FROM v$asm_disk d, v$asm_diskgroup dg
WHERE d.group_number = dg.group_number
GROUP BY dg.name;
 2   3   4 Â
NAMEÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Num Disks
—————————— ———-
DATA_A_DM01Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 36
DATA_DM01Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 36
DBFS_DGÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 30
RECO_A_DM01Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 36
RECO_DM01Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 36
SQL> SELECT d.group_number, dg.name, d.os_mb
FROMÂ v$asm_disk d, v$asm_diskgroup dg
WHERE d.group_number = dg.group_number
AND dg.name IN (‘DATA_A_DM01′,’RECO_A_DM01’)
ORDER BY dg.name
…
          4 DATA_A_DM01                     184992
          4 DATA_A_DM01                     184992
          5 RECO_A_DM01                     298416
          5 RECO_A_DM01                     298416
          5 RECO_A_DM01                     298416   Â
c. Query 3: Find the number of failure groups (cells):Â
SELECT dg.name DG_NAME, count(distinct failgroup) NUM_FAILGROUPS
FROM v$asm_disk d, v$asm_diskgroup dg
WHERE d.group_number = dg.group_number
AND dg.name like ‘RECO%’
GROUP BY dg.name;
Â
DG_NAMEÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â NUM_FAILGROUPS
—————————— ————–
RECO_A_DM01Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 3Â
For this specific example, based on our needs, we have determined that RECO_A requires 1 TB of usable file space (instead of the 3.5 TB it currently has) and will resize disk groups DATA_A and RECO_A accordingly. This should give DATA_A an increase of about 2.5 TB for a total of about 4.6 TB of usable free space.Â
We decide to reserve space in case of cell failure instead of disk failure, so we will use this formula to calculate the total size of the RECO disk group we will need (see the discussion in the Exadata Database Machine Owners Guide, Chapter 7, Resizing Storage Grid Disks:Â
 RECO_TOTAL_MB = 2 * C /(C-1) * USABLE_MB Â
Using the above queries, we know the following:
C = NUM_FAILGROUPS (in query 3) = 3
D = DISK_COUNT (in query 2) in the RECO and DATA disk groups = 36Â
We apply the formulas, with our chosen value of USABLE_MB = 1 TB:Â
RECO_TOTAL_MB = 2 * C /(C-1) * USABLE_MBÂ = 2 * 3/2 * 1048576 = 3145728 MB
RECO_GD_MB = RECO_TOTAL_MB / DÂ Â = CEIL(3145728 / 36/16) * 16 = 87392 MB
Note: The CEIL() function returns values to the next largest integer; this is used to round up to the next 16MB boundary.Â
DATA GD_MB = (DATA_OS_MB + RECO_OS_MB) – RECO GD_MBÂ
                          = (184992 + 298416) – 87392 = 396016 MB
New DATA disk group total size = DATA GD_MB * D = 396016* 36 = 14256576 MB ~ 14 TBÂ
As a sanity check, we look at the existing grid disk definitions for RECO_A:Â
$ dcli -g cell_group -l celladmin “cellcli -e list griddisk attributes name,size where name like \’RECO_A.*\'”
dm01cel04: RECO_A_CD_01_dm01cel04Â Â Â 291.421875G
…
dm01cel06: RECO_A_CD_12_dm01cel06Â Â Â 291.421875G
Â
The current size of the RECO_A grid disks agree with the value of OS_MB in query 2 (291.421875G = 298416 MB). These will be re-created with the size of 87392 MB we calculated above.
This command will be used to find the DBFS_DG starting offset to use when we recreate the RECO grid disks (this will likely be different in your environment):Â
 dcli -g cell_group -l celladmin “cellcli -e list griddisk attributes name,offset | grep –i dbfs_dg“
 dm01cel04: DBFS_DG_CD_03_dm01cel04          854.9375G
…
dm01cel06: DBFS_DG_CD_12_dm01cel06Â Â Â Â Â Â Â Â Â Â 854.9375GÂ
We also look at the existing grid disk definitions for DATA_A:
$ dcli -g cell_group -l celladmin “cellcli -e list griddisk attributes name,size where name like \’DATA_A.*\'”
dm01cel04: DATA_A_CD_01_dm01cel04Â Â Â Â Â Â Â Â 180.65625G
…
dm01cel06: DATA_A_CD_12_dm01cel06Â Â Â Â Â Â Â Â 180.65625G
The current size of the DATA_A grid disks agree with the value of OS_MB in query 2. These grid disks will be resized to the value of 396026 MB. We sanity-check our calculations here to ensure we are growing and not shrinking the DATA_A grid disks (e.g., 396026 > 184992).Â
3. DROP ASM disks from RECO Disk group on the first cell
First, ensure no rebalancing operations are occurring by running this query (and all subsequent SQL statements shown in the examples below in an ASM instance); no rows should return from this query:Â
SQL> SELECT * FROM gv$asm_operation;
If no rebalancing operations are occurring then, drop the ASM disks from the RECO disk group for one cell (in this example we have cells dm01cel04 – dm01cel06 in a split-rack):    Â
SQL> alter diskgroup RECO_A_DM01 drop disks in failgroup DM01CEL04 rebalance power 32 NOWAIT;Â
 In the example above we use a power level of 32 with the assumption that ASM is running version 11.2.0.2 or higher. Your choice of power level depends on how much workload is on the system at the time of the rebalance operation, how quickly you wish the rebalance to finish, and how much higher power levels will impact the workload. We do not recommend rebalance power levels higher than 32. If there is application activity that is sensitive to I/O performance, you may want to start with a small rebalance power and increase it (using the ALTER DISKGROUP command) to higher power levels until the application begins to be impacted.Â
4. Wait for Rebalance to Finish
First, ensure no rebalancing operations are occurring by running this query; no rows should return:Â Â Â Â Â Â Â Â Â Â Â
SQL> SELECT * FROM gv$asm_operation;Â
Note: Rebalancing involves a “compaction†phase after the file extents are rebalanced and redundancy is established. This compaction phase may take some time and its progress is not reported in gv$asm_operation. Please be patient while this activity proceeds. To monitor progress, please see MOS note 1274322.1 for the script in entry, “Check ASM rebalance forward progress if you suspect a problemâ€Â
5. Drop the RECO grid disks on the cell
a. Ensure that ASM disks were properly dropped:Â
column header_status heading HSTATUS format A15
column name format A35
column path format A55
set lines 200
set pages 100Â
SQL> SELECT group_number,path,failgroup,header_status,mount_status
  FROM v$asm_disk
  WHERE failgroup = ‘DM01CEL04’
  ORDER BY header_status,path;Â
PATHÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â FAILGROUPÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â HSTATUSÂ Â Â Â Â Â Â Â MOUNT_S
——————————————————- —————————— ————— ——-
o/192.168.227.203/RECO_A_CD_01_dm01cel04Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â DM01CEL04Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â FORMERÂ Â Â Â Â Â Â Â Â CLOSED
o/192.168.227.203/RECO_A_CD_02_dm01cel04Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â DM01CEL04Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â FORMERÂ Â Â Â Â Â Â Â Â CLOSED
…
o/192.168.227.203/RECO_A_CD_11_dm01cel04Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â DM01CEL04Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â FORMERÂ Â Â Â Â Â Â Â Â CLOSED
o/192.168.227.203/RECO_A_CD_12_dm01cel04Â Â Â Â Â Â Â Â Â Â Â Â Â Â DM01CEL04Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â FORMERÂ Â Â Â Â Â Â Â Â CLOSEDÂ
 The GROUP_NUMBER should be zero if the disks were successfully dropped (usually you will also see the HEADER_STATUS as “FORMER†and the MOUNT_STATUS as “CLOSED†), otherwise do not proceed until you investigate the state of the disks further.Â
b. Verify the grid disks are in the proper status to proceed:Â
$ dcli –c dm01cel04 –l celladmin “cellcli –e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome“
        RECO_A_CD_01_dm01cel04       UNUSED Yes
…
        RECO_A_CD_12_dm01cel04       UNUSED YesÂ
If the ASMMODESTATUS is “UNUSED†and the ASMDEACTIVATIONOUTCOME is “Yesâ€, then it is safe to proceed. Otherwise, investigate further and correct before continuin
c. Drop the grid disks from the cell
From the first DB node, run the following DCLI command as the oracle user (be sure the cell name is correct!):
 $ dcli –c dm01cel04 –l celladmin “cellcli -e drop griddisk all harddisk prefix=RECO_A”
NOTE: you may need to add the FORCE option to the “drop griddisk†command if the cell doesn’t allow this command to succeed and you are sure that the ASM disks on the specific cell were successfully dropped in steps 3 and 4 above.Â
6. Recreate RECO grid disks on the cell to a smaller size
We will need the size and offset calculations that were done in step 2 here. For this particular example:
- The DBFS_DG disk group has its starting offset at 854.9375 GB = 875456 MB
- We calculated the new size for the RECO_A grid disks as  87392 MB
 Therefore, the starting offset for the RECO_A grid disk will be: 875456 – 87392 = 788064 MB. This will make the RECO_A grid disks contiguous with the DBFS_DG grid disks on the inner part of the celldisk.
 From the first DB node, run the following DCLI command as the oracle user to create the RECO_A grid disks:
 $ dcli -c dm01cel04 -l celladmin “cellcli -e CREATE GRIDDISK ALL HARDDISK PREFIX=’RECO_A’, offset=788064M, size=87392M“
 Verify the grid disks were created properly and contiguous with DBFS_DG:
 $ dcli -c dm01cel04 -l celladmin “cellcli -e list griddisk attributes name, offset, size”
 Verify free space available now in the celldisk to use for the DATA_A grid disks in a later step (there should only be one “offsetâ€, “size†pair listed per celldisk):Â
 $ dcli -c dm01cel04 -l celladmin “cellcli -e list celldisk attributes name,freespace,freespacemap“
 dm01cel04: CD_01_dm01cel04  291.421875G ((offset=769.59375G,size=291.421875G)) Â
7. Add RECO ASM disks from the cell to the RECO Disk group and drop RECO ASM disks from the next cellÂ
If sufficient free space exists in the RECO DG (twice the amount of REQUIRED_MIRROR_FREE_MB), then you can run this command:
SQL> alter diskgroup RECO_A_DM01 drop disks in failgroup DM01CEL05 add disk ‘o/*/RECO_A*dm01cel04′ rebalance power 32 NOWAIT;Â
Otherwise, you’ll need to run the add and drop commands separately:
SQL> alter diskgroup RECO_A_DM01 add disk ‘o/*/RECO_A*dm01cel04′ rebalance power 32 NOWAIT;
Wait for rebalance to finish successfully:Â
SQL> SELECT * FROM gv$asm_operation;
SQL> alter diskgroup RECO_A_DM01 drop disks in failgroup DM01CEL05 rebalance power 32 NOWAIT;Â
8. Wait for Rebalance to Finish
Ensure no rebalancing operations are occurring by running this query; no rows should return:
SQL> SELECT * FROM gv$asm_operation;
 Verify ASM disks were dropped and grid disks are ready to be dropped as in steps 5a and 5b above.
 Ensure dm01cel04 RECO_A ASM disks are added back successfully with HEADER_STATUS=â€MEMBER†and MOUNT_STATUS=â€CACHEDâ€; and dm01cel05 RECO ASM disks are now unused:Â
 PATH                                        FAILGROUP    HSTATUS MOUNT_S
——————————————– ————- ——– ——-
o/192.168.227.203/RECO_A_CD_01_dm01cel04Â Â DM01CEL04Â Â MEMBERÂ Â CACHED
o/192.168.227.203/RECO_A_CD_02_dm01cel04Â Â DM01CEL04Â Â MEMBERÂ Â CACHED
…
o/192.168.227.204/RECO_A_CD_01_dm01cel05Â Â DM01CEL05Â Â FORMERÂ Â CLOSED
o/192.168.227.204/RECO_A_CD_02_dm01cel05Â Â DM01CEL05Â Â FORMERÂ Â CLOSED
9. Repeat steps 5 through 8 for the remaining cells
Be careful to change the name of the cells and failure groups as you copy and execute commands on the various cells.
Note: On the final cell, we only need to add the ASM disks back into the disk group with the command:
 SQL> alter diskgroup RECO_A_DM01 add disk ‘o/*/RECO_A*dm01cel06′ rebalance power 32 NOWAIT;Â
10. Wait for Rebalance to Finish
Ensure no rebalancing operations are occurring by running this query; no rows should return:
SQL> SELECT * FROM gv$asm_operation;
Verify ASM disks were added properly using the query in 5a above (be sure to change the failgroup name); the HEADER_STATUS should be “MEMBER†and the MOUNT_STATUS should be “CACHEDâ€:Â
PATHÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â FAILGROUPÂ Â Â Â Â Â Â Â HSTATUSÂ MOUNT_S
—————————————————- —————————— ————— ——-
o/192.168.227.205/RECO_A_CD_01_dm01cel06Â Â Â Â Â Â Â Â DM01CEL06Â Â MEMBERÂ Â Â Â Â Â CACHED
o/192.168.227.205/RECO_A_CD_02_dm01cel06Â Â Â Â Â Â Â Â DM01CEL06Â Â MEMBERÂ Â Â Â Â Â CACHED
 When rebalancing is done, you may proceed to the next step.Â
11. Resize all DATA grid disks up to the desired size
Now that all RECO_A grid disks have been recreated to a smaller size, there should be sufficient free space in every celldisk to extend the DATA_A grid disks.
 a. Confirm available spaceÂ
$ dcli –g cell_group -l celladmin “cellcli -e list celldisk attributes name,freespaceâ€
dm01cel04: CD_01_dm01cel04Â Â 206.078125GÂ Â
…Â
      b. Confirm current size of the DATA_A grid disks: Â
$ dcli –g cell_group -l celladmin “cellcli -e list griddisk attributes name,size“
dm01cel04: DATA_A_CD_01_dm01cel04Â Â Â Â Â Â Â Â Â Â Â 180.65625G
…
…
      c. Calculate the new size for the DATA_A grid disks:
Current size = 180.65625G
Free space = 206.078125G
New size = current size + free space = 180.65625G + 206.078125G = 386.734375G = 396016 MB
 We note that this is exactly what we calculated in step 2 for “DATA GD_MB”, so we will proceed; any discrepancies should be investigated.Â
     d. Execute the DATA_A grid disk resizing commands:
Hint: You can obtain a list of disks to resize using this command:
SQL>Â select name||’,’ from v$asm_disk where group_number = order by path
Â
$ dcli -c dm01cel04 -l celladmin “cellcli -e alter grid disk DATA_A_CD_01_DM01CEL04, DATA_A_CD_02_DM01CEL04, DATA_A_CD_03_DM01CEL04, DATA_A_CD_04_DM01CEL04, DATA_A_CD_05_DM01CEL04, DATA_A_CD_06_DM01CEL04,DATA_A_CD_07_DM01CEL04, DATA_A_CD_08_DM01CEL04, DATA_A_CD_09_DM01CEL04,DATA_A_CD_10_DM01CEL04,DATA_A_CD_11_DM01CEL04,DATA_A_CD_12_DM01CEL04 size=396016M“Â
This operation is immediate and can be done online. After running this on all cells (change the –c parameter and grid disk names accordingly), verify the DATA_A grid disks are indeed larger on all cells:
$ dcli –g cell_group -l root “cellcli -e list griddisk attributes name,offset,sizeÂ
Verify that all free space on the celldisks are allocated to the grid disks:Â
$ dcli -g cell_group -l celladmin “cellcli -e list celldisk attributes name,freespace,freespacemap“
…
dm01cel06: CD_01_dm01cel06Â Â 0
dm01cel06: CD_02_dm01cel06Â Â 0Â
 e. Verify the ASM disk sizes match the grid disk sizes for DATA_AÂ
SELECT name, failgroup, os_mb FROM v$asm_disk
WHERE group_number = (SELECT group_number FROM v$asm_diskgroup
                   WHERE name = ‘DATA_A_DM01’)
ORDER BY failgroup;Â
Note: Executing this query will refresh ASM’s cache and ensure the next step succeeds. Â
12. Resize all DATA ASM disks up to the same size as grid disks in previous stepÂ
ALTER DISKGROUP DATA_A_DM01 RESIZE ALL SIZE 396016M REBALANCE POWER 32 NOWAIT;
 This will trigger a rebalance.
CAUTION: Be sure to set the size for the ASM disks to EXACTLY the same size as the griddisks created and verified in step 11.c ; failure to match the sizes could corrupt the diskgroup.Â
13. Wait for Rebalance to Finish
Ensure no rebalancing operations are occurring by running this query; no rows should return:
SQL> SELECT * FROM gv$asm_operation;Â
When the rebalance is finished, verify that the disk group sizes match what was calculated in step 2:
GROUP_NUMBER NAMEÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â TYPEÂ Â Â Â TOTAL_MBÂ Â Â FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MBÂ Â Â USED_MB
———— ———————————– —— ———- ———- ———————– ————– ———-
… Â Â Â Â Â Â Â Â Â Â 4 DATA_A_DM01Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â NORMALÂ Â 14256576Â Â 14173916Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â 4752192Â Â Â Â Â Â Â 4710862Â Â Â Â Â 82660
          5 RECO_A_DM01                        NORMAL   3146112   3124952                1048704       1038124     21160
We confirm that DATA_A_DM01 disk group grew to 14 TB and RECO shrank to about 3 TB.Â
Â
Discussion ¬