Conceptually, ASM resembles the classical logical volume manager (LVM) we know from Linux, but it can do a lot more and—most importantly—it is cluster aware. Individual physical volumes, called ASM disks, are aggregated into volume groups, referred to as ASM disk groups. In a RAC environment, the ASM disks have to be located on shared storage accessible from all nodes in the cluster. The administrator can define redundancy levels for the disks in a disk group.

Unlike Linux LVM, there are no logical volumes based on a disk group. Instead, the disk group as a whole can be used to create and store database files. A list of supported database files is listed in the next section. Figure 9-1 demonstrates the difference between LVM and ASM:

Figure 9.1. ASM vs. LVM

Note that Figure 9-1 lists partitions as part of the block device. Partitions were mandatory in ASM 10g but are no longer required from 11.1 onwards. We listed partitions here anyway to acknowledge the fact that manual partitioning can make use of the outer part of the spindle for optimal disk performance.

Besides the non-existing logical volume, ASM does not suffer the overhead of a file system layer—all database files reside directly on an ASM disk group. The analogy to raw devices is intentional.

ASM files are organized in extents, which are spread across the disks in the disk group, depending on the redundancy level of the disk group.

The list of supported file types has increased steadily, initially with a focus on database-related files. Oracle ASM's intrinsic file system supports the following file types:

ASM also supports storing the voting disk and OCR. While the OCR location is externalized and visible in the disk group, the voting files are stored internally in ASM disks and cannot directly be modified by the user. The ASM disk headers will specify where a voting file starts and where it ends within the ASM disk.

In Oracle 11.2, the new ASM Clustered Filesystem (ACFS) can manage any file type including shared Oracle database homes, trace files, and external application files. Note that database files are not supported in ACFS. We understand from the ASM development team that this limitation has been imposed primarily for performance reasons. Files stored in ACFS can be accessed using traditional Linux O/S commands such as ls, pwd, and more, since ACFS supports a classic vnode interface. In this aspect, ACFS in our opinion is better than the initial OCFS release, which required special tools for administration and was furthermore plagued by problems.

ASM was originally designed to reduce management overhead. The DBA can perform storage-related tasks directly without involving storage administrators. This does not imply that storage administrators or system administrators become obsolete; they still perform essential tasks in managing a RAC cluster. It does mean, however, that the departments work more closely together. For many sites, ASM can potentially replace third-party Logical Volume Managers. It can also implement redundancy, replacing expensive external RAID storage. Most of the sites, however, still use external redundancy and rely on the storage array to protect from individual disk failures.

However, experience has shown that for many larger users, ASM offers fewer benefits, as the complexity of storage management and the related network infrastructure still requires specialist knowledge. Most users still require storage administrators to manage legacy Oracle databases running on file systems and also to manage non-Oracle storage. In Oracle 11.1 and above, therefore, the roles of the DBA and storage administrator have been separated within ASM, allowing responsibilities to be assigned to different teams or even different organizations if necessary.

ASM simplifies management of large numbers of disks by allowing groups of disks sharing similar characteristics to be assigned to disk groups. In ASM, the basic unit of administration is normally the disk group.

ASM can perform some of the tasks traditionally performed by third-party management procedures such as striping and mirroring, providing increased reliability and performance. By default, ASM implements two-way mirroring to maintain a redundant copy of data providing fault tolerance, but for particularly important data, ASM can also implement three-way mirroring. If the storage layer already provides mirroring capabilities or implements RAID, then ASM can utilize these features instead.

ASM automatically stripes data across all disks in a disk group, providing improved I/O performance. The default stripe size is determined by the class of database file. ASM also performs limited load balancing of files between disks. Rebalancing operations that uniformly distribute file extents that make up a file across all available disks are automatically performed whenever disks are added or dropped from the disks group. Rebalancing is intended to eliminate physical disk hotspots. In Oracle 10.2 and above, rebalancing operations can also be managed manually.

It is important to remember that much of the functionality within ASM is implemented by the RDBMS kernel. Therefore, for example, if you wish to take full advantage of Oracle 11.2 functionality, you would need to upgrade both ASM and the RDBMS to that release; it is not sufficient or possible simply to upgrade the ASM component. You also need to raise the compatibility level of your ASM disk group to make use of more advanced features.

ASM supports all versions of the RDBMS from Oracle 10.1 upwards. The ASM version must be more recent or the same as the RDBMS version. If the version differs between ASM and the RDBMS, then only the older functionality will apply. In Oracle 10.2 and above, ASM supports multiple database versions. For example, Oracle 11.2 ASM can support one database running Oracle 11.2 and another running 10.2 in the same cluster.

Prior to Oracle 11.2, the Clusterware version must also be more recent or the same as the ASM version. In Oracle 11.2, ASM has been integrated into the Grid Infrastructure. It now shares an Oracle Home with Clusterware. Future patch sets will include both components. This also eliminates the discussion about the number of ASM instances per RAC node—with Grid Infrastructure, there will be only one. Grid Infrastructure patches will apply to all components within the Grid Infrastructure home, including ASM, but also ACFS and Clusterware.

In Oracle 10.1, ASM typically shared the RDBMS Oracle Home. In Oracle 10.2 and 11.1, Oracle recommends that ASM was installed in a separate Home to the RDBMS, providing increased flexibility during upgrades. Note that this separation is not mandatory and many users, particularly those running smaller databases, continue to use the same Oracle Home for ASM and RDBMS. For systems with limited memory, combining the ASM and RDBMS Homes significantly reduces memory usage, as only one version of the oracle executable will be loaded. In Oracle 11.2 and above, the separation of ASM and RDBMS is enforced by the new Grid Infrastructure architecture. But contrary to the earlier releases, Grid Infrastructure includes the cluster software stack as well as ASM in one single Oracle home.

In order to use ASM in a RAC cluster, a separate ASM instance must be running on each node. The ASM instance maintains metadata in the ASM disk groups and supports with the database instances. The ASM instance must be started before other database instances can access files located in ASM storage. If the ASM instance is terminated, all client database instances will also be terminated. The ASM instance also handles adding and dropping disks and rebalancing operations,

The database instance communicates with the ASM instance to obtain information about files stored in ASM. However, the ASM instance does not perform I/O directly for applications. Application I/O is still performed by the server processes and background processes associated with the database instances. The ASM instance performs I/O only during rebalancing operations.

The ASM instance is a stripped-down version of a standard instance that is now referred to as an RDBMS instance. The instance type is defined by the INSTANCE_TYPE that takes the values RDBMS (default) or ASM.

You can connect to a running ASM instance and query it using dynamic performance (V$ and X$) views. Unlike a database instance, an ASM instance does not mount or open the database. Consequently, it does not have access to a data dictionary—the ASM instance does not have persistent information stored, and no data files of its own. Therefore it does not support queries against DBA/ALL/USER views.

In Oracle 10g, connections to the ASM instance required the SYSOPER or SYSDBA privilege. In Oracle 11.1, the SYSASM privilege was introduced, allowing separator groups of storage administrators and DBAs to be defined. In Oracle 11.2, you must have SYSASM privilege to perform most administrative tasks in ASM. SYSOPER privilege is required to start up and shut down the ASM instance, to mount, dismount, check and offline disks groups, and to access ASM dynamic performance views. Users with SYSASM privilege can also perform these tasks and in addition can create disk groups and add or drop disks.

[oracle@ondon1 ˜]$ crsctl stat res ora.asm -p
NAME=ora.asm
TYPE=ora.asm.type
[...]
GEN_USR_ORA_INST_NAME@SERVERNAME(london1)=+ASM1
GEN_USR_ORA_INST_NAME@SERVERNAME(london2)=+ASM2

SQL> select component,current_size,min_size,max_size
  2   from V$MEMORY_DYNAMIC_COMPONENTS
  3  where current_size <> 0
  4  /

COMPONENT                           CURRENT_SIZE   MIN_SIZE   MAX_SIZE
----------------------------------- ------------ ---------- ----------
shared pool                            142606336  142606336  142606336
large pool                               4194304    4194304    4194304
SGA Target                             180355072  180355072  180355072
PGA Target                             104857600  104857600  104857600
ASM Buffer Cache                        25165824   25165824   25165824

SQL> select name,description
  2  from v$bgprocess
  3  where paddr <> '00';

NAME  DESCRIPTION
----- ----------------------------------------------------------------
PMON  process cleanup
VKTM  Virtual Keeper of TiMe process
GEN0  generic0
DIAG  diagnosibility process
PING  interconnect latency measurement
PSP0  process spawner 0
DIA0  diagnosibility process 0
LMON  global enqueue service monitor
LMD0  global enqueue service daemon 0
LMS0  global cache service process 0
LMHB  lm heartbeat monitor
MMAN  Memory Manager
DBW0  db writer process 0
LGWR  Redo etc.
LCK0  Lock Process 0
CKPT  checkpoint
SMON  System Monitor Process
RBAL  ASM Rebalance master
ASMB  ASM Background
GMON  diskgroup monitor
VBG0  Volume BG 0
VBG1  Volume BG 1
VBG2  Volume BG 2
VDBG  Volume Driver BG
VMB0  Volume Membership 0
ACFS  ACFS CSS
MMON  Manageability Monitor Process

MMNL  Manageability Monitor Process 2

28 rows selected.

ASMCMD> ls -l
Type              Redund  Striped  Time             Sys  Name
ASMPARAMETERFILE  HIGH    COARSE   APR 27 00:00:00  Y    REGISTRY.253.717382259

If a disk controller fails, then all disks that are connected to it will be inaccessible. ASM defines failure groups, which are groups of disks that are dependent on a single point of failure, such as a controller. To ensure redundancy, each mirror should be located on a different failure group. In the event of a controller failure, ASM can, therefore, guarantee that it can reconstruct the failed disk group from mirrored copies of extents in remaining disk groups.

A special type of failure group has been introduced in ASM 11.2, and it's called a quorum failure group. This type of failure group is used in the context of extended distance clusters, when the voting files are deployed in ASM. A quorum failure group does not contain user data, and it does not count for disk group redundancy requirements.

With stretched clusters, we need at least three voting files, one on each site plus a third that is normally deployed via NFS. Unfortunately, the OUI doesn't allow such a complex setup, so when configuring the extended distance cluster, you should start with a normal configuration—two voting disks on site A and one for site B—and add the third voting disk after the installation finishes.

Begin by exporting a file system on the NFS appliance, containing a zero-padded file to serve as the voting disk later. The file should be the same size as the ASM disks already used to store the voting files. This NFS export should then be concurrently mounted by the two sites. Continue by adding the new zero-padded file as an ASM disk, marked as a quorum failure group—for this you have to change the disk discovery string. Once the new disk is added to the disk group, ASM will automatically readjust the voting files and include the NFS "disk."

Oracle 11.1 introduced a new feature called "PREFERRED_READ_FAILURE_GROUP" to speed up access to storage in extended distance clusters. In stretched-RAC deployments with ASM, each local storage array forms its own failure group. This way, ASM can be used for providing redundancy. We said earlier that ASM always uses the primary extent when dealing with ASM files, which could potentially be on the remote site. Oracle 10g did not offer any possible way of instructing ASM to read the local extent instead. With ASM 11.1, this has changed for the better. Setting the COMPATIBLE.RDBMS parameter at the ASM disk group level to at least 11.1, and the ASM instances' initialization parameter PREFERRED_READ_FAILURE_GROUPS to the respective local site, ASM will try to prefer the local site over the remote site, potentially reducing latency.

Files created within ASM follow the naming standard set by the Oracle Managed File feature first introduced on Oracle 9i. This offers the significant advantage of not having to specifying file names and accidentally storing data files in the wrong directory. But it comes at a price—the administrator loses control over file names.

+DiskGroupName/DBUniqueName/Filetype/FiletypeTag.file#.incarnation

SQL> create tablespace users datafile size 1g extent management local
 2   segment space management auto;

Tablespace created.

SQL> create tablespace users datafile '+DATA' size 1g extent management local
 2   segment space management auto;

Tablespace created.

SQL> alter database datafile '+DATA/PROD/DATAFILE/users.259.679156903' resize 10G;

SQL> create tablespace users datafile '+DATA/PROD/myDataFile.dat' size 1G;

Tablespace created.

SQL> select redundancy,stripe,name,primary_region,mirror_region
 2  from v$asm_template
  3  where group_number = 3
  4  /

REDUND STRIPE NAME                           PRIM MIRR
------ ------ ------------------------------ ---- ----
UNPROT COARSE PARAMETERFILE                  COLD COLD
UNPROT COARSE ASMPARAMETERFILE               COLD COLD
UNPROT COARSE ASMPARAMETERBAKFILE            COLD COLD
UNPROT COARSE DUMPSET                        COLD COLD
UNPROT FINE   CONTROLFILE                    COLD COLD
UNPROT COARSE FLASHFILE                      COLD COLD
UNPROT COARSE ARCHIVELOG                     COLD COLD
UNPROT COARSE ONLINELOG                      COLD COLD
UNPROT COARSE DATAFILE                       COLD COLD
UNPROT COARSE TEMPFILE                       COLD COLD
UNPROT COARSE BACKUPSET                      COLD COLD
UNPROT COARSE AUTOBACKUP                     COLD COLD
UNPROT COARSE XTRANSPORT                     COLD COLD
UNPROT COARSE CHANGETRACKING                 COLD COLD
UNPROT COARSE FLASHBACK                      COLD COLD
UNPROT COARSE DATAGUARDCONFIG                COLD COLD
UNPROT COARSE OCRFILE                        COLD COLD
UNPROT COARSE OCRBACKUP                      COLD COLD
UNPROT COARSE ASM_STALE                      COLD COLD

19 rows selected.

SQL> alter diskgroup data
  2  add template allhot attributes (hot);

REDUND STRIPE NAME                           PRIM MIRR S
------ ------ ------------------------------ ---- ---- -
UNPROT COARSE ALLHOT                         HOT  COLD N

SQL> create tablespace hottbs datafile '+DATA(allhot)' size 10M;

SQL> select bytes,type,redundancy,primary_region,mirror_region,
  2  hot_reads,hot_writes,cold_reads,cold_writes
  3* from v$asm_file where file_number = 284
SQL> /

     BYTES TYPE       REDUND PRIM MIRR  HOT_READS HOT_WRITES COLD_READS COLD_WRITES
---------- ---------- ------ ---- ---- ---------- ---------- ---------- -----------
  10493952 DATAFILE   UNPROT HOT  COLD          4        127          0           0

ASM supports three types of redundancy for ASM disk groups:

From our experience, the majority of ASM deployments make use of the external redundancy setting. Most enterprises use storage arrays with internal protection against disk failure. If a disk fails within a disk group in the array, the storage administrator is usually tasked with replacing the failed disk long before the ASM administrator even notices that an individual disk has been lost. Among the very few users of normal redundancy disk groups is CERN. The Centre for Nuclear Research has decided very early to leverage the power of ASM's mirroring to save on the cost of expensive storage arrays. Their setup has been in production for some time, and performance and user experience are excellent.

ASM implements striping of files across the disks within a disk group to optimize I/O performance. Therefore every disk within the group should have the same type and performance characteristics. Two types of striping—coarse and fine—are implemented depending on the database file type.

Coarse striping is used for most file types, including database files, transportable tablespaces, backup sets, dump sets, autobackup control files, change tracking files, Data Guard configuration files, and some others. The coarse stripe size depends on the size of the allocation unit. When setting the disk group compatibility to 11.1 for RDBMS and ASM, variable extent sizes can be used for coarse striping. The first 20,000 extents always equal the allocation unit (AU) size. The next 20,000 extents are 4 times the size of the AU, and the extents thereafter will use 16 times the AU size.

By default, fine striping is used only for control files, online redo logs, and flashback logs. The stripe size is 128KB. Also by default, eight stripes are created for each file; therefore, the optimum number of disks in a disk group is a multiple of eight.

ASM uses mirroring to provide data redundancy, and it mirrors extents at the file level. This differs from most operating system mirroring, which is performed at disk level. If an ASM disk is lost, the extents mirrored on other ASM disks can be used to continue operations without data loss or interruption to service. In the event of a disk failure, ASM can reconstruct the failed extents using the mirrored extents from other disks in the same group. The use of striping means that the I/O required to reconstruct the new disk is spread evenly across the remaining disks in the group. If the disk error is transient, and the failed disk can be brought online again within a time threshold, the Fast Mirror Resync feature can help speed up the resynchronization operation. The requirement is to have a disk group with COMPATIBLE.RDBMS set to ≥ 11.1, and optionally a DISK_REPAIR_TIME. A so-called Staleness Registry keeps track of changes to the AUs within the offline disk. When the disk is brought online again, the Staleness Registry is used to recover the contents of the disk. If a disk cannot be repaired in time, it will automatically be dropped from the disk group, resulting in a full rebalance operation. This was the default behavior in 10g. Another rebalance operation occurred when the disk was eventually repaired and added back into the disk group.

Read errors are not a severe problem with mirrored ASM extents. Remember that ASM by default always reads the primary extent first. In case of a read error on the primary extent, ASM tries to read its mirrored extent. In high redundancy disk groups, it also tries to read the second mirror copy. Only if none of the mirrored copies are readable does ASM bail out; all other failures are merely recorded in the ASM instance's alert.log.

Write errors are more severe. A write operation issued by the ASM client is successful only if at least one extent copy can be written to an ASM disk. Additionally, the disk containing the primary extent will be taken offline, and the RDBMS instance has to acknowledge that fact for the write to complete successfully. If none of the mirrored extents can be written, the RDBMS instance has to take action and offlines the data file. The ASM instance, upon receiving a write error notification from the RDBMS, checks whether to offline the ASM disk only, or even the whole disk group. Once the disk is taken offline (or dropped, depending on COMPATIBLE.RDBMS), the algorithm is the same as in the foregoing read error example.

For high performance storage architecture, it has always been recommended to partition individual hard disks into an outer and an inner part. Data in the outer part of the spindle is accessed faster and generally speaking offers best performance.

Intelligent Data Placement is a new 11.2 feature, and it allows the administrator to divide the ASM disk into a hot and cold region. The outmost tracks of a disk are referred to as the hot region, and the innermost tracks make up the cold region. The Intelligent Data Placement feature requires the disk group's RDBMS and ASM compatible flag to be set to 11.2, otherwise the options discussed here won't be available.

Intelligent Data Placement is unlikely to provide any benefit for disk groups filled to less than a quarter—the management overhead is not likely to outweigh the benefit offered by specifying a special disk region. Furthermore, the feature does not seem to apply to LUNs provided by a storage array, as the disk geometry is masked from ASM. We see the greatest potential for IDP with deployments where every disk in the storage backend is mapped to a LUN and presented to the database server.

By default, all files are created in the COLD region of the disk, as shown in the PRIMARY_REGION and MIRROR_REGION columns in the V$ASM_TEMPLATE view. Additional views with information about data placement are V$ASM_DISK, V$ASM_DISK_IOSTAT, V$ASM_FILE and the aforementioned V$ASM_TEMPLATE views. You can either use templates or make use of the alter diskgroup modify file command to change the placement policy. We discussed the use of templates in the section "Understanding Extents, Allocation Units, Types, and Templates" earlier in this chapter.

ASM can optionally be instructed to grant access to the content of a disk group based on explicitly assigned access control lists. These ACLs limit the power of connections using the SYSDBA privilege, i.e., RDBMS instances in most cases.

Oracle 11.1 began separating super user access to ASM from super user access to the RDBMS instance. In 10g the SYSDBA role was the all-powerful role, both for ASM and the RDBMS. With 11.1 Oracle introduced a new privilege, called SYSASM. The SYSDBA privilege previously used still exists, but it has been redesigned.

[root@london1 ˜]# id -a orafin
uid=503(orafin) gid=502(oinstall) groups=502(oinstall),504(asmdba),505(dbafin) context=root:system_r:unconfined_t:SystemLow-SystemHigh

[root@london1 ˜]# usermod -G asmdba,dbareport orareport

SQL> ALTER DISKGROUP DATA set attribute 'compatible.asm' = '11.2';

Diskgroup altered.

SQL> ALTER DISKGROUP DATA set attribute 'compatible.rdbms' = '11.2';

Diskgroup altered.

SQL> ALTER DISKGROUP DATA set attribute 'access_control.enabled' = 'TRUE';

Diskgroup altered.

SQL  select name, permissions,user_number,usergroup_number
  2  from v$asm_file f natural join v$asm_alias t
  3  where group_number = 3
SQL> /

NAME                           PERMISSIONS      USER_NUMBER USERGROUP_NUMBER
------------------------------ ---------------- ----------- ----------------
SYSTEM.256.722359017           rw-rw-rw-                  0                0
SYSAUX.257.722359017           rw-rw-rw-                  0                0
TEMP.278.723575013             rw-rw-rw-                  0                0
[...]
spfile.283.723575205           rw-rw-rw-                  0                0
spfilePROD.ora                 rw-rw-rw-                  0                0
HOTTBS.284.725562071           rw-------                  1                0

31 rows selected.

SQL> select group_number,user_number,os_id,os_name
  2  from v$asm_user;

GROUP_NUMBER USER_NUMBER OS_ID      OS_NAME
------------ ----------- ---------- ----------
           3           1 503        orafin

[root@london1 ˜]# id -a orafin
uid=503(orafin) gid=502(oinstall) groups=502(oinstall),504(asmdba),505(dbafin) context=root:system_r:unconfined_t:SystemLow-SystemHigh

SQL> alter diskgroup data set ownership
  2  owner = 'orareport'  --case sensitive!
  3  for file '+DATA/PROD/DATAFILE/users.259.679156903';

Diskgroup altered.

SQL> select 'alter diskgroup data set ownership owner = ''orareport'' '||
  2  'for file ''' || name || ''';' from v$datafile;

SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/fin/spfilefin.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/fin/spfilefin.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATA/fin/spfilefin.ora
ORA-17503: ksfdopn:2 Failed to open file +DATA/fin/spfilefin.ora

ORA-15260: permission denied on ASM disk group

ORA-06512: at line 4
SQL> exit

ASMCMD> cd +data/fin
ASMCMD> ls --permission
User    Group  Permission  Name
                           spfilefin.ora =>
   +DATA/FIN/PARAMETERFILE/SPFILE.270.726353479
ASMCMD> ls --permission  +DATA/FIN/PARAMETERFILE/SPFILE.270.726353479
User    Group  Permission  Name
oracle          rw-------  SPFILE.270.726353479

SQL> select 'alter diskgroup data set permission owner = read write, ' ||
  2  'group = read only, other = none' ||
  3  ' for file ''' || name || ''';' from v$datafile;

SQL> alter diskgroup data set permission owner = read write,
  2  group = read only, other = none
  3  for file '+DATA/FIN/datafile/example.279.723575017';

Diskgroup altered.

SQL> select name, permissions,user_number,usergroup_number
  2  from v$asm_file f natural join v$asm_alias t
  3  where group_number = 3
  4 and name = 'EXAMPLE.279.723575017'
  5 /

NAME                   PERMISSIONS      USER_NUMBER USERGROUP_NUMBER
---------------------- ---------------- ----------- ----------------
EXAMPLE.279.723575017  rw-r-----                  1                0

Disk groups can be created using asmca, asmcmd, or Enterprise Manager. Disk groups can also be created manually using the CREATE DISKGROUP command in SQL*Plus when connected to the ASM instance, as in the following example:

SQL> create diskgroup DATA
> normal redundancy
> failgroup controllerA disk 'ORCL:DATA01', 'ORCL:DATA02', 'ORCL:DATA03', 'ORCL:DATA04'
> failgroup controllerB disk 'ORCL:DATA05', 'ORCL:DATA06', 'ORCL:DATA07', 'ORCL:DATA08';

Diskgroup created.

The redundancy clause in the preceding example specifies which level of redundancy should be provided by ASM. If no FAILGROUP clause is specified, then each disk will be in its own failure group. In this example, two failgroups were explicitly created for two storage controllers. Since you are using a server parameter file in ASM 11.2, the disk group name will be added automatically to the ASM_DISKGROUPS parameter. Clusterware will also detect a new ASM disk group and add it as a resource.

Oracle 11.1 added attributes to the ASM disk group, which were extended in Oracle 11.2. At the time of this writing, the following attributes could be set.

The ASM compatibility flag sets which features are available with a disk group. It also specifies the minimum version of the ASM to access the disk group. By default, the ASM compatibility is set to 10.1 when a disk group is created, except for asmca, which defaults to 11.2. Many features of Grid Infrastructure such as ACFS volumes and storing the OCR and voting disk in ASM require the COMPATIBLE.ASM flag to be 11.2.

The database instance's "compatible" parameter has to be at least the same as or higher than the RDBMS compatibility flag of the disk group. It defaults to 10.1.

The final compatible flag, COMPATIBLE.ADVM, is new to ASM 11.2 and defines whether a disk group can store an ASM Dynamic Volume. COMPATIBLE.ADVM requires COMPATIBLE.ASM to be set to 11.2 as well. By default the value for COMPATIBLE.ADVM is blank.

The compatible attributes just described can be set either during the creation of the disk group or at a later stage. The CREATE DISKGROUP and ALTER DISKGROUP commands take the settings in the attribute clause, as shown in this example:

SQL> ALTER DISKGROUP ocrvote SET ATTRIBUTE 'compatible.asm' = '11.2';

Diskgroup altered.

SQL> ALTER DISKGROUP ocrvote SET ATTRIBUTE 'compatible.advm' = '11.2';

Diskgroup altered.

Table 9-2 maps features against minimum compatible settings.

Beginning with Oracle 11.1, the administrator has the option to specify a larger allocation unit than the default size of 1M. This was possible in 10.2 as well but required setting an underscore parameter. The larger a file gets, and a big file tablespace can grow to considerable size, the more extents have to be managed by the ASM instance. Increasing the ASM AU size, which is responsible for the extent size, can be greatly beneficial. The AU is set as an attribute to the disk group, and it can be set only during disk group creation. Use the attribute "AU_SIZE" to specify any of the following allocation unit sizes in the range of 1M to 64M; increments are to be in powers of 2. Oracle has also extended the maximum file sizes with ASM 11.1. If your compatibility level is set to greater than version 11.1, then file sizes can range up to the petabyte scale.

ASM takes into account that new hard disks are manufactured now with support for larger than the previously default sector size of 512 bytes. If all disks specified in the create diskgroup statement support the new sector size of 4k, then the disk group can make use of it. To enable larger sector sizes, use the attribute SECTOR_SIZE, which can take either the default value of 512 or, alternatively, 4096/4k. You should not use 4k sector size when you intend to create an ADVM volume on the disk, as there is a performance penalty associated with this.

The DISK_REPAIR_TIME attribute has been introduced in Oracle 11.1 to prevent transient disk failures to cause a disk to be dropped after it goes offline. In Oracle 10.2 a transient disk failure caused the disk to be dropped, followed by a rebalance operation. The algorithm employed in Oracle 10.2 made this a costly operation, and the amount of data moved during a rebalance operation usually is a multiple of the failed disk size.

With Oracle 11.1 and later, a transient disk failure still causes a disk to go offline, but if the failure is rectified within the DISK_REPAIR_TIME, the disk can be brought online again. ASM keeps track of the changes made to the offline disk, and can quickly rebuild it when it's brought online again. Use the "alter diskgroup diskgroupName online disk diskName" command to bring a disk online again.

Other new features include ACCESS.CONTROL_ENABLED and ACCESS_CONTROL.UMASK. These two are related to ASM access control and are discussed in the section "Access Control", earlier in this chapter.

Once the CREATE DISKGROUP command is initiated, ASM will automatically mount the newly created disk group. You can query the dynamic performance view GV$ASM_DISKGROUP to check the status of the new ASM disk group on all cluster nodes.

The disk group attributes are visible in the V$ASM_ATTRIBUTE view, for example:

SQL> select name,value from v$asm_attribute where group_number=3
  2  and name not like 'template%'
  3> /

NAME                                               VALUE
-------------------------------------------------- ------------------------------
disk_repair_time                                   3.6h
au_size                                            1048576
sector_size                                        512
compatible.asm                                     11.2.0.0.0
compatible.rdbms                                   11.2
cell.smart_scan_capable                            FALSE
access_control.enabled                             TRUE
access_control.umask                               066

8 rows selected.

Output not printed here is related to the ASM file templates.

Even with the most careful planning, there might be the need to extend an ASM disk group. Extending an ASM disk group is a relatively simple task once the underlying block devices have been presented to all cluster nodes. When adding a new LUN to ASM, it is recommended to use performance and size characteristics identical (or as close as possible to) to the existing ASM disks.

The device-mapper-multipath utility we discussed in Chapter 6 does not format the logical device in /dev/mapper, but rather requires the administrator to format the underlying block device, usually /dev/sd*. This operation should be performed on only one node of the cluster. With the device partitioned, the administrator can use either partprobe or kpartx to re-read the partition table on the other cluster nodes. A restart of the multipath daemon ("service multipathd restart") should show the new partition in /dev/mapper. We have seen cases where it was necessary to flush the unpartitioned multipathed device temporarily using multipath -f, followed by a reload of the multipath daemon. Alternatively, a cluster reboot will certainly detect the new partition tables.

With the new block devices detected on all nodes, you could use ASMLib to mark the disk as an ASM disk on one node of the cluster. On the remaining nodes, issue /etc/init.d/oracleasm scandisks to detect the new disk. With the operation completed, connect to the ASM instance as SYSASM and add the new disk to the disk group as in the following example, where the ASMLib disk "DATA11" is added to the disk group:

[oracle@london1 ˜]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 13 11:48:21 2010

Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup DATA add disk 'ORCL:DATA11' rebalance power 6 nowait;

Diskgroup altered.

The command prompt will return almost immediately, triggering a rebalance operation in the background. You can check V$ASM_OPERATION for an estimate of how long the rebalance will take. Alternatively, a script to add a disk to a disk group could specify WAIT instead of NOWAIT, with the effect that the control is not returned to the script until the rebalance operation finished.

In some rare occasions, it is necessary to drop a disk from a disk group. The only times we have seen the need to drop a disk were during a SAN migration (more on that later in this section) or when an ASM disk with different, non-matching performance characteristics had to be removed from a disk group.

Dropping a disk triggers a rebalance operation. It is safe to remove the ASM disk physically from the cluster only when the HEADER_STATUS of the V$ASM_DISK view shows "FORMER" for the disk you dropped. Use the alter diskgroup command to drop a disk from a disk group:

SQL> alter diskgroup data drop disk 'DATA11' rebalance power 3;

Diskgroup altered.

Control is returned to the user almost immediately, and the rebalance operation starts in the background. Again, check V$ASM_OPERATION for information about a time remaining estimate.

SQL> alter diskgroup data add disk 'ORCL:NEWSAN01', 'ORCL:NEWSAN02', 'ORCL:NEWSAN03',
drop disk 'ORCL:OLDSAN01', 'ORCL:NEWSAN02', 'ORCL:NEWSAN03' rebalance power 11;

Diskgroup altered.

We discussed how to make ASM disks available to the operating system in Chapter 6. Depending on which method you choose to provide the ASM disks, you may need to instruct ASM to look at different places for disks.

The most important parameter during disk discovery is the initialization parameter "ASM_DISKSTRING". When using ASMLib, you do not need to modify it at all—the ASMLib-provided disks are automatically discovered.

The asm_diskstring variable needs to be adjusted to reflect the location of the ASM candidate disks if you are still using block or raw devices. Similarly, NFS volumes require zero padded files to be used as ASM disks, in which case the asm_diskstring should point to the mountpoint of the ASM disks.

ASM places a lot of meta-information into each ASM disk's header. The header information can be made visible using the kfed utility. In versions prior to 11.1, kfed had to be linked manually, but manual linking is no longer necessary. To show the ASM header, use the kfed utility as shown here:

[oracle@london1 ˜]$ kfed read /dev/oracleasm/disks/OCRVOTE1

You should adjust your path if you are not using ASMLib to provide disks to your ASM instance. When executing the command, you should consider diverting the output to another location for inspection. Once saved, you can view it with any editor you like. In the header, you can find a lot of interesting information that is also externalized through the various V$-views. In the dump of my OCRVOTE1 disk, I found the following useful information, summarized in Table 9-3.

Although kfed can also be used to write a modified header back into the disk header, this should be done only under the explicit request and supervision of Oracle Support.

When installing Grid Infrastructure under a different operating system account than the RDBMS, you need to take some extra care when configuring ASMLib. Recall from Chapter 6 that prior to using ASMLib you have to configure it. Consider the following example, where the Grid Infrastructure software is installed by using the operating system account "grid." The operating system user "grid" has the primary group "oinstall," and secondary groups "asmowner" and "asmdba." In this case, you should configure ASMLib as shown here:

[root@london1 grid]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on-boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmdba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

Any other setting is likely to provoke a failure during the creation of the ASM instance and associated disk groups during the execution of the root.sh script.

The ASM metadata area is re-created at the start of each ASM file and contains details of all the extents contained in the file. If you need to re-create the ASM instance on disks that have previously been used for ASM, then you will need to reinitialize the ASM metadata area using the dd utility. For example, you might use the following commands to initialize the disks after having very carefully verified that these are indeed the disks you intend to erase:

[rootglondonl ˜] # dd if=/dev/zero of=/dev/mapper/mpath0p1 bs=4096 count=l
[root@londonl ˜] # dd if=/dev/zero of=/dev/mapper/mpath1p1 bs=4096 count=l
[root@londonl ˜] # dd if=/dev/zero of=/dev/mapper/mpath2p1 bs=4096 count=1

Note that this command applies to shared storage, making the disks wiped by dd unusable across all cluster nodes! Please be extra careful not to wipe out the wrong disks. Once the metadata has been removed from the ASM disk, it can be unpresented from the host and reused. After the storage has been unpresented from the node, it is recommended to remove the devices from the SCSI bus.

The easiest way to create an ASMCFS is to use the ASM Configuration Assistant asmca. We recommend creating a separate ASM disk group for the ADVM volume, although this is optional.

Start the asmca utility from the $GRID_HOME/bin directory to begin the ACFS creation. As with all GUI tools, you need to have access to an X11 server to run it—refer back to Chapter 7 for more information about starting the graphical user tools. Once the initial screen completes loading, click on the "Create" button to begin the creation of a new disk group on the "Disk Groups" tab. Figure 9-2 shows the "Create Disk Group" configuration dialog.

Figure 9.2. Create Disk Group configuration dialog

The important information in Figure 9-2 is the new disk group name, which should take a user-friendly, non-cryptic name, and the redundancy level plus the ASM disks to form the disk group. You should choose the redundancy level based on your storage array. You also need to set the ASM and ADVM compatibility to 11.2.0.0.0 at least. The disk group attributes are accessible as part of the advanced options. Define failure groups if appropriate; in our example we didn't need to. Click OK to finish the disk group creation. The command will then execute in the background, and if all went well, you are rewarded with a success message, as shown in Figure 9-3.

Figure 9.3. Disk Group created successfully

The new disk group can then be used to store an ADVM volume. On the main screen of the asmca tool, click on the "volume" tab. You are shown the screen as in Figure 9-4.

Figure 9.4. Volumes configuration screen

Click on the "Create" button to bring up the "Create Volume" dialog. This dialog is shown in Figure 9-5.

Figure 9.5. Volumes creation screen

We placed the new ADVM volume on the disk group ACFSDG we just created, assigned the name "apachevol," and created it with 1G of available space. Unlike classical LVM, the volume we create will not need to be partitioned. It can be dynamically resized when needed, which is a one-step operation, i.e., thanks to the absence of a classic ext3 or ext4 file system, no calls to resize2fs are needed.

The advanced options shown in the lower half of the configuration screen allow you to fine-tune the new ADVM volume. You can optionally specify a stripe width in the range of 4k to 1M, and also define the number of columns in the stripe set. The range of columns specifiable is 1 to 8, and defaults to 4. If you have a disk group with normal redundancy and would like to specify a different redundancy level than the one of the disk group, make sure to set it accordingly. After you confirm the creation of the volume by clicking "OK," asmca creates the volume and displays it in the overview screen. This is shown in Figure 9-6.

Figure 9.6. Volumes apachevol created succesfully

The new volume is enabled by default and can be used for the next step. In its current state, the volume needs an ACFS file system. Select the "ASM Cluster File Systems" tab to proceed with its creation. Figure 9-7 shows the ACFS creation dialog.

Figure 9.7. ACFS creation

The user has two options in this dialog: either specify the ACFS home as a shared home for an Oracle RDBMS installation, for which approx 4.6 GB are required, or create a general purpose file system. At the time of this writing, we do not recommend storing the RDBMS binaries in the ACFS home.

When creating a general purpose file system, you have the option to specify a mount point and to register that mountpoint. If you choose not to register the mount point, you can always do so at a later stage. Registering the mountpoint offers the advantage that the file system is mounted automatically when Grid Infrastructure starts. The ACFS will be mounted on all cluster nodes. Clicking OK initiates the creation of the ACFS volume and optionally registers it on all cluster nodes. The final screen in asmca with the file system created is shown in Figure 9-8.

Figure 9.8. ACFS creation successfully completed

As you can see from the output, the ACFS volume is mounted with its registered mount point on all cluster nodes.

The previous example showed how to create an ACFS volume and how to mount it automatically on all nodes of the cluster. The same procedure can also be completed by using the command line. The example in this section walks you through the creation of the same ACFS, but this time from the command line.

First of all, a new disk group is to be created. This can be done in SQL*Plus, for example. Connect to your local ASM instance as SYSDBA, as shown here:

[oracle@london1 ˜]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on Wed Jul 14 21:10:54 2010

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Automatic Storage Management option

After connecting, issue the CREATE DISKGROUP command to create a new group:

SQL> CREATE DISKGROUP ACFSDG normal REDUNDANCY
  2   DISK 'ORCL:ACFS1', 'ORCL:ACFS2', 'ORCL:ACFS3', 'ORCL:ACFS4';

Diskgroup created.

Next, issue some ALTER DISKGROUP commands to set the two compatibility parameters:

SQL> ALTER DISKGROUP ACFSDG SET ATTRIBUTE 'compatible.asm'='11.2',;

Diskgroup altered.

SQL> ALTER DISKGROUP ACFSDG SET ATTRIBUTE 'compatible.advm='11.2';

Diskgroup altered.

You should mount the newly created disk group on all cluster nodes. You can use the srvctl start diskgroup command to do so:

[oracle@london1 ˜]$ srvctl start diskgroup -g ACFSDG

Once the group is mounted on all nodes, you can use either SQL*Plus or asmcmd to create the new ADVM volume. For the sake of simplicity, we performed this step in SQL*Plus again:

SQL> alter diskgroup acfsdg add volume apachevol size 1g
  2  /

Diskgroup altered.

The add volume clause to the alter diskgroup command can also take additional parameters, exactly the same way as the asmca if the disk group uses normal redundancy. Consider this example:

SQL> ALTER DISKGROUP ACFSDG add volume apachevol size 1G
  2  mirror stripe_width 128k stripe_columns 4;

Diskgroup altered.

The dynamic performance view V$ASM_VOLUME then displays information about the newly created volume:

SQL> select volume_name,state,usage,volume_device,mountpath
  2  from v$asm_volume
  3  /

VOLUME_NAME     STATE    USAGE VOLUME_DEVICE         MOUNTPATH
--------------- -------- ----- --------------------- --------------------
APACHEVOL       ENABLED        /dev/asm/apachevol-444

Future commands require the volume device, so it's a good idea to note it down somewhere.

As an alternative to issuing the "alter diskgroup... add volume" command in SQL*Plus, you could use the volcreate command in asmcmd, as the following example demonstrates:

[oracle@london1 ˜]$ asmcmd
ASMCMD>  volcreate -G ACFSDG -s 1G apachevol

If you don't see an error message, the command succeeded. You can verify success with the volinfo command:

ASMCMD> volinfo -G ACFSDG -a
Diskgroup Name: ACFSDG

         Volume Name: APACHEVOL
         Volume Device: /dev/asm/apachevol-444
         State: ENABLED
         Size (MB): 1024
         Resize Unit (MB): 256
         Redundancy: MIRROR
         Stripe Columns: 4
         Stripe Width (K): 128
         Usage:
         Mountpath:

The next step is to create an ACFS file system on the volume we just created. The following command creates that file system. Notice the reference to the volume device in the command:

[oracle@london1 ˜]$ /sbin/mkfs -t acfs /dev/asm/apachevol-444
mkfs.acfs: version                   = 11.2.0.1.0.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/apachevol-444
mkfs.acfs: volume size               = 1073741824
mkfs.acfs: Format complete.

Before we can mount the file system, a mount point is needed. Ensure that you create the mount point on every node of the cluster where you like to mount the file system. The following commands create a mountpoint "acfsdg_apachevol" under $ORACLE_BASE/acfsmounts. The "mkdir" commands need to be executed on each node.

[oracle@london1 ˜]$ mkdir $ORACLE_BASE/acfsmounts
[oracle@london1 ˜]$ mkdir $ORACLE_BASE/acfsmounts/acfsdg_apachevol

Mount the file system as root:

[root@london1 ˜]# mount -t acfs /dev/asm/apachevol-444 
>  /u01/app/oracle/acfsmounts/acfsdg_apachevol/

Optionally, register the ACFS with the mount registry:

[oracle@london1 ˜]$ /sbin/acfsutil registry -a /dev/asm/apachevol-444 
> /u01/app/oracle/acfsmounts/apachevol/

acfsutil registry: mount point /u01/app/oracle/acfsmounts/acfsdg_apachevol successfully
added to Oracle Registry

At the end of the ACFS creation, the file system is created:

[oracle@london1 ˜]$ /sbin/acfsutil info fs
/u01/app/oracle/acfsmounts/acfsdg_apachevol
    ACFS Version: 11.2.0.1.0.0
    flags:        MountPoint,Available
    mount time:   Wed Jul 28 16:20:02 2010
    volumes:      1
    total size:   1073741824
    total free:   998002688
    primary volume: /dev/asm/apachevol-444
        label:
        flags:                 Primary,Available,ADVM
        on-disk version:       39.0
        allocation unit:       4096
        major, minor:          252, 227329
        size:                  1073741824
        free:                  998002688
        ADVM diskgroup         ACFSDG
        ADVM resize increment: 268435456
        ADVM redundancy:       mirror
        ADVM stripe columns:   4
        ADVM stripe width:     131072
    number of snapshots:  0
    snapshot space usage: 0

[oracle@london1 ˜]$ df -h /u01/app/oracle/acfsmounts/acfsdg_apachevol/
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/apachevol-444
                      1.0G   39M  986M   4% /u01/.../acfsdg_apachevol

From this point onwards, you can use the ACFS like any other Linux file system.

A number of maintenance operations can be performed with each of the ACFS components. Resizing a file system is trivial, and the new file system size is changed instantaneously on all cluster nodes—for example:

[root@london1 ˜]# acfsutil size +256M /u01/app/oracle/acfsmounts/tomcatvol
acfsutil size: new file system size: 536870912 (512MB)

You can see the new size of the volume straightaway:

[root@london1 ˜]# df -h /u01/app/oracle/acfsmounts/tomcatvol
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/tomcatvol-32

512M   73M  440M  15% /u01/app/oracle/acfsmounts/tomcatvol
[root@london1 ˜]# ssh london2  df -h /u01/app/oracle/acfsmounts/tomcatvol
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/tomcatvol-32
                      512M   73M  440M  15% /u01/app/oracle/acfsmounts/tomcatvol

Mounting and unmounting an ACFS are identical to any other Linux file system:

/bin/mount -t acfs -rw /dev/asm/deviceName mountPoint

Unmounting the file system is even easier:

/bin/umount mountPoint

Snapshots can be taken using the acfsutil command, as in the following example:

[root@london1 ˜]#  mkdir /u01/app/oracle/acfsmounts/snap

[root@london1 ˜]# acfsutil snap create prerelease 
> /u01/app/oracle/acfsmounts/acfsdg_apachevol/                                                        acfsutil snap create: Snapshot operation is complete.

Using the acfsutil command, you can verify that the snapshot was indeed taken:

[root@london1 ˜]# acfsutil info fs
/u01/app/oracle/acfsmounts/acfsdg_apachevol
    ACFS Version: 11.2.0.1.0.0
    flags:        MountPoint,Available
    mount time:   Wed Jul 28 16:20:02 2010
    volumes:      1
    total size:   1073741824
    total free:   930897920
    primary volume: /dev/asm/apachevol-444
        label:
        flags:                 Primary,Available,ADVM
        on-disk version:       39.0
        allocation unit:       4096
        major, minor:          252, 227329
        size:                  1073741824
        free:                  930897920
        ADVM diskgroup         ACFSDG
        ADVM resize increment: 268435456
        ADVM redundancy:       mirror
        ADVM stripe columns:   4
        ADVM stripe width:     131072
    number of snapshots:  1
    snapshot space usage: 49152

The snapshots don't have to be mounted by the administrator; they are available under the mountpoint's ".ACFS/snaps/" directory (note the dot in .ACFS). Each snapshot gets its own name:

[root@london1 ˜]# ls -l /u01/app/oracle/acfsmounts/acfsdg_apachevol/.ACFS/snaps/
total 4
drwxrwx--- 4 root dba 4096 Jul 28 18:17 prerelease

To delete a snapshot, use the acfsutil snap delete <snap_name> <mountpoint> command.

Unlike Grid Infrastructure for clusters, Oracle Restart does not have an orarootagent process owned by root in the 11.2 base release. Since loading kernel modules and mounting file systems require root access in Linux, Oracle Restart cannot perform the tasks normally performed by the agent in a cluster. If you are using ASM Cluster File systems in your Oracle Restart environment, you need to add the following code to /etc/rc.local for Red Hat–based systems, which should load the needed kernel modules during the boot process.

# modification to automatically load the ACFS kernel modules
# and to automatically mount registered file systems
# change ORACLE_SID to match the ASM SID for your cluster node!
ORACLE_SID=+ASM1; export ORACLE_SID
ORAENV_ASK=NO;    export ORAENV_ASK

source /usr/local/bin/oraenv

$ORACLE_HOME/bin/acfsload start -s
sleep 15
mount -t acfs acfsVolumeName fileSystemMountPoint

The kernel modules are required for the subsequent mount operation to succeed. We added a 15-second delay to allow the kernel to load the drivers before mounting the file system.

The SQL*Plus command interpreter has always been available for administering ASM. In Oracle 10.1, when the asmcmd utility was not available, the use of SQL*Plus was mandatory for many tasks.

The CREATE, ALTER, and DROP DISKGROUP commands allow you to create and manage ASM disk groups and their contents. A number of ASM-specific dynamic performance views are available and populated in the ASM instance.

[oracle@london1 ˜]$ . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base for ORACLE_HOME=/u01/app/crs is /u01/app/oracle

SQL> select * from v$pwfile_users;

USERNAME                       SYSDB SYSOP SYSAS
------------------------------ ----- ----- -----
SYS                            TRUE  TRUE  TRUE
ASMSNMP                        TRUE  FALSE FALSE

SQL> shutdown immediate
ORA-15097: cannot SHUTDOWN ASM instance with connected client

SQL> select db_name,status,instance_name from v$asm_client

DB_NAME  STATUS       INSTANCE_NAME
-------- ------------ ------------------------------
+ASM     CONNECTED    +ASM2

[oracle@london2 ˜]$ srvctl stop diskgroup -g DATA -n london2
[oracle@london2 ˜]$ srvctl stop diskgroup -g OCRVOTE -n london2

SQL> select name,state from v$asm_diskgroup

NAME                           STATE
------------------------------ -----------
DATA                           DISMOUNTED
OCRVOTE                        QUIESCING

SQL> alter diskgroup DATA check;

We recommend the use of srvctl over other command line tools. Users of Oracle 11.1 and earlier will be surprised that it is not possible to entirely stop ASM using srvctl. In pre–Oracle 11.2, the command srvctl stop asm -n nodeName stopped an ASM instance. The same command in Oracle 11.2 produces the following error:

[oracle@london2 ˜]$ srvctl stop asm -n london2
PRCR-1014 : Failed to stop resource ora.asm
PRCR-1065 : Failed to stop resource ora.asm
CRS-2529: Unable to act on 'ora.asm' because that would require stopping or
 relocating 'ora.registry.acfs', but the force option was not specified

The force option forces the command to execute, but some components won't stop, due to too many dependencies on other resources. The only option to completely stop ASM is to stop the Clusterware stack on the local node:

[root@london1 ˜]# crsctl stop crs

In Oracle 11.1 and earlier, administrators used to create a dependency between local ASM and database instance. This ensured that the ASM instance was started together with the RDBMS instance when this was required. In Oracle 11.2, the dependency has been redefined. Instead of linking to the ASM instance, the dependency is created with one or more ASM disk groups. Consider the following example of database PROD (unneeded output not printed for clarity):

[oracle@london1 ˜]$ crsctl status resource ora.prod.db -p
NAME=ora.prod.db
TYPE=ora.database.type
ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r—
[...]
SPFILE=+DATA/prod/spfileprod.ora
START_DEPENDENCIES=hard(ora.DATA.dg) [...] pullup(ora.DATA.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg)

You can see that a start dependency has been created for the disk group DATA, which will also be started if needed as part of the database start. Similarly, a stop dependency exists for the same disk group. If you have multiple disk groups defined for the database, you will see additional dependencies for each dependent disk group.

We said earlier in the chapter that ASM disk groups are now resources within Grid Infrastructure. So instead of mounting the disk group via the "alter diskgroup mount all" commands, you should use the following syntax to mount a disk group in the cluster:

[oracle@london1 ˜]$ srvctl start diskgroup -g diskgroupName

Alternatively the "-n" switch allows you to specify which host the disk group should be mounted on.

In early versions of Oracle, ASM files within an ASM instance could not easily be accessed, and it could appear as a black box. This changed for the better with the introduction of asmcmd in Oracle 10.2. Alternatively, Enterprise Manager can be used to view files in ASM.

The ASM command line utility has been significantly enhanced with every release. In Oracle 11.2, it has reached a very mature state and supports many file-related commands. The asmcmd utility resides in $GRID_HOME/bin; therefore you need to point your $ORACLE_HOME and $ORCLE_SID accordingly. Start the utility by invoking $GRID_HOME/bin/asmcmd.

Navigation around the file structure within ASM follows the same technique you would use to navigate a standard Linux filesystem:

Oracle Enterprise Manager also offers a way to navigate around in ASM.

You can create an ASM file system based on operating system files instead of devices on Linux. You should not attempt this configuration on a production system except when on a supported NAS appliance, but it can be useful for testing or educational reasons. The following steps have been successfully tested on a RAC database but have limited applicability for RAC, as only one instance has access to the ASM disks that are created on the local disks of one node. The main focus is on single instance ASM.

Create a directory to contain the ASM files:

[root@london1 ˜]# mkdir /asm

Create the ASM files using the dd command:

[root@london1 ˜]# dd if=/dev/zero of=/asm/vol4 bs=1M count=1024

This command will create a file of 1GB in size. As the root user, associate a loop device with each file using the loset up command:

[root@london1 ˜]# losetup /dev/loopl /asm/vol4

You can check if the loop device is properly presented to the host:

[root@london1 ˜]# losetup /dev/loopl

Now use your favorite tool to mark this as an ASM disk; the ASMLib command createdisk can be used for this purpose, as shown in this example:

[root@london1 ˜]# /etc/init.d/oracleasm createdisk VOL4 /dev/loop1

From this point on, you can make use of this file in the ASM instance, using your favorite tool to create a disk group on it. The command line option is shown here:

[oracle@london1 ˜]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on Wed Jul 28 18:23:13 2010

Copyright (c) 1982, 2009, Oracle. All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> create diskgroup FILEDG external redundancy disk 'ORCL:VOL4';

Diskgroup created.

Oracle VM and other Xen-based virtualization software can mark block devices in the guest domain as shared storage. The simple solution is to use the graphical user interface, but the command line also allows the administrator to add shared storage to a running virtual cluster. The following example assumes that the virtual shared block device is a logical volume on the dom0. First create the logical volume from your volume group. The example volume group is named "data_vg" on the dom0.

[root@dom0 ˜]# lvcreate --name virtcluster_112_asm_data01 --size 10G data_vg

Then add the logical volume to the running cluster, comprised of virtcluster_112_node1 and virtcluster_112_node2:

[root@dom0 ˜]# xm block-attach virtcluster_112_node1 
>  phy:/dev/data_vg/virtcluster_112_asm_data01 xvdd w!
[root@dom0 ˜]# xm block-attach virtcluster_112_node2 
>  phy:/dev/data_vg/virtcluster_112_asm_data01 xvdd w!

The foregoing command xm block-attach instructs the hypervisor to add a new block device called /dev/data_vg/virtcluster_112_asm_data01 in the backend (dom0) to the virtual machines virtcluster_112_node1 and 2 as frontend device xvdd. The new block device is presented in read-write mode, and is sharable across the cluster.

Once the command completed, the virtual machines feature a new shared block device, /dev/xvdd. You can use fdisk on any of the nodes to create a partition on the device, and either udev or ASMLib commands to make it an ASM disk.