Clusters configured in an active/active role feature cluster members that all serve user requests—none of them is idly waiting on standby. In most setups, the hardware of the cluster members is identical to prevent skewed performance and to facilitate effective load balancing across the cluster. Active/active clusters require more complex management software to operate because access to all resources, such as disk and memory, needs to be synchronized across all nodes. Most often, a private interconnect will be employed as a heartbeat mechanism. The cluster management software will have to detect problems with nodes, such as node failures and intercluster communication problems.

The so-called "split-brain" is a dreaded condition in clustering where the communication between cluster members is interrupted while all the cluster nodes are still running. Adhering to its programming, the cluster software in each of the cluster halves will try to fail over resources of the nodes it considers crashed. The danger of data corruption looms here: if the application communicates with effectively unsynchronized cluster halves, different data can be written to disk. A split brain scenario is a well-known threat to clustering, and vendors of cluster management software have got the right tools to prevent this from happening.

Oracle's cluster management software is referred to as Cluster Ready Services in 10g Release 1 and Clusterware in 10g Release 2. In 11g Release 2, it was finally rebranded as Grid Infrastructure, and it uses the cluster interconnect and a quorum device, called a voting disk, to determine cluster membership. A voting disk is shared by all nodes in the cluster, and its main use comes into play when the interconnect fails. A node is evicted from the cluster if it fails to send heartbeats through the interconnect and the voting disk. A voting disk can also help in cases where a node can't communicate with the other nodes via the interconnect, but still has access to the voting disk. The subcluster elected to survive this scenario will send a node eviction message to the node. Clusterware performs node eviction using the STONITH algorithm. This is short for shoot the other node in the head—a software request is sent to the node to reboot itself. This can be tricky if the node to be rebooted is hung, and responding to a software reset might not be possible. But luckily, the hardware can assist in these cases, and Grid Infrastructure has support for IPMI (intelligent platform management interface), which makes it possible to issue a node termination signal. In the event of a node failure or eviction, the remaining nodes should be able to carry on processing user requests. Software APIs should make the node failure transparent to the application where possible.

An active/passive cluster operates differently from an active/active cluster. Hardware in an active/passive cluster is also identical or nearly identical, but only one of the two nodes is processing user requests at a time. The cluster management software constantly monitors the health of the resource(s) in the cluster. Should a resource fail, the cluster-management layer can try to restart the failed resource a number of times before it fails it over to the standby node.

Depending on the setup, the cluster resource can be located on shared storage or a file system that is also failed over as part of the resource failover. Use of a shared file system offers advantages over the use of an unshared file system, which might have to be checked for corruption through fsck (8) before being remounted to the standby node. Veritas Cluster Suite, Sun (Oracle) Cluster, and IBM HACMP are examples for cluster managers that allow the setup of active/passive clusters.

A little known fact is that using Oracle Grid Infrastructure makes it very simple to set up a cost effective active/passive cluster. Leveraging the Grid Infrastructure API and Oracle's Automatic Storage Management as a cluster logical volume manager makes it easy to constantly monitor a single instance Oracle database. In case of a node failure, the database will automatically be relocated to the standby node. Depending on the fast_start_mttr_target initialization parameter and the size of the recovery set, the failover to the standby node can be very quick; however, users will be disconnected from the database as part of the failover process.

A cluster in which all nodes have concurrent access to shared storage and data is termed a shared-all or shared-everything configuration. Oracle RAC is an example of a shared-everything architecture: a single database located on shared storage is accessed by multiple database instances running on cluster nodes. In the Oracle terminology, an instance refers to the non-persistent memory structures, such as the shared global area (SGA) background- and foreground-user processes. In contrast, the database is the persistent information stored in data files on disk. With RAC, unlike a shared-nothing design, instance failure doesn't translate to a loss of access to the information mastered by that instance. After an instance failure, one of the remaining instances in the cluster will perform instance recovery, which you'll learn more about in Chapter 3, which covers RAC Architecture. If this happens, all remaining instances will continue to be accessible to users. Using high availability technologies such as Fast Connection Failover for connection pools or Transparent Application Failover can mask the instance failure from the user to varying degrees. The failed instance will eventually rejoin the cluster and reassume its share of the workload.

A shared-nothing database cluster is configured as a number of members in a cluster, where each node has its own private, individual storage that is inaccessible to others in the cluster. The database is vertically partitioned between the nodes; result sets of queries are returned as the unioned result sets from the individual nodes. The loss of a single node results in the inability to access the data managed by the failed node; therefore, a shared-nothing cluster is often implemented as a number of individual active/passive or active/active clusters to increase availability. MySQL Cluster is an example for a shared-nothing architecture.

The basic building block of any cluster is the individual cluster node. In Oracle RAC, the number of nodes you can have in your cluster is version dependent. Publicly available documentation states that Oracle 10.2 Clusterware supports 100 nodes in a cluster, while 10.1 Cluster Ready Services supported 63 instances. Even though RAC-based applications continue to be available when individual nodes fail, every effort should be made to ensure that individual components in the database server don't prove to be a single point of failure (SPOF).

Hot swappable components, such as internal disks, fans, and other components should be found on the part list when procuring new hardware. Additionally, power supplies, host bus adapters, network cards, and hard disks should be redundant in the server. Where possible, components should be logically combined, either by hardware in form of RAID controllers or software. Examples of the latter would be software RAID, network bonding, or multipathing to the storage area network. Attention should also be paid to the data center, where the nodes are hosted—you should use an uninterruptible power supply, sufficient cooling, and professional racking of the servers as a matter of course. A remote lights-out management console should also be added to the item list—sometimes a node hangs for whatever reason, and it urgently needs troubleshooting or rebooting.

The cluster interconnect is one of the main features in Oracle RAC. Not only does the interconnect allow the cluster to overcome the limitations by the block pinging algorithm when transferring data blocks from one instance to another, it can also be used as a heartbeat and general communication mechanism. Interconnect failure will result in a cluster reconfiguration to prevent a split-brain situation: one or more cluster nodes will be rebooted by Grid Infrastructure.

It is possible to have a different interconnect for RAC and Grid Infrastructure, in which case you need to configure RAC to use the correct interconnect. The interconnect should always be private—no other network traffic should ever be sent across the wires. Users of RAC can choose from two technologies to implement the interconnect: Ethernet and Infiniband.

Grid Infrastructure is tightly integrated with the operating system and offers the following services to RAC:

Grid Infrastructure is required to run RAC, and it must be installed in its own, non-shared Oracle home, referred to as ORA_CRS_HOME or CRS_HOME. Oracle Grid Infrastructure will contain the binaries to run the Automatic Storage Management component, as well.

Oracle's Cluster software has gone through a number of name changes, as already indicated in other sections of this chapter, which Table 2-1 details. As we explained in this chapter's introduction, one of the purposes of clustering multiple computers is to make them appear as a single entity to the cluster's users. The cluster management software is responsible for doing exactly this, and Oracle Grid Infrastructure is remarkable because it means Oracle development has come up with unified cluster management software for all RAC-supported platforms.

Until Oracle 9i, vendor-specific clusterware was used to provide the same services that Oracle Clusterware provides today. So why did Oracle reinvent the wheel? The foreword by one of the ASM architects in the excellent Oracle Automatic Storage Management: Under-the-Hood & Practical Deployment Guide by Nitin Vengurlekar et al (McGraw Hill, 2007) gives you a hint at how Oracle works internally. The foreword writer noted that Oracle was not satisfied by the cluster file systems available from third-party vendors. There was also the problem that any improvement suggested by Oracle would also benefit Oracle's competitors. Hence, Oracle came up with an entirely new product (ASM) that only the Oracle database could use.

Similar thoughts may have played a role with the creation of unified cluster management software. The use of third-party cluster software is still possible, but it's getting less and less common since Oracle's Grid Infrastructure provides a mature framework. Should you decide to use a third-party cluster solution, then Grid Infrastructure must be installed on top of that. Multi-vendor solutions, especially in the delicate area of clustering, can lead to a situation where none of the vendors takes responsibility for a problem, and each blames the other parties involved for any problems that occur. By using only one vendor, this blame-game problem can be alleviated.

[oracle@node1 ˜]$ crsctl status resource -init -t
------------------------------------------------------------------------------
NAME           TARGET   STATE        SERVER                   STATE_DETAILS
------------------------------------------------------------------------------
Cluster Resources
------------------------------------------------------------------------------
ora.asm
      1        ONLINE   ONLINE       node1            Started
ora.crsd
      1        ONLINE   ONLINE       node1
ora.cssd
      1        ONLINE   ONLINE       node1
ora.cssdmonitor
      1        ONLINE   ONLINE       node1
ora.ctssd
      1        ONLINE   ONLINE       node1            OBSERVER
ora.diskmon
      1        ONLINE   ONLINE       node1
ora.drivers.acfs
      1        ONLINE   ONLINE       node1
ora.evmd
      1        ONLINE   ONLINE       node1

ora.gipcd
      1        ONLINE   ONLINE       node1
ora.gpnpd
      1        ONLINE   ONLINE       node1
ora.mdnsd
      1        ONLINE  ONLINE        node1

Automatic Storage Management was introduced to the RAC software stack as part of Oracle 10g Release 1. Oracle ASM is a cluster-aware logical volume manager for Oracle's physical database structures. Files that can be stored in ASM include control files, database files, and online redo logs. Until 11g Release 2, it was not possible to store any binaries or other type of operating system files—neither was it a suitable option for installing a shared Oracle home.

ASM is built around the following central concepts:

A number of individual ASM disks—either physical hard drives or external storage provided to the database server—form an ASM disk group. There is an analogy with LVM in that ASM disks correspond with physical volumes (see the PV in Figure 2-1). ASM disks sharing a common point of failure such as a disk controller can be grouped in a failure group for which there is no equivalent in LVM. An ASM disk group can be used to store physical database structures: data files, control files, online redo logs and other file types. In contrast to Linux's logical volume manager, LVM2, no logical volumes are created on top of a disk group. Instead, all files belonging to a database are logically grouped into a directory in the disk group. Similarly, a file system is not needed; this explains why ASM has a performance advantage over the classic LVM/file system setup (see Figure 2-1 for a comparison of LVM2 and ASM).

Figure 2.1. A comparison between Linux LVM and Automatic Storage Management

The restriction to store general purpose files has been lifted in Grid Infrastructure, which introduces the ASM Cluster File System (ACFS). We will discuss this in more detail in the "11g Release 2 New Features" section later in this chapter. ASM uses a stripe-and-mirror-everything approach for optimal performance.

The use of ASM and ACFS is not limited to clusters; single-instance Oracle can benefit greatly from it, as well. In fact, with Oracle Restart, the standalone incarnation of Grid Infrastructure will provide ASM as an added benefit—there is no longer any reason to shy away from this technology. Initially met with skepticism by many administrators, this technology has (at least partly) moved responsibility for storage from the storage administrators to the Oracle database administrator.

Technically, Oracle Automatic Storage Management is implemented as a specific type of Oracle instance—it has its own SGA, but no persistent dictionary. In RAC, each cluster node has its own ASM instance—and only one. When starting, each instance will detect the ASM disk group resources in Grid Infrastructure, through an initialization parameter in Clusterware. Each instance will then mount those disk groups. With the correct permissions granted—ASM 11.2 introduced Access Control Lists (ACLs)—databases can access their data files. Using ASM requires using Oracle Managed Files (OMF), which implies a different way of managing database files. Initialization parameters in the RDBMS instance such as db_create_file_dest and db_create_online_dest_n, as well as db_recovery_file_dest to an extent, define which disk group data files, online redo logs/control files and files for the flash recovery area are created. When requesting the creation of a new file, Oracle Managed Files will create a filename based on the following format:

+diskGroupName/dbUniqueName/file_type/file_type_tag.file.incarnation

An example data file name might look like this:

+DATA/dev/datafile/users.293.699134381

ASM is the only supported way of storing the database in Standard Edition RAC. This shows that Oracle is taking ASM seriously. ASM allows you to perform many operations online; as an added benefit, ASM 11.1 and newer can be upgraded in a rolling fashion, minimizing the impact on the database availability.

ASM operates on the raw partition level; LVM2 logical volumes should be avoided for production systems to reduce overhead. ASM is supported over NFS, as well. However, instead of mounting the directories exported by the filer directly, zero-padded files created by the dd utility have to be used as ASM volumes. When using NFS, you should also check with your vendor for best practice documents. We recommend direct NFS over NFS for ASM for reasons of practicality.

Environments with special requirements, such as very large databases with 10TB and more of data, can benefit from customizable extent sizes defined on the disk group level. A common storage optimization technique involves using only the outer regions of disk platters, which offer better performance than the rest of the platter. ASM Intelligent Data Placement allows administrators to define hot regions with greater speed and bandwidth. Frequently accessed files can be specifically placed into these areas for overall improved performance. Hard disk manufacturers are going to ship hard disks with 4k sector sizes soon, increasing storage density in the race for ever faster and larger disks. ASM is prepared for this, and it provides a new attribute for disk groups called sector size that can be set to either 512bytes or 4096bytes.

A typical workflow for most installations begins with the storage administrator presenting the storage to be used for the ASM disk to all cluster nodes. The system administrator creates partitions for the new block devices and adds necessary configuration into the multipathing configuration. Either ASMLib or udev can be used to indicate the new partitioned block device as a candidate disk. After handover to the database team, the Oracle administrator uses the new ASM disk in a disk group. All of these operations can be performed online without having to reboot the server.

With the setup and configuration of Grid Infrastructure completed, it is time to turn our attention towards the clustered Oracle software installation. As we just saw, Grid Infrastructure provides the framework for running RAC, including intercluster communication links, node fencing, node-membership services, and much more. Automatic Storage Management is Oracle's preferred way of storing the database. RAC uses all of these concepts and extends the basic services where necessary. In the following sections, we will discuss installation considerations for RAC; the difference between RAC and single-instance Oracle storage considerations for the RAC database; and finally, Cache Fusion and internal RAC metadata information.

Additional background processes are used in Real Application Clusters for cache synchronization—remember that RAC uses the Cache Fusion architecture to simulate a global SGA across all cluster nodes. Access to blocks in the buffer cache needs to be coordinated for read consistency and write access, and enqueues to shared resources are now global across the cluster. These two main concepts are implemented in the Global Cache Service (GCS), for accessing the common buffer cache; and the Global Enqueue Service (GES), for managing enqueues in the cluster.

Both GCS and GES work transparently to applications. The meta structure used internally is the previously mentioned Global Resource Directory (GRD), which is maintained by the GCS and GES processes. The GRD is distributed across all nodes in the cluster and part of the SGA, which is why the SGA of a RAC database is larger than a single instance's equivalent. Resource management is negotiated by GCS and GES. As a result, a particular resource is managed by exactly one instance, referred to as the resource master. Resource mastering is not static, however. Oracle 9i Release 2 (to a degree) and subsequent versions of Oracle implemented dynamic resource mastering (DRM). In releases prior to Oracle 9i Release 2, resource premastering would only happen during instance failure, when the GRD was reconstructed. Resource mastering in newer releases can happen if Oracle detects that an instance different from the resource master accesses a certain resource more than a given number of times during a particular interval. In this case, the resource will be remastered to the other node. Many users have reported problems with dynamic remastering, which can add undesired overhead if it happens too frequently. Should this happen, DRM can be deactivated.

Figure 2-2 provides an overview of how the GCS and GES background processes work hand-in-hand to maintain the GRD; other RAC background processes have been left out intentionally.

Figure 2.2. The Global Resource Directory in the context of Global Enqueue Services and Global Cache Services

Cache Fusion is the most recent evolution of inter-instance data transfer in Oracle. Instead of using the block ping mechanism implemented through Oracle 8i, Oracle uses a fast interconnect to transfer data blocks for reads and writes across all cluster nodes.

The block ping method of transferring blocks between instances was hugely expensive; at the time, it was strongly advised that you tie workloads to instances to ensure a minimum amount of inter-instance block transfer. In Oracle Parallel Server, when an instance other than the one holding a current block requested the current block for modification, it signaled that request to the holding instance, which in turn wrote the block to disk, signaling that the block could be read. The amount of communication and the write/read operation to disk were hugely undesirable.

Cache Fusion block transfers rely on the Global Resource Directory, and they never require more than three hops, depending on the setup and the number of nodes. Obviously, if there are only two cluster nodes, then there will be a two-way cache transfer. For more than two nodes, a maximum of three hops will be necessary. Oracle has instrumented communication involving the cache with dedicated wait events ending either in two-way or three-way cache transfers, depending on the scenario.

When an instance requests a data block through Cache Fusion, it first contacts the resource master to ascertain the current status of the resource. If the resource is not currently in use, it can be acquired locally by reading from the disk. If the resource is currently in use, then the resource master will request that the holding instance passes the resource to the requesting resource. If the resource is subsequently required for modification by one or more instances, the GRD will be modified to indicate that the resource is held globally. The resource master, requesting, and holding instance can all be different, in which case the maximum of three hops must be used to get the block.

The previously mentioned two-way and three-way block transfers are related to how resources are managed. In case the resource master holds the requested resource, then the request for the block can be satisfied immediately and the block shipped, which is a two-way communication. In a three-way scenario, the requestor, resource master, and holder of the block are different—the resource master needs to forward the request, introducing a new hop.

From this discussion, you can imagine that the effort to coordinate blocks and their images in the global buffer cache is not to be underestimated. In a RAC database, Cache Fusion usually represents both the most significant benefit and the most significant cost. The benefit is that Cache Fusion theoretically allows scale-up, potentially achieving near-linear scalability. However, the additional workload imposed by Cache Fusion can be in the range of 10% to 20%.

One of the main characteristics of the Oracle database is the ability to simultaneously provide different views of data. This characteristic is called multi-version read consistency. Queries will be read consistently; writers won't block readers, and vice versa. Of course, multi-version read consistency also holds true for RAC databases, but a little more work is involved.

The System Change Number is an Oracle internal timestamp that is crucial for read consistency. If the local instance requires a read-consistent version of a block, it contacts the block's resource master to ascertain if a version of the block that has the same SCN, or if a more recent SCN exists in the buffer cache of any remote instance. If such a block exists, then the resource master will send a request to the relevant remote instance to forward a read-consistent version of the block to the local instance. If the remote instance is holding a version of the block at the requested SCN, it sends the block immediately. If the remote instance is holding a newer version of the block, it creates a copy of the block, called a past image; applies undo to the copy to revert it to the correct SCN; and sends it over the interconnect.

System Change Numbers are internal time stamps generated and used by the Oracle database. All events happening in the database are assigned SCNs, and so are transactions. The implementation Oracle uses to allow read consistency relies heavily on SCNs and information in the undo tablespaces to produce read-consistent information. System change numbers needs to be in sync across the cluster. Two different schemes to keep SCNs current on all cluster nodes are used in Real Application Clusters: the broadcast-on-commit scheme and the Lamport scheme.

The broadcast-on-commit scheme is the default scheme in 10g Release 2 and newer; it addresses a known problem with the Lamport scheme. Historically, the Lamport scheme was the default scheme—it promised better scalability as SCN propagation happened as part of other (not necessarily related) cluster communication and not immediately after a commit is issued on a node. This was deemed sufficient in most situations by Oracle, and documents available on My Oracle Support seem to confirm this. However, there was a problem with the Lamport scheme: It was possible for SCNs of a node to lag behind another node's SCNs—especially if there was little messaging activity. The lagging of system change numbers meant that committed transactions on a node were "seen" a little later by the instance lagging behind.

On the other hand, the broadcast-on-commit scheme is a bit more resource intensive. The log writer process LGWR updates the global SCN after every commit and broadcasts it to all other instances. The deprecated max_commit_propagation_delay initialization parameter allowed the database administrator to influence the default behavior in RAC 11.1; the parameter has been removed in Oracle 11.2.

Many new features in 11g Release 2 aim to make maintenance tasks in Oracle RAC simpler, and Grid Plug and Play (GPnP) is no exception. It seems to the authors that another design goal of Oracle software is to provision support for tasks and services that were traditionally performed outside the DBA department. Simplifying the addition of cluster nodes to provide true grid computing where computing power is a utility seems to have also been a contributing factor to the design decisions made with RAC 11.2. There is a big demand in the industry for consolidation. Making maximum use of existing hardware resources has become very important, and RAC is well suited for this role. Grid Plug And Play helps administrators maintaining the cluster. For example, a number of manual steps previously required when adding or removing nodes from a cluster are automated with GPnP.

Grid Plug and Play is not a monolithic concept; rather, it relies on several other new features:

Grid Plug and Play works behind the scenes. You will most likely notice the absence of configuration dialogs when performing cluster maintenance. For example, you won't be prompted for information such as a new node's name or its virtual IP address.

GPnP defines a node's meta data-network interfaces for public and private interconnect, the ASM server parameter file, and CSS voting disks. The profile, an XML file, is protected by a wallet against modification. If you have to manually modify the profile, it must first be unsigned with $GRID_HOME/bin/gpnptool, modified, and then signed again with the same utility. Don't worry, though; the profile is automatically updated without administrator intervention when using cluster management tools such as oifcfg.

The CTSS daemon, part of Grid Infrastructure, synchronizes time between cluster nodes in the absence of a network accessible network time protocol (NTP) server. This could remove the dependency on an NTP server, however we recommend using NTP wherever possible—you might otherwise end up with incorrect (but consistent!) system time on all nodes.

Prior to Oracle 11.2, a node's public and virtual IP addresses had to be registered in a Domain Name Server (DNS) for clients to connect to the database correctly and to support connect time load balancing. Cluster maintenance, such as adding or removing nodes, requires changes in DNS that can be a burden if such maintenance is performed often. The idea behind Grid Naming Service is to move the mapping of IP addresses to names and vice versa out of the DNS service and into Clusterware. Technically, Clusterware runs its own little nameserver, listening on yet another virtual IP address using a method called subdomain delegation. In simple terms, you create a new subdomain (e.g., ebsprod) to your domain (example.com) and instruct the root name server for your domain to hand off all requests for the subdomain (ebsprod.example.com) to GNS. During subsequent steps of the installation, you won't be asked to supply virtual IP addresses and names—only public and private network information has to be supplied. The addresses GNS uses have to come from a dynamic host configuration protocol (DHCP) server on the public network. Table 2-5 gives an overview of addresses and their use, their default names, and details on which part of the software stack is responsible for resolving them. The only dependency on DNS exists during the initial cluster installation, when the GNS virtual address is allocated and assigned in the DNS.

We recommend defining the private IP addresses for the cluster interconnect in the /etc/hosts file on each cluster node; this will prevent anyone or anything else from using them.

You choose to use GNS during the installation of Grid Infrastructure by ticking a small box and assigning a subdomain plus a virtual IP address for GNS. The instruction to DNS to perform subdomain delegation needs to be completed prior to the installation.

The next feature in our list is called Single Client Access Name (SCAN), which we will discuss later. SCAN helps with abstracting the number of nodes from client access. The addition and deletion of nodes is completely transparent—the SCAN relates to the cluster rather than the database.

To minimize additional effort after the addition of a node to the cluster, server pools have been introduced to simplify the addition and removal of database instances to a RAC database. We will discuss server pools next.

Server pools are an interesting new feature. They provide a new way of modeling resources in Clusterware. They allow you to subdivide a cluster into multiple logical subunits that can be useful in shared environments. All nodes in Clusterware 11.2 are either implicitly or explicitly part of a server pool. By default, two pools exist after a fresh installation: the free pool and the generic pool. The generic pool is used for backward compatibility, and it stores pre-11.2 databases or administrator-managed 11.2 databases. The free pool takes all non-assigned nodes.

Server pools are mutually exclusive and take a number of attributes, such as the minimum and maximum number of nodes, an importance, and a name. The importance attribute in a server pool ensures that low priority workloads don't starve out more important ones for resources. It is possible to reallocate servers from one pool to another, which can lead to very interesting scenarios in capacity management. Clusterware can automatically move servers from other server pools to meet an important server pool's minimum size requirement.

Server Pools go hand-in-hand with a new way of managing RAC databases. Prior to Oracle 11.2, administrators were responsible for adding and removing instances from a RAC database, including the creation and activation of public online redo log threads and undo tablespaces. Server pools—and the use of Oracle Managed Files as in ASM—automate these tasks through the use of the form of policy managed databases. Administrator-managed databases are, as the name suggests, managed entirely by the database administrator. In other words, this is the RAC database until to Oracle 11.1. Policy-managed databases use automated features for adding and removing instances and services. The number of nodes a policy-managed database starts is configured by the server pool's cardinality; in other words, if you need another instance, then all you need to do is to assign a new node to the database's server pool, and Oracle will do the rest.

In conjunction with server pools, Grid Infrastructure introduced another feature called Role Separated Management. In shared environments, administrators should be restricted to managing their respective server pool. Access Control Lists are implemented for Clusterware resources, including server pools, to govern access to resources. A new role, the cluster administrator, is introduced. By default, the Grid Infrastructure software owner "grid" and the root user are permanent cluster administrators. Additional operating system accounts can be promoted to cluster administrators, each of which can have a set of permissions on resources, types, and server pools. Separation of duties now seems to be possible on the cluster level—but bear in mind that the grid owner and root users are all powerful.

ASM has been discussed at great length in this chapter in relation to shared storage for the RAC database. Oracle 11.2 extends ASM so it not only stores database-related file structures, but also provides a POSIX compliant cluster file system called ACFS. POSIX compliance means that all the operating system utilities we use with ext3 and other file systems can be used with ACFS. A space efficient read-only, copy-on-write snapshot mechanism is available as well, allowing for up to 63 snapshots to be taken. ASM Cluster File System uses 64-bit structures internally; it is a journal file system, and it uses metadata checksums to ensure data integrity. ACFS can be resized online, and it uses all the services ASM offers, even (most importantly) I/O distribution.

ACFS solves a problem users had in RAC. Database directory objects and external tables can now point to ACFS file systems, allowing a single view on all the external data in the cluster. In the real world, users of external tables had to make sure to connect to the correct instance to access the underlying data in the external table or directory object. With ACFS, it is no longer necessary to connect to a specific node. Presenting the file system to all nodes solves the problem. ACFS also addresses the fact that the use of an additional cluster file system—OCFS2, for example—could cause problems on RAC systems because it meant the existence of a second set of heartbeat processes.

ACFS is not limited to RAC. Another scenario for ACFS is a web server farm protected by the Grid Infrastructure high availability framework that uses ACFS to export the contents of the web site or application code to the server root directories.

ACFS can store database binaries, BFILEs, parameter files (pfiles), trace files, logs, and, of course, any kind of user application data. Database files are explicitly not supported in ACFS, and Oracle won't allow you to create them in an ACFS mount point. However, Oracle does explicitly support installing the Oracle binaries as a shared home in ACFS; the mounting of the file system can be integrated into Clusterware, ensuring that the file system is available before trying to start the database.

ACFS uses the services of the ASM Dynamic Volume Manager (ADVM). It provides volume management services and a standard device driver interface to its file system clients, such as ACFS and ext3. The ACFS file system is created on top of an ADVM volume, which is a specific ASM file type that differentiates it from other ASM managed-file types, such as datafiles, controlfiles, and so on. Figure 2-3 details the relationship between ASM disks, the ASM disk group, a dynamic volume, and ACFS file system:

Figure 2.3. ACFS and ASM dynamic volumes

Management of ACFS and ADVM volumes is tightly integrated into Enterprise Manager, the ASM Configuration Assistant (ASMCA), and command line utilities, as well as into SQL*Plus.

Oracle Restart, or Single Instance High Availability as it was called previously, is the standalone version and counterpart for Grid Infrastructure for RAC. Similar to its big brother, Oracle Restart keeps track of resources such as (single-instance) databases; the ASM instance and its disk groups; and listener and other node applications, such as Oracle Notification Service. It also uses the Oracle High Availability Service (OHAS) and its child processes to monitor the state of registered resources and restarts failed components if necessary. Oracle Restart's meta information is stored in the Oracle Local Registry (OLR), and it also runs a CSSD instance you don't need to create through a call to localconfig add for ASM up to Oracle 11.1. The great thing about Oracle Restart is its integration with ONS and the resource administration through the commands already known from RAC. Finally, database administrators don't need to worry about startup scripts—Oracle Restart will start a database when the server boots. Dependencies defined in Oracle Restart ensure that a database doesn't start before ASM is up, and so on. This means that mistakes such as forgetting to start the listener should be a problem of the past. Managed service providers will appreciate that Oracle Restart is identical across all platforms, thereby providing a uniform way of managing databases and their startup scripts.

Oracle Restart reduces the number of recommended Oracle homes by one—all you need is a Grid Infrastructure home that contains the binaries for ASM and the RDBMS software home. Unlike Grid Infrastructure, Oracle Restart can share the ORACLE_BASE with the database binaries. The installation is very similar to the clustered installation, minus the missing copy to remote locations. Another benefit: It doesn't require the definition of an OCR or voting disk locations. The execution of the root.sh script initializes the OLR and creates the ASM instance with the disk groups specified.

After installing Oracle Restart, you will see the following list of resources in your system:

[oracle@devbox001 ˜]$ crsctl status resource -t
------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
------------------------------------------------------------------------------
Local Resources
------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       devbox001
ora.DATA.dg
               ONLINE  ONLINE       devbox001
ora.REDO.dg
               ONLINE  ONLINE       devbox001
ora.FRA.dg
               ONLINE  ONLINE       devbox001
ora.asm
               ONLINE  ONLINE       devbox001            Started
------------------------------------------------------------------------------
Cluster Resources
------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       devbox001
ora.diskmon
      1        ONLINE  ONLINE       devbox001

As you can see, ASM disk groups are resources, both in clustered and single-instance installations, which makes it far easier to mount and unmount them. Disk groups are automatically registered when first mounted; you no longer need to modify the asm_diskgroups initialization parameter.

Once Oracle Restart is configured, you can add databases to it just as you do in RAC, but with one important difference: no instances need to be registered. Databases can be configured with their associated disk groups, creating a dependency: if the disk group is not mounted when the database starts, Oracle Restart will try to mount the disk group(s) first.

Services are defined through a call to srvctl—again, just as in the RAC counterpart. Do not set the initialization parameter service_names to specify a database's services.

By default, Oracle Restart won't instantiate the Oracle Notification Service. Adding the daemons is useful in conjunction with the Data Guard broker, which can send out FAN events to inform clients of a failover operation. FAN events are also sent for UP and DOWN events, but the lack of high availability in single-instance Oracle limits the usefulness of this approach.

The Single Client Access Name is a new feature that simplifies access to a clustered database. In versions prior to Oracle 11.2, an entry in the tnsnames.ora file for a n-node RAC database always referenced all nodes in the ADDRESS_LIST section, as in the listing that follows:

QA =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (LOAD_BALANCE = ON)
      (FAILOVER = ON)
      (ADDRESS = (PROTOCOL = tcp)(HOST = london1-vip.examle1.com)(PORT =
          1521))
      (ADDRESS = (PROTOCOL = tcp)(HOST = london2-vip.examle1.com)(PORT =
          1521))
      (ADDRESS = (PROTOCOL = tcp)(HOST = london3-vip.examle1.com)(PORT =
          1521))
      (ADDRESS = (PROTOCOL = tcp)(HOST = london4-vip.examle1.com)(PORT =
          1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = qaserv)
    )
  )

Adding and deleting nodes from the cluster required changes in the ADDRESS_LIST. In centrally, well managed environments, this might not be much of a problem; however, for certain environments, where clients are distributed across a farm of application servers, this process might take a while and is error prone. The use of a SCAN address removes this problem—instead of addressing every single node as before, the SCAN virtual IP addresses refer to the cluster. Using the SCAN, the preceding connection entry is greatly simplified. Instead of listing each node's virtual IP address, all we need to do is enter the SCAN scanqacluster.example1.com. Here's the simplified version:

QA =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = tcp)(HOST = scanqacluster.examle.com)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = qaserv)
    )
  )

As a prerequisite to the installation or upgrade of Grid Infrastructure, at least one but preferably three previously unused IP addresses in the same subnet as the public network must be allocated and registered in DNS. Alternatively, in case you decide to use the Grid Naming Service (GNS), the GNS daemon will allocate three IP addresses from the range of addresses offered by the DHCP server. The IP addresses are Address (A) records in DNS that resolve to the SCAN name in a round robin fashion; you also need to make sure a reverse lookup is possible. Oracle Universal Installer will create new entities called SCAN listeners, along with the SCAN virtual IP addresses. The SCAN listeners will register with the local database listeners. A SCAN listener and a SCAN virtual IP address form a resource pair—both will be relocated to a different cluster node if a node fails. In case you ever need to, you can use the server control utility to administer the SCAN listener and IP.

The SCAN listeners are responsible for connect time load balancing, and they will hand off connections to the least loaded node offering the service the client requested. Figure 2-4 shows how these listeners fit into the overall RAC picture.

Figure 2.4. SCAN listeners in the context of Real Application Clusters

Figure 2-4 demonstrates the use of the SCAN. Assuming a three-tier application design and a manual—a non-GNS configuration, in other words—the application server's sessions connect to the SCAN scanname.examle.com on behalf of users. A DNS server is contacted to resolve the SCAN, and it will return one of the three IP addresses defined; this helps when spreading the load to all three SCAN listeners. The SCAN listener in turn will redirect the request to the local listener of the least loaded node, offering the service requested by the client. At this stage, the process is no longer different from the pre-SCAN era: the client resolves the virtual IP address of the node and establishes the connection.

The transition to the use of SCAN addresses is not mandatory—you can continue to use the old connection strings unless you would like to connect to a policy-managed database.