The Oracle Cluster Registry (OCR) is used to store cluster configuration information. The OCR contains information about the resources controlled by Oracle Clusterware, including the following:

Logically, the OCR represents a tree structure; physically, each element of data is stored in a separate 4096 byte physical block.

The data in the OCR is essential to the operation of the cluster. It is possible to back up the contents of the OCR online, but restoring this data will almost always result in an interruption of service. Since Oracle 10gR2, it has been possible to configure two OCR mirrors that are automatically maintained by Oracle Clusterware. This helps ensure against media failure or human error. In Oracle 11gR2 and later, it is possible to configure up to five mirrored copies of the OCR. If there is more than one OCR mirror, then it is possible to replace a failed OCR mirror without an outage.

In 11gR2, the OCR can be stored in an ASM disk group or a cluster file system. The OCR can only be stored on raw devices or block devices if the cluster has been upgraded from an earlier version of Oracle. Also, Oracle has announced that raw devices and block devices are deprecated for Oracle 12—it is a good idea to migrate the OCR (and voting disks for that matter) off block devices and to either a supported cluster file system such as GFS or OCFS2 or follow the best practice to move these files into ASM.

The OCR should only be updated by Oracle Clusterware processes, Enterprise Manager, supported utilities such as crsctl, srvctl, ocrconfig; and configuration tools such as the OUI, dbca, and netca.

The Oracle Local Registry is the OCR's local counterpart and a new feature introduced with Grid Infrastructure. The information stored in the OLR is needed by the Oracle High Availability Services daemon (OHASD) to start; this includes data about GPnP wallets, Clusterware configuration, and version information. Comparing the OCR with the OLR reveals that the OLR has far fewer keys; for example, ocrdump reported 704 different keys for the OCR vs. 526 keys for the OLR on our installation.

If you compare only the keys again, you will notice that the majority of keys in the OLR deal with the OHASD process, whereas the majority of keys in the OCR deal with the CRSD. This confirms what we said earlier: you need the OLR (along with the GPnP profile) to start the High Availability Services stack. In contrast, the OCR is used extensively by CRSD. The OLR is maintained by the same command-line utilities as the OCR, with the appended -local option. Interestingly, the OLR is automatically backed up during an upgrade to Grid Infrastructure, whereas the OCR is not.

The voting disk is used to provide fencing and to determine cluster-node membership. During normal operations, the OCSSD daemon on each node in the cluster updates the voting disk once a second with the current status of that node. It then reads back the status structures of all other nodes in the cluster. In the event of an interconnect failure, all nodes in the cluster attempt to place a lock in the voting disk. If a node can lock a majority of the voting disks, then it gains control of the cluster.

In 11gR2, the voting disk can be stored in an ASM disk group or a cluster file system. As with the OCR, the voting disk can only be stored on raw devices or block devices if the cluster has been upgraded from an earlier version of Oracle. In any configuration, there should always be an odd number of voting disks. In the event of an interconnect failure in a two node-cluster, this ensures that one node always secures a majority of voting disks. For clusters containing three or more nodes, a more complex algorithm is used to determine which node ultimately controls the cluster.

If ASM is used to store the voting disks, then Oracle Clusterware automatically performs the mirroring. All copies are stored in the same ASM disk group. The number of mirrored voting disks copies stored in the disk group is dependent on the redundancy of the disk group (see Table 8-1).

Within the ASM disk group, each voting disk must be stored in a separate failure group on physically separate storage. By default, each disk within an ASM disk group is a separate failure group. This translates into the requirement that, for normal redundancy, you need to define at least three failure groups. For high redundancy, at least five failure groups must use ASM for storing the voting disks. Also, the disk group attribute must be set to compatible.asm=11.2.

If a cluster file system is used to store the voting disks, then Oracle recommends that at least three copies be maintained on physically separate storage. This approach eliminates the possibility of a single point of failure. In Oracle 11gR2, up to 15 voting disk copies can be stored. However, Oracle recommends configuring no more than five copies.

The GPnP profile is an important part of the new 11.2 Grid Infrastructure, and it records a lot of important information about the cluster itself. The file is signed to prevent modifications, and administrators generally should not edit it by administrators. The profile is an XML document, which is the main reason why adding nodes requires a lot less input from the administrator. Here is a sample profile, shortened for readability:

<?xml version="1.0" encoding="UTF-8"?>
<gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile"
   xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile"
   xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd"
   ProfileSequence="3" ClusterUId="002c207a7175cf38bffcea7bea5b3a49"
   ClusterName="ocfs2" PALocation="">
  <gpnp:Network-Profile>
    <gpnp:HostNetwork id="gen" HostName="*">
      <gpnp:Network id="net1" IP="172.17.1.0" Adapter="bond0" Use="public"/>
      <gpnp:Network id="net2" IP="192.168.1.0" Adapter="bond1"
        Use="cluster_interconnect"/>
    </gpnp:HostNetwork>
  </gpnp:Network-Profile>
  <orcl:CSS-Profile id="css"
   DiscoveryString="/u03/oradata/grid/vdsk1,/u03/oradata/grid/vdsk2,/u03/oradata/grid/vdsk3"
   Lease Duration="400"/>
  <orcl:ASM-Profile id="asm"
   DiscoveryString="++no-value-at-profile-creation--never-updated-through-ASM++"
   SPFile=""/>
  <ds:Signature...>...</ds:Signature>
</gpnp:GPnP-Profile>

The preceding profile can be obtained by invoking gpnptool with the get option. This dumps the XML file into standard output. For better readability, we saved this to a local PC and opened it with Firefox, which displays the XML a lot more nicely. The cluster this profile is taken from uses OCFS2 for storing the voting disk and OCR.

We have also removed the signature information from the profile because it didn't provide information needed for this discussion. The example illustrates that the cluster has a unique identifier, and that the name is also recorded. The next item in the profile is the network profile, an important piece of information. Oracle uses the GPnP profile and the information stored in the OCR when adding a node to the cluster. The fact that the network interfaces are stored in the GPnP profile allows the administrator to specify less information on the command line. This is also a convenient place to look up which network interface is used for what purpose; here, the Use attribute on the Network tag gives the interface's purpose away.

Another important piece of information is recorded in the CSS-Profile tag, which records the location of the CSS voting disks. In the preceding example, there are three copies of the voting disks on a cluster file system /u03 in directory /u03/oradata/grid/. Because the voting disks are stored outside of ASM, the ASM information is not populated. Unless your database uses ASM specifically, no ASM instance will be brought online by Grid Infrastructure because it simply is not needed. If ASM is used, the relevant tags <orcl:CSS-Profile/> and <orcl:ASM-Profile/> will look something like the following:

<orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400" />
<orcl:ASM-Profile id="asm" DiscoveryString=""
   SPFile="+DATA/prod/asmparameterfile/registry.253.699915959" />

The only supported case where the file should be modified is when the ASM instance's spfile is corrupted and must be changed.

Oracle re-architected Grid Infrastructure into two different stacks. The official documentation refers to them as the High Availability Services stack and the Cluster Ready Services stack. Other sources divide the software stack into the lower and upper stack. We will stick to Oracle's terminology in this chapter, to minimize confusion. Figure 8-1 provides an overview of the processes and their position in the stack.

Figure 8.1. The Grid Infrastructure software stack and process dependencies

The High Availability Services stack consists of daemons that communicate with their peers on the other nodes. As soon as the High Availability Services stack is up, the cluster node can join the cluster and use the shared components (e.g., the OCR). The startup sequence of the High Availability Services stack is stored partly in the Grid Plug and Play profile, but that sequence also depends on information stored in the OLR.

Next, we will take a closer look at each of these processes in the context of their position in the software stack, beginning with the lower part of the software stack.

The high availability stack is based on the Oracle High Availability Services (OHAS) daemon. In previous versions of Oracle Clusterware, three processes were started by the operating system init process: CRSD, OCSSD, and EVMD. In Oracle 11gR2 and later, the only process under direct control of the init process is the Oracle High Availability Services (OHAS) daemon. This daemon is responsible for starting all other Oracle Clusterware processes. It is also responsible for managing and maintaining the OLR. In a cluster, the OHAS daemon runs as the root user; in an Oracle Restart environment, it runs as the oracle user.

In Unix systems, the Oracle High Availability Services Stack is managed by the init process. On Linux, the following entry is appended to /etc/inittab on each node in the cluster during Grid Infrastructure installation:

h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

In previous releases, entries were added to /etc/inittab for the CRS daemon, CSS daemon, and EVM daemon. These background processes are now indirectly controlled by the OHAS daemon. The advantage of having the OHAS daemon do this is that the administrator can now issue cluster-wide commands. The OHAS daemon will start even if Grid Infrastructure is disabled explicitly. Indeed, you can use this to restart the cluster layer in a single command. In Grid Infrastructure 11.2, OHAS is comprised of the following daemons and services:

The Cluster Ready Services stack builds on the services provided by the High Availability Services stack and requires that its services to be up and running. It is hinged on the Cluster Ready Services daemon. The following daemons and services are part of the Cluster Ready Services stack:

In Oracle 11gR2 and later, there are two new types of agent processes: the Oracle Agent and the Oracle Root Agent. These processes interface between Oracle Clusterware and managed resources. In previous versions of Oracle Clusterware, this functionality was provided by the RACG family of scripts and processes.

To slightly complicate matters, there are two sets of Oracle Agents and Oracle Root Agents, one for the High Availability Services stack and one for the Cluster Ready Services stack.

The Oracle Agent and Oracle Root Agent that belong to the High Availability Services stack are started by ohasd daemon. The Oracle Agent and Oracle Root Agent pertaining to the Cluster Ready Services stack are started by the crsd daemon. In systems where the Grid Infrastructure installation is not owned by Oracle—and this is-probably the majority of installations—there is a third Oracle Agent created as part of the Cluster Ready Services stack. Similarly, the Oracle Agent spawned by OHAS is owned by the Grid Infrastructure software owner.

In addition to these two processes, there are agents responsible for managing and monitoring the CSS daemon, called CSSDMONITOR and CSSDAGENT. CSSDAGENT, the agent process responsible for spawning CSSD is created by the OHAS daemon. CSSDMONITOR, which monitors CSSD and the overall node health (jointly with the CSSDAGENT), is also spawned by OHAS.

You might wonder how CSSD, which is required to start the clustered ASM instance, can be started if voting disks are stored in ASM? This sounds like a chicken-and-egg problem: without access to the voting disks there is no CSS, hence the node cannot join the cluster. But without being part of the cluster, CSSD cannot start the ASM instance. To solve this problem the ASM disk headers have new metadata in 11.2: you can use kfed to read the header of an ASM disk containing a voting disk. The kfdhdb.vfstart and kfdhdb.vfend fields tell CSS where to find the voting file. This does not require the ASM instance to be up. Once the voting disks are located, CSS can access them and joins the cluster.

The high availability stack's Oracle Agent runs as the owner of the Grid Infrastructure stack in a clustered environment, as either the oracle or grid users. It is spawned by OHAS directly as part of the cluster startup sequence, and it is responsible for starting resources that do not require root privileges. The list of processes Oracle Agent starts includes the following:

The Oracle Root Agent that is spawned by OHAS in turn starts all daemons that require root privileges to perform their programmed tasks. Such tasks include the following:

Once CRS is started, it will create another Oracle Agent and Oracle Root Agent. If Grid Infrastructure is owned by the grid account, a second Oracle Agent is created. The grid Oracle Agent(s) will be responsible for:

There can be a maximum of three SCAN listeners in the cluster at any given time. If you have more than three nodes, then you can end up without a SCAN listener on a node. Likewise, in the extreme example where there is only one node in the cluster, you could end up with three SCAN listeners on that node.

The oracle Oracle Agent will only spawn the database resource if account separation is used. If not—i.e., if you didn't install Grid Infrastructure with a different user than the RDBMS binaries—then the oracle Oracle Agent will also perform the tasks listed previously with the grid Oracle Agent.

The Oracle Root Agent finally will create the following background processes:

The functionality provided by the Oracle Agent process in Oracle 11gR2 was provided by the racgmain and racgimon background processes in Oracle 11gR1 and earlier.

The startup sequence in Grid Infrastructure is not completely documented in the Oracle Clusterware Administration Guide 11.2. Therefore, this section will elaborate on some of the more elaborate aspects of managing that sequence.

The init process, father of all processes in Linux, spawns OHAS. This occurs in two stages: first, the /etc/init.d/init.ohasd script is invoked with the run argument. Second, this script then calls the /etc/init.d/ohasd script, which starts $GRID_HOME/bin/ohasd.bin. The init scripts log potential problems using the syslog facility present on all Linux systems. The ohasd.bin executable uses $GRID_HOME/log/hostname/ohasd/ohasd.log to report its activities. All processes mentioned so far run with root privileges. If the administrator has disabled the Grid Infrastructure, then the Grid Infrastructure high availability stack must be started manually; otherwise, the startup sequence continues.

The ohasd.bin process then spawns four processes, all of which are located in $GRID_HOME/bin. Note that that Grid Infrastructure can be installed to an operating system account other than oracle. This chapter refers to the owner of the software stack as the Grid software owner. This chapter also assumes that the RAC binaries were installed as the oracle user. Finally, this chapter assumes that $LOG_HOME points to $ORACLE_HOME/log/hostname

It is important to remember that these processes are created by ohasd.bin, rather than the CRS daemon process, which has not been created yet. Next, the Oracle Root Agent starts the following executables, which are also located in $GRID_HOME/bin:

The Oracle Agent (started as the Grid software owner) in turn will start these executables:

The cssdagent executable is responsible for starting ocssd.bin, which runs as the Grid software owner. The cssdagent executable doesn't spawn additional processes.

Once the CRS daemon is created by the OHAS's Oracle Root Agent, the Cluster Ready Services stack will be started. The following actions depend on CRS to create additional Oracle Agents (owned by the Grid software owner and oracle) and another Oracle Root Agent. Again, it is important to note the distinction between these agents and the ones created by OHAS. You will also see that they are different because their log files are located in $LOG_HOME/agent/crsd/ rather than $LOG_HOME/agent/ohasd. You will see the following processes spawned by crsd.bin:

These agents are henceforth responsible for continuing the start process. The Grid software owner's Oracle Agent (oraagent.bin) will start the following infrastructure components, all as the Grid Software owner:

The ONS binary is an exception because it is not started from $GRID_HOME/bin, but from $GRID_HOME/opmn/bin. The enhanced ONS service is a Java process. Therefore, it doesn't start from $GRID_HOME/bin, either; however, its JAR files are located there.

The oracle Oracle Agent oraagent.bin will start the database and services associated with the database resource, as defined in the Oracle Cluster Registry.

The Oracle Root Agent will start the following resources:

Both Enterprise Manager Database Control and Enterprise Manager Grid Control can be used to manage Oracle Clusterware environments. The functionality of Enterprise Manager Database Control is restricted to managing a single database that may have multiple instances. If Enterprise Manager Database Control is deployed, then the management repository must be stored in the target database. Enterprise Manager Database Control is often configured in test systems; it is less frequently deployed in production environments.

Enterprise Manager Grid Control provides a much more flexible management solution, and many Oracle sites now use this tool to manage their entire Oracle estate. The Enterprise Manager Grid Control management repository can be stored in a separate database, outside the cluster. Enterprise Manager Grid Control supports a wider range of administrative tasks, such as the ability to configure and maintain Data Guard.

If you do not already use Enterprise Manager Grid Control to manage your database environment, then we strongly recommend that you investigate this tool, which is subject to ongoing development. At the time of writing, a new version of the tool with a more user-friendly interface was planned for Oracle 11gR2.

The Clusterware Control Utility crsctl is the primary command-line tool for managing Oracle Clusterware. It has existed since Oracle 10gR1; however, its functionality has been significantly enhanced in Oracle 11gR2.

In Oracle 11gR2, crsctl has been extended to include cluster-aware commands that can be used to start and stop Clusterware on some or all nodes in the cluster. It can also be used to monitor and manage the configuration of the voting disks and to configure and manage individual cluster resources. The crsctl utility also supports new functionality, such as the configuration of administrative privileges to ensure role separation.

As you saw in the startup section, the OHAS daemon always starts when the system boots. If the Oracle High Availability Services stack is disabled, then none of the daemons for the High Availability Services stack will start, and that node cannot join the cluster. To manually start and stop Oracle Clusterware on all nodes in the cluster, execute the following commands as the root user:

[root@london1]# crsctl start cluster -all
[root@london1]# crsctl stop cluster -all

Alternatively, you could use the -n switch to start Grid Infrastructure on a specific (not local) node. To check the current status of all nodes in the cluster, execute the following command:

[root@london1]# crsctl check cluster -all
**************************************************************
london1:
CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
london2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

The documentation suggests that crsctl start cluster and crsctl start crs are identical when executed on the same node. We found that this is not the case. The start cluster option successfully starts any daemon that has not yet started. In other words, in cases where all daemons but the CRS daemon have started, crsctl start crs fails with an error that states: "High Availability Services is already active." When invoked with the start cluster option, crsctl detected that crsd was not started, and it brought it back online.

In Oracle 11gR2 and later, the crs_stat utility has been deprecated. However, this utility is still shipped to provide backwards compatibility. The functionality of crs_stat has been integrated into the crsctl utility. You can use crsctl status resource -t to list the current status of all resources, as in the following example:

[root@london1]# crsctl status resource -t
----------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
----------------------------------------------------------------------------
ora.ACFS1.dg
               ONLINE  ONLINE       london1
               ONLINE  ONLINE       london2
               ONLINE  ONLINE       london3
ora.DATA.dg
               ONLINE  ONLINE       london1
               ONLINE  ONLINE       london2
               ONLINE  ONLINE       london3
ora.LISTENER.lsnr
               ONLINE  ONLINE       london1
               ONLINE  ONLINE       london2
               ONLINE  ONLINE       london3
ora.RECO.dg
               ONLINE  ONLINE       london1
               ONLINE  ONLINE       london2
               ONLINE  ONLINE       london3
ora.acfs1.acfs1_vol1.acfs
               ONLINE  ONLINE       london1
               ONLINE  ONLINE       london2
               ONLINE  ONLINE       london3
...

In Oracle 11gR1 and earlier, similar output can be generated using crs_stat -t. Note that in Oracle 11gR2, this report no longer truncates resource names, and the output for each node appears on a separate line. An interesting fact about the state details field: If there is a severe problem with the database—if the online redo logs can't be archived, for example—then this will be reported in that column.

The output of the crsctl status resource command does not list the daemons of the High Availability Services stack! You must use the initially undocumented -init option to accomplish this:

[root@london1 ˜]# crsctl status resource -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       london1                  Started
ora.crsd
      1        ONLINE  ONLINE       london1
ora.cssd
      1        ONLINE  ONLINE       london1
ora.cssdmonitor
      1        ONLINE  ONLINE       london1
ora.ctssd
      1        ONLINE  ONLINE       london1                  OBSERVER
ora.diskmon
      1        ONLINE  ONLINE       london1
ora.drivers.acfs
      1        ONLINE  ONLINE       london1
ora.evmd
      1        ONLINE  ONLINE       london1
ora.gipcd
      1        ONLINE  ONLINE       london1
ora.gpnpd
      1        ONLINE  ONLINE       london1
ora.mdnsd
      1        ONLINE  ONLINE       london1

The functionality of the deprecated crs_start, crs_stop, and crs_relocate utilities has been integrated into crsctl. User-defined resources are covered in more detail in the "Protecting Applications with Clusterware" section later in this chapter; for now, you can initiate a resource using crsctl start resource resourceName and stop it using crsctl stop resource resourceName. If Oracle Support recommends doing so, you can stop resources of the High Availability Services stack by appending the -init parameter to crsctl start/stop resource resourceName. Bear in mind that this command merely instructs the high availability daemons to perform the task; you still need to check the log file to see whether it succeeded past the start request.

Many of the remaining options of the crsctl utility will be discussed in different sections later this chapter. For example, to make applications highly available, crsctl commands can be used to create user-defined resources. Again, refer to the "Protecting Applications with Clusterware" section for more information on this topic. This section will cover how to access control lists, as well as how to set and get resource permissions. This section also covers how the framework works to protect resources.

crsctl is also required to move voting disks into ASM or to other storage options (see the "Maintaining Voting Disks" section for more information on this topic).

The srvctl utility is a command-line tool that manages Oracle resources configured in the cluster. srvctl has been available since Oracle 10gR1. Like crsctl, srvctl has been significantly enhanced in Oracle 11gR2. In Oracle 11gR1 and earlier, srvctl managed six object types: asm, database, instances, services, node applications, and listeners. Oracle 11gR2 adds an additional ten object types: GNS, VIP Addresses, SCAN VIP addresses, SCAN listeners, Oracle homes, OC4J, servers, server pools, ASM disk groups, and ASM file systems.

The same options are available in Oracle 11gR2 that were available in previous releases: enable, disable, start, stop, relocate, status, add, remove, modify, config, getenv, setenv, and unsetenv. Note that all options have not been implemented for all of the objects. Table 8-2 summarizes srvctl options available in Oracle 11gR2 for each object type.

You need to keep a couple points in mind when shutting down an ASM instance using srvctl stop asm. In Oracle 11.1 and earlier, you could stop an ASM instance, which resulted in a dismount command and a shutdown of the instance. In Oracle 11.2 the srvctl stop asm command does not work, especially if voting disks and OCR are located in ASM itself. To stop ASM in Oracle RAC 11.2, you need to shut down all clients of ASM, including CSSD. The only way to do this is to stop the High Availability Services stack on the node. The same applies to the srvctl stop home command when the Oracle home is the ASM home (you can find more information about this topic in Chapter 9).

Another useful command to know is srvctl's config option. This option retrieves information about a system, reporting all database resources if no additional arguments are provided. If an object such as a database, asm, scans, or scan listener is passed, then this option provides detailed information about the specified resource. This functionality is essential for deployments that GNS where DHCP is responsible for assigning network addresses. For example, you could use the following to find out which IP addresses are used for the SCAN:

[oracle@london1 ˜]$ srvctl config scan
SCAN name: cluster1, Network: 1/172.17.1.0/255.255.255.0/
SCAN VIP name: scan1, IP: /cluster1.example.com/172.17.1.205
SCAN VIP name: scan2, IP: /cluster1.example.com/172.17.1.206
SCAN VIP name: scan3, IP: /cluster1.example.com/172.17.1.207

The srvctl stop home command simplifies patching the RDBMS. This command records which resources are currently started from the current node. This information is stored in a state file that the administrator specifies. Upon patch completion, you can use the srvctl start home command when the Grid Infrastructure restarts, and then use the state file and all resources that were active before the patch application is brought online again.

The Cluster Verification Utility (CVU) is a command-line tool that was introduced in Oracle 10gR2. The CVU checks the configuration of a cluster and reports whether each component is successfully configured. The CVU checks operating system versions and patches, kernel parameters, user limits, operating system groups and users, secure shell configuration, networking configuration, and shared storage devices. The CVU comes in three versions; be sure to run cluvfy.sh on the staging area of the Grid Infrastructure media. You can download this file from Oracle's Technet website; you will find the cluvfy file in $GRID_HOME/bin after a successful installation.

The CVU can be invoked at a number of stages during the installation and configuration process. Each step represents a set of components that should be configured at that time. In Oracle 11gR2 and later, the OUI automatically executes the CVU as part of the installation process. In our experience, however, it is still more efficient to run the CVU prior to starting the installer to address any oversights or errors in the initial configuration. The CVU should also be run following the completion of administrative tasks, such as node addition and deletion.

Table 8-3 shows the stages when you should run the CVU in Oracle 11gR2.

You can also invoke the CVU to check individual components. This functionality can be useful for isolating a problem. Table 8-4 lists the CVU components in Oracle 11gR2.

The CVU is implemented as a set of Java classes. On Linux, the CVU can be executed using the runcluvfy.sh shell script. This script configures the Java environment prior to starting the CVU off the installation media. After the installation has completed, the cluvfy file can be found in $GRID_HOME/bin (like any other executable).

The default form of the command includes a list of nodes that should be checked. For example, the following snippet verifies the configuration of the hardware and the operating system on nodes london1 and london2:

[oracle@london1 ˜]$ cluvfy stage -pre hwos -n london1,london2

You can append the -verbose switch to generate more detailed output:

[oracle@london1 ˜]$ cluvfy stage -pre hwos -n london1,london2 -verbose

It is good practice to keep the output of the execution as part of the installation documentation. In Oracle 11gR2 and later, the CVU can optionally generate fixup scripts to resolve a limited subset of errors. In Linux, fixup scripts can be generated to adjust kernel parameters and modify user limits. The resulting scripts must be executed by the root user.

The CVU is supplied with the Oracle Clusterware software distribution. Alternatively, it can be downloaded from the downloads section of the Oracle website at this location: www.oracle.com/technology/products/database/clustering/cvu/cvu_download_homepage.html.

The CVU must be executed by a regular user (e.g., grid or oracle; it cannot be executed by the root user).

Occasionally it is necessary to gather more than the verbose output from the CVU, especially before the software has been successfully instantiated on the cluster nodes. In this case, a little modification to the script is necessary. By default, the runcluvfy.sh script on the Grid Infrastructure staging location removes all the intermediate trace output in line 129:

128 # Cleanup the home for cluster verification software
129 $RM -rf $CV_HOME

To preserve the output, it is necessary to comment this line out. The script has hard-coded dependencies on files in the staging location. If your staging location is read-only (e.g., it is an NFS mount or DVD), then you can still make the change quite easily. Copy the runcluvfy.sh file to a writable location, such as /u01/software. Next, create cluster verification home and export this as an environment variable. Now change the script in line 129 and comment out this line:

RM -rf $CV_HOME

Also change the variable EXEC_DIR in line 29 so it points to the location where your Grid Infrastructure software is staged. Make sure to set CV_TRACE to another writable directory for the trace output, and then execute runcluvfy.sh.

...

Note that the CVU is only designed to verify that a configuration meets minimum installation requirements; in other words, it isn't foolproof. For example, it occasionally reports failures that can be safely ignored. In other cases, it misses some configuration errors that may subsequently cause an installation to fail.

The Oracle Interface Configuration Tool (oifcfg) is a command-line tool that can be used to configure network interfaces within Oracle Clusterware. The oifcfg utility can be used to add new public or private interfaces, as well as to modify existing subnet information.

Prior to Oracle 11gR2, oifcfg updated the OCR only. In Oracle 11gR2, oifcfg has been extended to update the OLR and the GPNP profile. Please refer back to the "Recording Information with the Grid Plug And Play Profile" section for more information on where the network information is stored in the profile.

The Oracle Cluster Registry Configuration tool ocrconfig is a command-line utility that can be used to administer the OCR and OLR. The ocrconfig tool has a number of options, including the ability to add and delete OCR mirrors, perform manual backups of the OCR, restore OCR backups, and export and import OCR configuration data. Many of the options for ocrconfig are also applicable to maintaining the OLR; options that target the OLR use the -local switch.

The Oracle Cluster Registry Check tool, ocrcheck, checks the state of the OCR and its mirrors. The behavior of ocrcheck is determined by the user that invokes it. When invoked by a regular user, such as grid or oracle, ocrcheck checks the accessibility of all OCR mirror copies. It also reports on the current size and free space in the OCR. When invoked by the root user, ocrcheck also performs a structural check on the contents of the OCR and reports any errors. The ocrcheck command is most useful when trying to determine logical corruption in the OCR.

The Oracle Cluster Registry Dump tool, ocrdump, can be used to dump the contents of the OCR to a text or XML file. ocrdump can only be executed by the root user. If requested, ocrdump can also dump the OLR to a file. The dump file name will default to OCRDUMP; however, this name can be changed by specifying an alternative file name on the command line. If desired, ocrdump can write to standard output. A very useful option is to extract the contents of a backed up OCR or OLR. This enables you to perform before and after comparisons when applying patchsets, for example. If you are unsure where your backup files are located, consult the output of ocrconfig -showbackup [-local]. For very specific troubleshooting needs, ocrdump offers the option to print only a specific key from the registry.

The clusterwide shared OCR contains the information about a system's resources, called a resource profile. In pre-11.2 installations, the crs_stat utility was used to dump a resource profile. Grid Infrastructure uses the crsctl status resource resourceName -p command to achieve the same result. Resource profiles have changed significantly with Oracle 11.2 (you can see a few examples that illustrate these differences in this chapter's "Configuring Active/Passive Clustering for Oracle Database" and "Configuring Active/Passive Clustering for Apache Tomcat" sections). Depending on the complexity of your application, it can be made up of a single or multiple resources. You can also define dependencies between the resources, ensuring they all start up in the correct order.

The changes to the resource profiles in Oracle 11.2 are so significant that Oracle dedicated a new appendix—Appendix B—to its release guide, "Clusterware Administration and Deployment Guide 11g Release 2 (11.2)." While the Oracle-supplied resources come with dedicated resource profiles that do not need to be touched, user-defined resources need to include a resource profile to work correctly.

In addition to a resource profile, user-defined resources need a controller, which is called an action script in the documentation. Grid Infrastructure uses agent programs (agents) to manage resources, and these agents invoke the action script's functions. The most common agent used to protect user-defined resources is the scriptagent. This agent allows developers to use simple shell scripts as action scripts for a resource. When used with scriptagent, an action script must implement the following callbacks and interfaces between the user resource and the framework:

For specific application needs, you can develop your own agent and provide callback functions for those entry points. If an agent does not provide a callback function for the entry point, the action script of the resource will be invoked to perform the required action.

The high availability framework will invoke the action script (or the agent directly) with the start option whenever crsctl start resource is called with the resource name. Similarly, you can stop, clean, or check by invoking the desired option. How well you code these options has a direct impact on the overall stability and responsiveness of the resource.

Resources sharing common attributes can be grouped into resource types. Predefined resource types are local resources or cluster resources; however, developers can also write their own types. In previous versions of Clusterware, Oracle supported another resource type, called an application resource; however, that resource type has been deprecated. It is still implemented for backward compatibility, but it should no longer be used.

The distinction between the resource types is visible in the output of crsctl status resource. Note how there are "cluster resources" and "local resources" sections. As their names suggest, local resources are tied to the node they are defined for, and they do not fail over. Cluster resources, on the other hand, can fail over and do not necessarily execute on all cluster nodes.

Grid Infrastructure continuously monitors resources as defined in their resource profiles. Any given resource is assigned a state, which can one of the following:

The Oracle database can easily be configured to use the Clusterware framework for high availability. Using Grid Infrastructure to protect a database resource is a very cost effective way of setting up an active/passive cluster. As an added advantage, using only one vendor's software stack to implement the cluster can make troubleshooting easier. Staff already familiar with RAC will easily be able to set up and run this configuration because it uses an identical software stack: all commands and log files are in familiar locations, and troubleshooting does not rely on the input from other teams.

To set up an active/passive cluster with 11g Release 2, you need to initially install Grid Infrastructure on all nodes of the cluster. Grid Infrastructure will provide the user with a cluster logical volume manager: ASM. If for some reason another file system is required, you have the option of choosing from the supported cluster file systems, including ACFS. Using a cluster file system that is mounted concurrently to both cluster nodes offers the advantage of not having to remount the database files and binaries in case of a node failure. Some configurations we saw suffered from extended failover periods caused by required file system checks before the file system could be remounted.

On top of the Grid Infrastructure build, you perform a local installation of the RDBMS. It is important that you do not choose a cluster installation when prompted so; otherwise, you risk violating your license agreement with Oracle.

When completed, the software stack consists of the components listed in Figure 8-2.

Figure 8.2. Oracle Software for an Active/Passive Cluster

After the binaries are installed and patched according to your standards, you need to create an ASM disk group or OCFS2/GFS mount point to store the database files. Next, start up the database configuration assistant from the first node to create a database. Please ensure that you store all the data files in ASM or the clustered file system. The same applies for the Fast Recovery Area: it should be on shared storage, as well.

After the database is created by dbca, it is automatically registered in the OLR. The profile of the resource is a good reference for the resource profile to be created in the next step. It is a good idea to save the resource profile in a safe location. You can extract the profile with the crsctl status resource ora. databaseName.db -p command. Next, remove the database resource from the OLR, as shown in the following example (this example assumes the database is named TEST):

[oracle@london1 ˜]$ srvctl remove database -d TEST

Next, we need an action script that allows the framework to start, stop, and check the database resource. A possible action script might look like this:

#!/bin/bash

export ORACLE_SID=TEST
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1
export PATH=/usr/local/bin:$ORACLE_HOME/bin:$PATH
export ORACLE_OWNER=oracle

case $1 in
'start')
  su - $ORACLE_OWNER <<EOF
export ORACLE_SID=$ORACLE_SID
export ORACLE_HOME=$ORACLE_HOME
$ORACLE_HOME/bin/sqlplus /nolog
conn / as sysdba
startup
exit
EOF
   RET=0
    ;;
'stop')
  su - $ORACLE_OWNER <<EOF
export ORACLE_SID=$ORACLE_SID
export ORACLE_HOME=$ORACLE_HOME
$ORACLE_HOME/bin/sqlplus /nolog
conn / as sysdba
shutdown immediate
exit
EOF
   RET=0
    ;;
'check')
   # check for the existance of the smon process for $ORACLE_SID
   # this check could be improved, but was kept short on purpose
   found=`ps -ef | grep smon | grep $ORACLE_SID | wc -l`
   if [ $found = 0 ]; then
     RET=1
   else
     RET=0
   fi
   ;;
*)
   RET=0
    ;;
esac
# A 0 indicates success, return 1 for an error.
if [ $RET -eq 0 ]; then

exit 0
else
  exit 1
fi

The preceding action script defines environment variables in the header section, setting the Oracle owner, the Oracle SID, and the Oracle home. It then implements the required start, stop, clean, and check entry points. The check could be more elaborate—for example, it could check for a hung instance—however, this example was kept short and simple for the sake of clarity.

The action script needs to be deployed to the other cluster node, and it must be made executable. Whenever there is a change to the script, the action script needs to be synchronized with the other cluster nodes! After defining the action script, you need to create a new cluster resource. Securing an Oracle database instance with Clusterware is simplified by the availability of the SCAN; users of the database do not need to worry about which node the database is currently started on because the SCAN abstracts this information from them. The communication of the SCAN with the local listener also makes a floating virtual IP (which other cluster stacks require) unnecessary. Using some values of the resource profile saved earlier, you need to configure the resource profile for the cluster resource next. It is easier to use a configuration file than to supply all the resource parameters on the command line in name-value pairs. To recreate the database cluster resource, you could use the following configuration file, which is saved as TEST.db.config:

PLACEMENT=restricted
HOSTING_MEMBERS=london1 london2
CHECK_INTERVAL=30
CARDINALITY=1
ACTIVE_PLACEMENT=0
AUTO_START=restore
DEGREE=1
DESCRIPTION="Oracle Database Resource"
RESTART_ATTEMPTS=1
ACTION_SCRIPT=/u01/app/crs/hadaemon/hacluster.sh

The preceding configuration file can be read as follows. Placement and hosting members go hand-in-hand; the restricted policy only allows executing the resource on the hosting members london1 and london2. The check interval of 30 seconds determines the frequency of checks, and setting active placement 0 prevents Oracle from relocating the resource to a failed node; a failed node could indicate a second outage in the system, and it would be better to let the DBAs perform the switch back to the primary node. The cardinality specifies that there will always be one instance of this resource in the cluster (never more or less); similarly, the degree of 1 indicates that there cannot be more than one instance of the resource on the same node. The parameters restart attempts and action script are self-explanatory in this context. Please note that the directory $GRID_HOME/hadaemon/ did not exist; it was created for this example.

Next, use the following command to register the resource in Grid Infrastructure:

$ crsctl add resource TEST.db -type cluster_resource -file TEST.db.config

If you get a CRS-2518 (invalid directory path) error while executing this command, you most likely forgot to deploy the action script to the other node.

The permissions on the resource at this moment are too strict. At this moment, only root can effectively modify the resource at this time. Trying to start the resource as the oracle account results in a failure, as in this example:

[oracle@london1 ˜]$ crsctl start resource TEST.db
CRS-0245:  User doesn't have enough privilege to perform the operation
CRS-4000: Command Start failed, or completed with errors.

You can confirm the cause of this failure checking the permissions:

[root@london1 ˜]# crsctl getperm resource TEST.db
Name: TEST.db
owner:root:rwx,pgrp:root:r-x,other::r--

You would like the oracle user to also be able to start and stop the resource; you can enable this level of permission using the crsctl setperm command, as in the following example:

[root@london1 ˜]# crsctl setperm resource TEST.db -o oracle
[root@london1 ˜]# crsctl getperm resource TEST.db
Name: TEST.db
owner:oracle:rwx,pgrp:root:r-x,other::r--

The preceding snippet allows users logging in as oracle (or using sudo su - oracle) to start and stop the database, effectively transferring ownership of the resource to the oracle account. You need to ensure that the oracle account can execute the action script; otherwise, you will get an error when trying to start the resource. Members of the oinstall group have the same rights. All other users defined on the operating system level with privileges to execute the binaries in $GRID_HOME can only read the resource status.

The final preparation steps are to copy the password file and pfile pointing to the ASM spfile into the passive node's $ORACLE_HOME/dbs directory. You are in 11.2, so the ADR will take care of all your diagnostic files. You need to create the directory for your audit files however, which normally is in $ORACLE_BASE/admin/$ORACLE_SID/adump.

From this point on, you need to use crsctl {start|stop|relocate} to manipulate the database. Relocating the resource from london1 to london2 is easy after you implement the preceding steps, as the next example demonstrates:

[root@london1 ˜]# crsctl status resource TEST.db
NAME=TEST.db
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on london1
[root@london1 ˜]# crsctl relocate resource TEST.db
CRS-2673: Attempting to stop 'TEST.db' on 'london1'
CRS-2677: Stop of 'TEST.db' on 'london1' succeeded
CRS-2672: Attempting to start 'TEST.db' on 'london2'
CRS-2676: Start of 'TEST.db' on 'london2' succeeded
[root@london1 ˜]# crsctl status resource TEST.db
NAME=TEST.db
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on london2

A slightly more complex example involves making Apache Tomcat or another web-accessible application highly available. The difference in this setup compared to the database setup described in the previous chapter lies in the fact that you need to use a floating virtual IP address. Floating in this context means that the virtual IP address moves jointly with the application. Oracle calls its implementation of a floating VIP an application VIP. Application VIPs were introduced in Oracle Clusterware 10.2. Previous versions only had a node VIP.

The idea behind application VIPs is that, in the case of a node failure, both VIP and the application migrate to the other node. The example that follows makes Apache Tomcat highly available, which is accomplished by installing the binaries for version 6.0.26 in /u01/tomcat on two nodes in the cluster. The rest of this section outlines the steps you must take to make Apache Tomcat highly available.

Oracle Grid Infrastructure does not provide an application VIP by default, so you have to create one. A new utility, called appvipcfg, can be used to set up an application VIP, as in the following example:

[root@london1 ˜]# appvipcfg
Production Copyright 2007, 2008, Oracle.All rights reserved

  Usage: appvipcfg create -network=<network_number> -ip=<ip_address> -vipname=<vipname>
                          -user=<user_name>[-group=<group_name>]
                   delete -vipname=<vipname>
[root@london1 ˜]# appvipcfg create -network=1 
> -ip 172.17.1.108 -vipname httpd-vip -user=root
Production Copyright 2007, 2008, Oracle.All rights reserved
2010-06-18 16:07:12: Creating Resource Type
2010-06-18 16:07:12: Executing cmd: /u01/app/crs/bin/crsctl add type app.appvip.type -basetype cluster_resource -file /u01/app/crs/crs/template/appvip.type
2010-06-18 16:07:13: Create the Resource
2010-06-18 16:07:13: Executing cmd: /u01/app/crs/bin/crsctl add resource httpd-vip -type app.appvip.type -attr USR_ORA_VIP=172.17.1.104,START_DEPENDENCIES=hard(ora.net1.network)

pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL='owner:root:rwx,pgrp:root:r-x,other::r--,user:root:r-x'

The preceding output shows that the new resource has been created, and it is owned by root exclusively. You could use crsctl setperm to change the ACL, but this is not required for this process. Bear in mind that no account other than root can start the resource at this time. You can verify the result of this operation by querying the resource just created. Note how the httpd-vip does not have an ora. prefix:

[root@london1 ˜]# crsctl status resource httpd-vip
NAME=httpd-vip
TYPE=app.appvip.type
TARGET=OFFLINE
STATE=OFFLINE

Checking the resource profile reveals that it matches the output of the appvipcfg command; the output has been shortened for readability, and it focuses only on the most important keys (the other keys were removed for the sake of clarity):

[root@london1 ˜]# crsctl stat res httpd-vip -p
NAME=httpd-vip
TYPE=app.appvip.type
ACL=owner:root:rwx,pgrp:root:r-x,other::r--,user:root:r-x
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=1
DEGREE=1
DESCRIPTION=Application VIP
RESTART_ATTEMPTS=0
SCRIPT_TIMEOUT=60
SERVER_POOLS=*
START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network)
STOP_DEPENDENCIES=hard(ora.net1.network)
USR_ORA_VIP=172.17.1.108
VERSION=11.2.0.1.0

The dependencies on the network ensure that, if the network is not started, it will be started as part of the VIP start. The resource is controlled by the CRSD orarootagent because changes to the network configuration require root privileges in Linux. The status of the resource revealed it was stopped; you can use the following command to start it:

[root@london1 ˜]# crsctl start res httpd-vip
CRS-2672: Attempting to start 'httpd-vip' on 'london2'
CRS-2676: Start of 'httpd-vip' on 'london2' succeeded
[root@london1 ˜]#

In this case, Grid Infrastructure decided to start the resource on server london2.

[root@london1 ˜]# crsctl status resource httpd-vip
NAME=httpd-vip
TYPE=app.appvip.type
TARGET=ONLINE
STATE=ONLINE on london2

You can verify this by querying the network setup, which has changed. The following output is again shortened for readability:

[root@london2 source]# ifconfig
...
eth0:3   Link encap:Ethernet  HWaddr 00:16:36:2B:F2:F6
          inet addr:172.17.1.108  Bcast:172.17.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

Next, you need an action script that controls the Tomcat resource. Again, the requirement is to implement start, stop, clean, and check functions in the action script. The Oracle documentation lists C, C++, and shell scripts as candidate languages for an action script. We think that the action script can be any executable, as long as it returns 0 or 1, as required by Grid Infrastructure. A sample action script that checks for the Tomcat webserver could be written in plan bash, as in the following example:

#!/bin/bash

export CATALINA_HOME=/u01/tomcat
export ORA_CRS_HOME=/u01/app/crs
export JAVA_HOME=$CRS_HOME/jdk
export CHECKURL="http://172.17.1.108:8080/tomcat-power.gif"

case $1 in
'start')
   $CATALINA_HOME/bin/startup.sh
   RET=$?
    ;;
'stop')
   $CATALINA_HOME/bin/shutdown.sh
   RET=$?
    ;;
'clean')
   $CATALINA_HOME/bin/shutdown.sh
   RET=$?
    ;;
'check')
   # download a simple, small image from the tomcat server
   /usr/bin/wget -q --delete-after $CHECKURL
   RET=$?
    ;;
*)
   RET=0
    ;;
esac
# A 0 indicates success, return 1 for an error.
if [ $RET -eq 0 ]; then

exit 0
else
  exit 1
fi

In our installation, we created a $GRID_HOME/hadaemon/ directory on all nodes in the cluster to save the Tomcat action script, tomcat.sh.

The next step is to ensure that the file is executable, which you can accomplish by running test to see whether the file works as expected. Once you are confident that the script is working, you can add the Tomcat resource.

The easiest way to configure the new resource is by creating a text file with the required attributes, as in this example:

[root@london1 hadaemon]# cat tomcat.profile
ACTION_SCRIPT=/u01/app/crs/hadaemon/tomcat.sh
PLACEMENT=restricted
HOSTING_MEMBERS=london1 london2
CHECK_INTERVAL=30
RESTART_ATTEMPTS=2
CHECK_INTERVAL=30
RESTART_ATTEMPTS=2
START_DEPENDENCIES=hard(httpd-vip)
STOP_DEPENDENCIES=hard(httpd-vip)

The following command registers the resource tomcat in Grid Infrastructure:

[root@london1 ˜]# crsctl add resource tomcat -type cluster_resource -file tomcat.profile

Again, the profile registered matches what has been defined in the tomcat.profile file, plus the default values:

[root@london1 hadaemon]# crsctl status resource tomcat -p
NAME=tomcat
TYPE=cluster_resource
ACL=owner:root:rwx,pgrp:root:r-x,other::r--
ACTION_SCRIPT=/u01/app/crs/hadaemon/tomcat.sh
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/scriptagent
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=30
DEFAULT_TEMPLATE=
DEGREE=1
DESCRIPTION=
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=london1 london2
LOAD=1
LOGGING_LEVEL=1

NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PLACEMENT=restricted
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=2
SCRIPT_TIMEOUT=60
SERVER_POOLS=
START_DEPENDENCIES=hard(httpd-vip)
START_TIMEOUT=0
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(httpd-vip)
STOP_TIMEOUT=0
UPTIME_THRESHOLD=1h

This example includes a hard dependency on the httpd-vip resource, which is started now. If you try to start the Tomcat resource, you will get the following error:

[root@london1 hadaemon]# crsctl start resource tomcat
CRS-2672: Attempting to start 'tomcat' on 'london1'
CRS-2674: Start of 'tomcat' on 'london1' failed
CRS-2527: Unable to start 'tomcat' because it has a 'hard' dependency
on 'httpd-vip'
CRS-2525: All instances of the resource 'httpd-vip' are already running;
relocate is not allowed because the force option was not specified
CRS-4000: Command Start failed, or completed with errors.

To get around this problem, you need begin by shutting down httpd-vip and then trying again:

[root@london1 hadaemon]# crsctl stop res httpd-vip
CRS-2673: Attempting to stop 'httpd-vip' on 'london1'
CRS-2677: Stop of 'httpd-vip' on 'london1' succeeded
[root@london1 hadaemon]# crsctl start res tomcat
CRS-2672: Attempting to start 'httpd-vip' on 'london1'
CRS-2676: Start of 'httpd-vip' on 'london1' succeeded
CRS-2672: Attempting to start 'tomcat' on 'london1'
CRS-2676: Start of 'tomcat' on 'london1' succeeded

The Tomcat servlet and JSP container is now highly available. However, please bear in mind that the session state of an application will not fail over to the passive node in the case of a node failure. The preceding example could be further enhanced by using a shared cluster logical ACFS volume to store the web applications used by Tomcat, as well as and the Tomcat binaries themselves.

Startup issues in RAC are usually caused by a change in the hardware setup and configuration, and they can sometimes be difficult to diagnose. For example, changes to routers and firewalls on the network layer can have a negative impact on the health of the cluster. Operating system upgrades should also not be underestimated. A simple upgrade of the running kernel can have devastating effects for a system using ASMLib if the system administrator forgets to install the matching kernel modules.

In our experience, the most common startup problem in Clusterware prior to version 11.2 issues concerned permissions on the raw (block) devices for the OCR and voting disk. Oracle 11.2 addresses this problem by storing OCR and voting disks in ASM. Should the cluster stack not start after a reboot or after stopping the cluster stack on a node, then there are a few locations to look for errors.

Oracle's cluster verification tool recommends the following settings for the OCR when block devices are used:

How you should set the permissions and user/group attributes depends on how you defined the devices in the first place. That said, the most commonly used scripts to do this are udev or the /etc/rc.local scripts.

Clusters configured to use GNS will not start completely until communication with the DHCP and DNS servers is established.

A good first indication of the status of the cluster node is the output from the crsctl check crs command. Typically, you define your troubleshooting strategy based on its output. The following output shows what the command should return for a healthy cluster node:

[oracle@london1 ˜]$ $GRID_HOME/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Several possible problems can be reported at this stage, including the following:

Systematic troubleshooting of Grid Infrastructure follows a bottom-up approach because the resources have clearly defined dependencies (please refer to the "Startup Sequence" section for more information about the starting Grid Infrastructure in general). The generic troubleshooting document Oracle provides to resolve such issue is "My Oracle Support note 1050908.1 How to Troubleshoot Grid Infrastructure Startup Issues"; the following sections are loosely modeled on that support note.

[ default][3046704848] OHASD Daemon Starting. Command string :reboot
[ default][3046704848] Initializing OLR
[  OCRRAW][3046704848]proprioo: for disk 0
 (/u01/app/crs/cdata/london1.olr),
 id match (1), total id sets, (1) need recover (0), my votes (0),
 total votes (0), commit_lsn (15), lsn (15)
[  OCRRAW][3046704848]proprioo: my id set: (2018565920, 1028247821, 0, 0, 0)
[  OCRRAW][3046704848]proprioo: 1st set: (2018565920, 1028247821, 0, 0, 0)
[  OCRRAW][3046704848]proprioo: 2nd set: (0, 0, 0, 0, 0)
[  CRSOCR][3046704848] OCR context init CACHE Level: 0xaa4cfe0
[ default][3046704848] OHASD running as the Privileged user

[root@london1 ˜]# crsctl check crs
CRS-4639: Could not contact Oracle High Availability Services

[ default][1381425744] OHASD Daemon Starting. Command string :restart
[ default][1381425744] Initializing OLR
[  OCROSD][1381425744]utopen:6m':failed in stat OCR file/disk
 /u01/app/crs/cdata/london1.olr,
 errno=2, os err string=No such file or directory
[  OCROSD][1381425744]utopen:7:failed to open any OCR file/disk, errno=2,
 os err string=No such file or directory
[  OCRRAW][1381425744]proprinit: Could not open raw device
[  OCRAPI][1381425744]a_init:16!: Backend init unsuccessful : [26]
[  CRSOCR][1381425744] OCR context init failure.  Error: PROCL-26: Error
 while accessing the physical storage Operating System
 error [No such file or directory] [2]

[ default][1381425744] OLR initalization failured, rc=26
[ default][1381425744]Created alert : (:OHAS00106:) :  Failed to initialize
 Oracle Local Registry
[ default][1381425744][PANIC] OHASD exiting; Could not init OLR

[root@london1 ohasd]# ocrcheck -local
Status of Oracle Local Registry is as follows :
    Version                  :          3
    Total space (kbytes)     :     262120
    Used space (kbytes)      :       2232
    Available space (kbytes) :     259888
    ID                       : 1022831156
    Device/File Name         : /u01/app/crs/cdata/london1.olr
                               Device/File integrity check failed

   Local registry integrity check failed

    Logical corruption check bypassed

[    CSSD][3412950784]clssscmain: Starting CSS daemon, version 11.2.xxx, in
(clustered) mode with uniqueness value xxx

2010-06-25 09:43:19.843: [    GPnP][3941465856]clsgpnpm_exchange:
 [at clsgpnpm.c:1175] Calling "ipc://GPNPD_london1", try 4 of 500...
2010-06-25 09:43:19.853: [    GPnP][3941465856]clsgpnp_profileVerifyForCall:
 [at clsgpnp.c:1867] Result: (87) CLSGPNP_SIG_VALPEER. Profile verified.
 prf=0x113478c0
2010-06-25 09:43:19.853: [    GPnP][3941465856]clsgpnp_profileGetSequenceRef:
 [at clsgpnp.c:841] Result: (0) CLSGPNP_OK. seq of p=0x113478c0 is '7'=7
2010-06-25 09:43:19.853: [    GPnP][3941465856]clsgpnp_profileCallUrlInt:
 [at clsgpnp.c:2186] Result: (0) CLSGPNP_OK. Successful get-profile CALL to
 remote "ipc://GPNPD_london1" disco ""
2010-06-25 09:43:19.853: [    GPnP][3941465856]clsgpnp_getProfileEx:
 [at clsgpnp.c:540] Result: (0) CLSGPNP_OK. got profile 0x113478c0

2010-06-25 10:25:17.057: [ GPnP][7256921234]clsgpnp_getProfileEx:
 [at clsgpnp.c:546] Result:
 (13) CLSGPNP_NO_DAEMON. Can't get GPnP service profile from local GPnP daemon
2010-06-25 10:25:17.057: [ default][7256921234]Cannot get GPnP profile. Error
 CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2010-06-25 10:25:17.057: [ CSSD][7256921234]clsgpnp_getProfile failed, rc(13)

2010-06-23 16:47:49.651: [    CLSF][1158797632]Opened hdl:0x32d5770 for
  dev:ORCL:OCRVOTE1:
2010-06-23 16:47:49.661: [    CSSD][1158797632]clssnmvDiskVerify: Successful discovery
for disk ORCL:OCRVOTE1, UID 84b990fb-73234f2b-bf82b7ae-c4d2e0a2,
Pending CIN 0:1276871678:0, Committed CIN 0:1276871678:0

2010-06-23 16:47:49.661: [    CLSF][1158797632]Closing handle:0x32d5770
2010-06-23 16:47:49.661: [   SKGFD][1158797632]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x3280cc0
for disk :ORCL:OCRVOTE1:
2010-06-23 16:47:49.661: [    CSSD][1158797632]clssnmvDiskVerify: discovered a
potential voting file

2010-06-23 16:47:49.703: [    CSSD][1158797632]clssnmvDiskVerify: Successful discovery
of 0 disks

2010-04-30 10:34:01.183: [ CRSMAIN][3257705040] Checking the OCR device
2010-04-30 10:34:01.183: [ CRSMAIN][3257705040] Connecting to the CSS Daemon
2010-04-30 10:34:01.184: [ CSSCLNT][1111550272]clssnsquerymode: not connected to CSSD
2010-04-30 10:34:01.204: [ CSSCLNT][3257705040]clssscConnect: gipc request failed with 29         (0x16)
2010-04-30 10:34:01.204: [ CSSCLNT][3257705040]clsssInitNative: connect failed, rc 29
2010-04-30 10:34:01.204: [  CRSRTI][3257705040] CSS is not ready. Received status 3 from CSS. Waiting for good status ..

2010-06-30 22:13:40.251: [  CRSOCR][968725072] OCR context init failure.  Error:
  PROC-26: Error while accessing the physical storage ASM error
[SLOS: cat=8, o
pn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +OCRVOTE.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +OCRVOTE.255.4294967295
ORA-15001: diskgrou
] [8]
2010-06-30 22:13:40.251: [    CRSD][968725072][PANIC] CRSD exiting: Could not init
 OCR, code: 26
2010-06-30 22:13:40.251: [    CRSD][968725072] Done.

[oracle@london1 ˜]$ nslookup - dnsServerName

When troubleshooting problems, it might be necessary to enable additional output. Some command-line tools support tracing and debugging output. To enable additional output, set the SRVM_TRACE environment variable to true, as in the following example:

[oracle@london1 ˜]$ export SRVM_TRACE=true

To disable tracing, use the built-in bash, unset SRVM_TRACE. A search in the $GRID_HOME/bin directory shows that several important utilities support additional trace information, including the following:

Even if it is impossible to get the meaning of the trace onscreen, this is always helpful information that can be sent to Oracle Support to speed up problem resolution.

This special type of node maintenance is always performed in two steps:

When adding nodes into the cluster, it is imperative to ensure that all cluster nodes are properly cabled, the operating system is installed and patched, and that all user IDs across the cluster have the same value. Additionally, you need to ensure user equivalence for the Grid Infrastructure software owner and the RDBMS software owner across all nodes in the cluster. If you are using ASMLib, you should ensure that the driver is loaded and that all disks are discovered on the new nodes.

Note that the steps are slightly different depending on whether you decided to use a GNS setup or a more conventional configuration. In any case, you should run the cluster verification tool to ensure the cluster is ready for the node addition or deletion.

Oracle has made cluster maintenance a high priority in Oracle 11g Release 2. GNS enables dynamic provisioning of IP addresses for use as virtual IP addresses and SCAN VIPs. GNS is attractive when corporate DNS cannot easily be changed, or when administrators wish to assume more responsibility for their software stack. Without GNS, node additions require changes to the corporate DNS servers, and such changes can sometimes be difficult because of separate management chains in the organization.

[oracle@london1]$ $ORACLE_HOME/bin/cluvfy stage -post hwos -n london4 -verbose

[oracle@london1 ˜]$ cluvfy stage -pre nodeadd -n london4  -fixup -fixupdir /tmp

Fixup information has been generated for following node(s):
London4
Please run the following script on each node as "root" user to execute the fixups:
'/tmp/CVU_11.2.0.1.0_oracle/runfixup.sh'

[oracle@london1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={newNodePublicIP}"

SEVERE:Number of new nodes being added are not equal to number of new virtual nodes.
Silent install cannot continue.

[oracle@london1 bin] $ ./addNode.sh -silent "CLUSTER_NEW_NODES={london4}" 
> "CLUSTER_NEW_VIRTUAL_HOSTNAMES={london4-vip}"

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3699 MB    Passed
Oracle Universal Installer, Version 11.2.0.1.0 Production
Copyright (C) 1999, 2009, Oracle. All rights reserved.
Performing tests to see whether nodes london2,london3,london4 are available
............................................................... 100% Done.
...
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/crs
   New Nodes
Space Requirements
   New Nodes
      london4
         /: Required 3.28GB : Available 48.00GB
Installed Products
   Product Names
      Oracle Grid Infrastructure 11.2.0.1.0
      Sun JDK 1.5.0.17.0
      ...
      Oracle Database 11g 11.2.0.1.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Friday, November 6, 2009 9:14:02 PM GMT)
.                                                                 1% Done.

Instantiation of add node scripts complete
Copying to remote nodes (Friday, November 6, 2009 9:14:08 PM GMT)
......................................................................
...............                                 96% Done.
Home copied to new nodes
Saving inventory on nodes (Friday, November 6, 2009 9:34:52 PM GMT)
.                                                               100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session.
However, it has not yet been registered as the central inventory of this
system. To register the new inventory please run the script at
'/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'london4'.
If you do not register the inventory, you may not be able to update or patch
the products you installed.
The following configuration scripts need to be executed as the "root" user
in each cluster node.

/u01/app/oraInventory/orainstRoot.sh #On nodes london4
/u01/app/crs/root.sh #On nodes london4
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/crs was successful.
Please check '/tmp/silentInstall.log' for more details.

[root@london4 ˜]# /u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

[root@london4 ˜]# /u01/app/crs/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/crs

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2009-11-06 21:41:31: Parsing the host name
2009-11-06 21:41:31: Checking for super user privileges
2009-11-06 21:41:31: User has super user privileges
Using configuration parameter file:
/u01/app/crs/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an
active CSS daemon on node london1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to
join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'london4'
CRS-2676: Start of 'ora.mdnsd' on 'london4' succeeded
[some output has been removed for clarity]
CRS-2672: Attempting to start 'ora.asm' on 'london4'
CRS-2676: Start of 'ora.asm' on 'london4' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'london4'
CRS-2676: Start of 'ora.crsd' on 'london4' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'london4'
CRS-2676: Start of 'ora.evmd' on 'london4' succeeded
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
london4     2009/11/06 21:46:12     /u01/app/crs/cdata/london4/backup_20091106_214612.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 4095 MB    Passed
The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/crs"
 TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="london1"/>
      <NODE NAME="london2"/>
      <NODE NAME="london3"/>
      <NODE NAME="london4"/>
   </NODE_LIST>
</HOME>

[oracle@london1 bin] ./cluvfy stage -post nodeadd -n london4

[oracle@london1 bin]$ cd $ORACLE_HOME/oui/bin
[oracle@london1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={london4}"

Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 3622 MB    Passed
Oracle Universal Installer, Version 11.2.0.1.0 Production
Copyright (C) 1999, 2009, Oracle. All rights reserved.

Performing tests to see whether nodes london2,london3,london4 are available
............................................................... 100% Done.
...
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
   Source: /u01/app/oracle/product/11.2.0/dbhome_1
   New Nodes
Space Requirements
   New Nodes
      london4
         /: Required 3.62GB : Available 44.95GB
Installed Products
   Product Names
      Oracle Database 11g 11.2.0.1.0
      Sun JDK 1.5.0.17.0
      Enterprise Manager Common Core Files 10.2.0.4.2
[some output removed]
      Enterprise Edition Options 11.2.0.1.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Friday, November 6, 2009 9:54:36 PM GMT)
.                                                                 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Friday, November 6, 2009 9:54:46 PM GMT)
..                             96% Done.
Home copied to new nodes
Saving inventory on nodes (Friday, November 6, 2009 10:28:22 PM GMT)
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh #On nodes london4
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/app/oracle/product/11.2.0/dbhome_1
was successful.
Please check '/tmp/silentInstall.log' for more details.

Node deletion is an operation that few sites undertake lightly. In our experience, nodes were deleted from a cluster after catastrophic failures of multiple components of the underlying hardware. We have also seen unrecoverable driver updates on non-Unix platforms be responsible for having to remove the node from the cluster. The nodes that were removed are most often re-imaged by the system administrators and then added back into the cluster. The steps that follow show how to remove software from the RDBMS and Grid homes; however, in many cases, these tasks cannot be performed. From a DBA's point of view, it is important to have a clean OCR that does not reference the deleted node, and this can be achieved even without the node you need to delete being online.

Before a node can be removed from the cluster, you need to ensure that no database instance or other custom resource type uses that node. Also, you need to ensure that you have a backup of the OCR before continuing. Create one, if necessary, using this command:

ocrconfig -manualbackup

The dbca utility can help you remove database instances from a node, a task that beyond the scope of this chapter. By the time dbca finishes, it has updated the OCR and removed the redo thread and undo tablespace of that instance. It has also updated the tnsnames.ora files if there were no errors during the execution of the operation. Once the database instances and custom resources are deconfigured, you can proceed to removing the node from the cluster.

The following subsections take you through a node deletion process. In this example, the london2 node will be removed from a cluster that initially consists of two nodes: london1 and london2. We assume that the node to be removed from the cluster is still actively forming a part of the cluster. The node deletion process is the inverted equivalent of the node addition:

[oracle@london1 ˜]$ srvctl config listener -a
Name: LISTENER
Network: 1, Owner: oracle
Home: <CRS home>
  /u01/app/crs/ on node(s) london2,london1
End points: TCP:1521

[oracle@london1 ˜]$ srvctl disable listener -l LISTENER -n london2
[oracle@london1 ˜]$ srvctl stop listener -l LISTENER -n london2

[oracle@london2 ContentsXML]$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2009, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0/dbhome_1"

TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="london1"/>
      <NODE NAME="london2"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
</INVENTORY>

[oracle@london2 bin]$ ./runInstaller -updateNodeList 
ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES={london2} -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 1023 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[oracle@london2 bin]$

[oracle@london2 ˜]$ cat /u01/app/oraInventory/ContentsXML/inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2009, Oracle. All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
...
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0/dbhome_1"
   TYPE="O" IDX="2">
   <NODE_LIST>
      <NODE NAME="london2"/>
   </NODE_LIST>
</HOME>
</HOME_LIST>
</INVENTORY>

[oracle@london2 deinstall]$ ./deinstall -local
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############
[...]

A log of this session will be written to:
'/u01/app/oraInventory/logs/deinstall_deconfig2010-07-02_10-40-54-PM.out'
Any error messages from this session will be written to:
'/u01/app/oraInventory/logs/deinstall_deconfig2010-07-02_10-40-54-PM.err'

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

[oracle@london1 ˜]$ $ORACLE_HOME/oui/bin/runInstaller -updateNodeList 
> ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={london1}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 1023 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

The Oracle base for ORACLE_HOME=/u01/app/crs/ is /u01/app/oracle
[root@london2 ˜]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@london2 ˜]# cd /u01/app/crs/crs/install/
[root@london2 install]# ./rootcrs.pl -deconfig -force

2010-07-03 10:16:08: Parsing the host name
2010-07-03 10:16:08: Checking for super user privileges
2010-07-03 10:16:08: User has super user privileges
Using configuration parameter file: ./crsconfig_params
VIP exists.:london1
VIP exists.: /london1-vip/172.17.1.201/255.255.255.0/eth0
VIP exists.:london2
VIP exists.: /london2-vip/172.17.1.202/255.255.255.0/eth0
GSD exists.
ONS daemon exists. Local port 6100, remote port 6200
eONS daemon exists. Multicast port 18904, multicast IP address 234.187.14.127,
listening port 2016
ACFS-9200: Supported
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'london2'
CRS-2677: Stop of 'ora.registry.acfs' on 'london2' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed
resources on 'london2'
CRS-2673: Attempting to stop 'ora.crsd' on 'london2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources
on 'london2'
CRS-2673: Attempting to stop 'ora.OCRVOTE.dg' on 'london2'
CRS-2677: Stop of 'ora.OCRVOTE.dg' on 'london2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'london2'
CRS-2677: Stop of 'ora.asm' on 'london2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources
on 'london2' has completed
CRS-2677: Stop of 'ora.crsd' on 'london2' succeeded
[some output removed for clarity]
CRS-2677: Stop of 'ora.gipcd' on 'london2' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'london2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources
on 'london2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node
[root@london2 install]#

[root@london1 ˜]# crsctl delete node -n london2
CRS-4661: Node london2 successfully deleted.

[oracle@london2 ˜]$ /u01/app/crs/oui/bin/runInstaller 
> -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=london2" CRS=TRUE 
> -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 1023 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

[oracle@london2 deinstall]$ ./deinstall -local

[oracle@london2 bin]$  ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={london1,london3}" CRS=TRUE

Some sites require using a different port than 1521 for the listener. When initially installing Grid Infrastructure, the administrator is not given the choice of which port she wants to use. This means that changing the listener port must happen after the installation has completed successfully.

The netca network configuration assistant provides the easiest way to change the listener port. Start netca as the owner of the Grid Infrastructure home. The following series of screen shots documents how to change the listener port from 1521 to 1526. Figure 8-3 shows the Welcome screen.

Figure 8.3. The Welcome screen

Next, select the Listener Configuration option and click Next. This brings up the options screen shown in Figure 8-4.

Figure 8.4. The Listener options

Now select Reconfigure and click Next to advance to the Listener Selection screen (see Figure 8-5).

Figure 8.5. The Listener Selection screen

Unlike previous versions, the current version of Oracle Grid Infrastructure no longer uses the LISTENER_hostname naming convention for node listeners. Instead, all listeners are simply called LISTENER. Select LISTENER and proceed; most systems probably do not have additional listeners in place at this stage.

The netca utility warns you that it will shut down the listener with name LISTENER (see Figure 8-6). This is a clusterwide shutdown, which means that no new connections can be established to the listener you are changing. Existing connections are not affected. Acknowledging the warning prompts netca to shut down the listener.

Figure 8.6. Shutdown Warning

The Protocol Selection screen lets you select the protocol the listener should support (see Figure 8-7). For most deployments, this should be TCP/IP.

Figure 8.7. The Protocol Selection screen

The Select Subnet dropdown box at the top of the screen is useful only if you have multiple public networks in use. We will discuss this option in the "Configuring the Network" section later in the chapter. For now, select the protocols you would like to use and click Next. You will land on the Port Number screen shown in Figure 8-8.

Figure 8.8. The Port Number screen

It is finally time to change the port from the default number 1521 to your preferred port number. The example in Figure 8-8 uses port 1526.

Click Next to complete the reconfiguration. The utility does not always reliably start the listener resource in the cluster, so you should ensure that the listener resource is up on all nodes before completing the work.

To verify that the listeners are really listening on the correct port, issue the srvctl command, as shown in the following example:

[oracle@london1 ˜]$ srvctl config listener
Name: LISTENER
Network: 1, Owner: oracle
Home: <CRS home>
End points: TCP:1526
[oracle@london1 ˜]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): london1,london2

The messages here indicate that the listener is enabled, and that it is running on two nodes. You can also see that they are listening on port 1526, which is the correct port for our example.

Changing the SCAN listener endpoint is simpler than changing the database listener endpoints. The srvctl command line utility offers following option for modifying the SCAN listener endpoints:

[oracle@london1 log]$ srvctl modify scan_listener -h

Modifies the SCAN listeners so that the number of SCAN listeners is the same
as the number of SCAN VIPs or modifies the SCAN listener endpoints.

Usage: srvctl modify scan_listener

{-u|-p [TCP:]<port>[/IPC:<key>][/NMP:<pipe>][/TCPS:<sport>] [/SDP:<port>]}
    -u Update SCAN listeners to match the number of SCAN VIPs
    -p [TCP:]<port>[/IPC:<key>][/NMP:<pipe>][/TCPS:<sport>] [/SDP:<port>]
           SCAN Listener endpoints
    -h Print usage

You should keep the current configuration for your change record manual for the cluster. The cluster can be queried using the srvctl config listener command, as shown in the following example:

[oracle@london1 log]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

Now you need to stop the SCAN listener, change the configuration, and then start it up again:

[oracle@london1 log]$ srvctl stop scan_listener
[oracle@london1 log]$ srvctl modify scan_listener -p  1526

[oracle@london1 log]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1526
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1526
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1526
[oracle@london1 log]$ srvctl start scan_listener
[oracle@london1 log]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node london2
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node london1
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node london1

The SCAN can take up to three IP addresses that have to resolve round-robin style in DNS. If the system has been set up with less than the maximum possible number of IP addresses, you can easily add more IP addresses after the installation. First, you need to change the DNS so it reflects all the IP addresses. Second, you check the current IP address list, as in the following example:

oracle@london1:˜> host cluster1.example.com
cluster1.example.com has address 172.17.1.205
cluster1.example.com has address 172.17.1.206
cluster1.example.com has address 172.17.1.207

As the Grid software owner (oracle, in this example, but it could be any operating system account), you need to stop the SCAN and SCAN listener:

[oracle@london1 ˜]$ srvctl stop scan_listener
[oracle@london1 ˜]$ srvctl stop scan

Next, you should verify that the both SCAN and SCAN listener have indeed stopped:

[oracle@london1 ˜]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is not running

[oracle@london1 ˜]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is not running

With the SCAN and SCAN listener down, modify the SCAN as root:

[root@london1 ˜]# srvctl modify scan -n cluster1.example.com
[root@london1 ˜]# srvctl config scan
SCAN name: cluster1, Network: 1/172.17.1.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /cluster1.example.com/172.17.1.205
SCAN VIP name: scan2, IP: /cluster1.example.com/172.17.1.206
SCAN VIP name: scan2, IP: /cluster1.example.com/172.17.1.207

Updating the SCAN listener itself is straightforward:

[root@london1 ˜]# srvctl modify scan_listener -u
[root@london1 ˜]# srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

Finally, you need to restart the SCAN and SCAN listener:

[root@london1 ˜]# srvctl start scan
[root@london1 ˜]# srvctl start scan_listener

The log files for the SCAN listeners are somewhat hidden in the $GRID_HOME/log directory. To view them, you need to point the ADR_BASE to $GRID_HOME/log, as in the following example:

[oracle@london1 ˜]$ adrci

ADRCI: Release 11.2.0.1.0 - Production on Sat Jul 3 11:26:11 2010

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

ADR base = "/u01/app/oracle"
adrci> set base /u01/app/crs/log
adrci> show homes
ADR Homes:
diag/tnslsnr/london1/listener_scan1
diag/tnslsnr/london1/listener_scan2
diag/tnslsnr/london1/listener_scan3
diag/clients/user_root/host_3052993529_76
diag/asm/+asm/+ASM1
adrci> set home diag/tnslsnr/london1/listener_scan1

With the ADR base set, you can view the log. Bear in mind that the SCAN listener can migrate. Even though there is a log file, that does not mean that this particular SCAN listener is currently running on that node. The srvctl status scan_listener command tells you which node a particular SCAN listener is running on.

Prior to Oracle 11.2, voting disks were stored on block devices or, alternatively, on raw devices. Block devices such as /dev/mapper/ocrvote1p1 provided an easier interface to the voting disks because no (deprecated) raw devices had to be created during the operating system's boot process. Nevertheless, and as you saw in the earlier "Troubleshooting" section, the CSSD process requires specific permissions and ownership of the voting disks. You set that ownership either through udev or the rc.local script.

In Oracle 10.2 and 11.1, we recommended that you create multiple voting disks to eliminate a potential single point of failure. Oracle 10.1 did not support this feature; in that version, there was always only one voting disk. Even with multiple voting disks, you should still back these disks up on a regular basis and test your recovery procedures. To back up a voting disk for these releases prior to version 11.2, you could use the dd operating system utility:

[root@londonl ˜]# dd if=voting_disk of=backup_file_name

You did not need to shut down the database or Oracle Clusterware before backing up the voting disk. Although the voting disks were (in theory) identical, backing up each configured voting disk was recommended. Because voting disk locations were configured in the OCR, modifying them during any recovery process was difficult. Therefore, the most straightforward strategy was to back up all of the voting disks and, if recovery was necessary, restore each copy to its original location.

To recover the voting disks, you had to ensure that all databases and Oracle Clusterware were shut down. You could then restore the voting disks using the dd operating system utility:

[root@londonl ˜]# dd if=backup_file_name of=voting_disk

With Oracle 11.2, you no longer need to back up voting disks because the contents of the voting disks are automatically backed up in the OCR. When a new voting disk is added, the contents of the voting disk will be copied to the new voting disk.

[root@london1 ˜]# /etc/init.d/oracleasm createdisk OCRVOTE1 /dev/mapper/ocr1p1
[root@london1 ˜]# /etc/init.d/oracleasm createdisk OCRVOTE2 /dev/mapper/ocr2p1
[root@london1 ˜]# /etc/init.d/oracleasm createdisk OCRVOTE3 /dev/mapper/ocr3p1

SQL> create diskgroup OCRVOTE normal redundancy disk
 2   'ORCL:OCRVOTE1','ORCL:OCRVOTE2','ORCL:OCRVOTE3';

Diskgroup created

SQL> alter diskgroup OCRVOTE set attribute  'compatible.asm'='11.2';

Diskgroup altered.

[root@london1: ˜]# crsctl replace votedisk '+OCRVOTE'

As noted in the "Storing Cluster Information with the Oracle Cluster Registry" section earlier in this chapter, voting disks and the OCR are located on shared storage. Your options are to use either ASM or a supporting cluster file system. Using block or raw devices is not recommended for Oracle 11.2. The fact that block and raw devices are deprecated should be a strong indicator of the future direction of the tool. During normal operation, you do not normally have to deal with the OLR and OCR because they work in the background, enabling the cluster to function correctly. The following sections will cover how to deal with OLR and OCR corruption, while the last section explains how to move the OCR into ASM.

CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.

Enterprise Linux Enterprise Linux Server release 5.4 (Carthage)
Kernel 2.6.18-164.el5 on an x86-64

[root@london1 ohasd]# ocrcheck -local
Status of Oracle Local Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2232
         Available space (kbytes) :     259888
         ID                       : 1022831156
         Device/File Name         : /u01/app/crs/cdata/london1.olr
                                    Device/File integrity check failed

         Local registry integrity check failed

         Logical corruption check bypassed

[root@london1 ohasd]# ocrconfig -local -restore 
/u01/app/crs/cdata/london1/backup_20091010_211006.olr
PROTL-19: Cannot proceed while the Oracle High Availability Service is running

[root@london1 ohasd] init 2

[root@london1 ˜]# ocrconfig -local -restore 
> /u01/app/crs/cdata/london1/backup_20091010_211006.olr

CRS-4123: Oracle High Availability Services has been started.

[root@london1 ohasd]# ocrcheck -local
PROTL-602: Failed to retrieve data from the local registry
PROCL-26: Error while accessing the physical storage Operating System error
  [No such file or directory] [2]

[root@london1 client]# cat ocrconfig_11254.log
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production
Copyright 1996, 2009
Oracle. All rights reserved.
2010-06-25 09:21:24.866: [ OCRCONF][3905760784]ocrconfig starts...
2010-06-25 09:21:24.867: [  OCROSD][3905760784]utopen:6m':failed in stat OCR
file/disk /u01/app/crs/cdata/london1.olr, errno=2, os err
string=No such file or directory
2010-06-25 09:21:24.867: [  OCROSD][3905760784]utopen:7:failed to open
any OCR file/disk, errno=2,os err string=No such file or directory
[ default][3905760784]u_set_gbl_comp_error: OCR context was NULL
2010-06-25 09:21:24.867: [  OCRRAW][3905760784]phy_rec:1:could not
open OCR device
2010-06-25 09:21:24.867: [ OCRCONF][3905760784]Failed to restore OCR/OLR from
[/u01/app/crs/cdata/london1/backup_20100625_085111.olr]
2010-06-25 09:21:24.867: [ OCRCONF][3905760784]Exiting [status=failed]...

[root@london1 client]# touch /u01/app/crs/cdata/london1.olr
[root@london1 cdata]# chown oracle:oinstall london1.olr
[root@london1 ˜]# ocrconfig -local -restore 
> /u01/app/crs/cdata/london1/backup_20100625_085111.olr

Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996,
2009 Oracle. All rights reserved.
2010-06-25 09:42:01.887: [ OCRCONF][2532662800]ocrconfig starts...
2010-06-25 09:42:03.671: [ OCRCONF][2532662800]Successfully restored OCR and set
  block 0
2010-06-25 09:42:03.671: [ OCRCONF][2532662800]Exiting [status=success]...

[oracle@london1 ˜]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[root@london1: bin]# ./ocrconfig -add +OCRVOTE
[root@london1 ˜]# ocrcheck
Status of Oracle Cluster Registry is as follows :
 Version                  :          3
 Total space (kbytes)     :     292924
 Used space (kbytes)      :       8096
 Available space (kbytes) :     284828
 ID                       :  209577144
 Device/File Name         : /dev/raw/raw1
 Device/File integrity check succeeded
 Device/File Name         : +OCRVOTE
 Device/File integrity check succeeded

 Device/File not configured

 Device/File not configured

 Device/File not configured

 Cluster registry integrity check succeeded

 Logical corruption check succeeded

[root@london1 ˜]# ocrconfig -delete /dev/raw/raw1
[root@racupgrade1 ˜]# ocrcheck

Status of Oracle Cluster Registry is as follows :
Version                  :          3
Total space (kbytes)     :     292924
Used space (kbytes)      :       8096
Available space (kbytes) :     284828
ID                       :  209577144
Device/File Name         :    +OCRVOTE
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded