To create a default partitioning scheme, begin by selecting the option from the dropdown menu to remove Linux partitions on selected drives. Next, create a default layout to ensure that only the internal system drives are selected. For example, you don't want to select external drives that may be shared with the other nodes in the cluster. Also, select the checkbox to review and modify the partitioning layout. You should now see something like what is shown in Figure 6-1. Click Next when you have everything right.

Figure 6.1. The Linux partitioning selection

A warning dialog will be displayed, indicating that proceeding will remove all partitions and data on the selected drives. Click Yes to proceed. Next, you will see the disk setup page, which includes options for partitioning (see Figure 6-2). The top pane displays the drives available and their corresponding sizes; the bottom pane shows the default partitioning scheme.

Figure 6.2. The disk setup

The default partitioning scheme illustrates how Oracle Enterprise Linux creates physical partitions and uses the Logical Volume Manager (LVM) to create a single Logical Volume Group named VolGroup00. This group is concatenated from the selected drives and their partitions. Under these partitions, the two logical volumes LogVol00 and LogVol01 are created for the / (root) partition and swap space, respectively. The first available drive also includes a /boot partition; this partition cannot reside on a Logical Volume because it cannot be read by the GRUB boot loader. The following snippet shows the result of a default partitioning scheme after installation:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      886G  2.4G  838G   1% /
/dev/sda1              99M   13M   82M  13% /boot
tmpfs                 7.9G     0  7.9G   0% /dev/shm

We recommend that you protect your server disk drives by using a form of onboard RAID storage (see Chapter 4 for more information on RAID). If onboard RAID is not available, you should consider the options for manually configuring both RAID and the Logical Volume Manager during the disk partitioning stage, as explained later in this section.

You might wish to customize the default partitioning scheme or to implement RAID and LVM manually. If so, then you have a number of options for proceeding. You also face some restrictions related to partitioning disk devices suitable for a node of Oracle Database 11g Release 2 RAC. As with the default configuration on systems based on Oracle Enterprise Linux 5 on x86 and x86-64, a minimum partitioning scheme without LVM must include at least the root partition (/) and swap partition (swap). If LVM is used, then the /boot partition must not be created under LVM, for the reason given in the previous section.

The choice of partitioning formats is dependent on whether the interface between the hardware and operating system is BIOS- or EFI-based; you will learn about your partitioning options in the upcoming sections.

The most commonly used partitioning format on BIOS-based systems is the MS-DOS–based master boot record (MBR) format. Typically found on x86 and x86-64 systems, a BIOS-based MBR format allows up to four primary partitions to be created on each disk. One of these primary partitions can be extended and subdivided into logical partitions. The logical partitions are chained together. Theoretically, this means the maximum number of logical partitions is unlimited. However, for SCSI disks, Linux limits the total number of partitions to 15. A 2TB limit is also enforced within the MBR formatting scheme. And because the fdisk command only supports MBR formatting, fdisk also has a limit of 2TB. The MBR format stores one copy of the partition table. It also stores some system data in hidden or unpartitioned sectors, which is an important factor when trying to ensure that disk partitions are correctly aligned with external storage arrays. You'll learn more about this topic later in this chapter.

Because Oracle Enterprise Linux 5 is based upon the 2.6 Linux kernel, SCSI commands support 64-bit block addresses. This means they can support individual devices beyond 2TB in size. However, the Host Bus Adapter (HBA), device driver, and the storage itself must all also support 64-bit block addressing before you can use it. If any of these components are limited to 32-bit block addresses, then a single device will also be limited to 2TB in size, regardless of the formatting scheme. Potential device and partitioning limitations aside, both the default ext3 file system and the OCFS2 file system support a maximum file system size of 16TB and a maximum individual file size of 2TB. Also, when using LVM, the maximum logical volume size is 16TB on x86 and 8EB (exabytes) on x86-64. Therefore, if you wish to create an ext3 file system larger than 2TB on a system with MBR formatting, it is possible to use LVM directly on a single disk device that has not been partitioned. It is also possible to concatenate a number of 2TB partitions with LVM, and then create the ext3 file system on the resulting logical volume.

EFI-based systems were introduced with Itanium-based systems, but they are now present on some of the more recent x86 and x86-64 systems. EFI-based systems also support MBR formatting; however, you may consider using the GPT format instead for the advantages that this scheme brings. First, the GPT format is not limited to the primary and logical partitioning of the MBR; instead, it can support up to 128 partitions per disk.

Subject to using 64-bit SCSI addressing and a 2.6 kernel–based system, the GPT format can be up to 18EB in size—considerably larger than the 2TB limit. In addition, the GPT format uses primary and backup partition tables and checksum fields for redundancy and improved integrity. This can help you protect against partition table corruption. GPT partition tables also do not store any data in hidden sectors, which makes alignment offsets for external storage arrays unnecessary. Because the fdisk command does not support the GPT format, the parted command must be used instead.

The ability to implement GPT partitions after installation is not restricted to EFI-based systems. However, when using an unmodified version of GRUB, the default bootloader on x86 and x86-64, it is not possible to boot from a GPT partition. Therefore, GRUB must always be installed on an MBR-formatted partition, which means it is not possible to boot from a device with partitions larger than 2TB.

It's possible you will want to customize the default partitioning scheme. Many system administrators prefer to partition the system with a number of partitions for distinct / (root device), swap, /boot, /home, /var, /usr /home, ORACLE_BASE, and other site-dependent configurations. Two of the main historical factors for adopting this schema were to reduce both the risk of corruption and the time required to complete file system checks in the event of a system restart when using a nonjournaled file system. We have found that Linux-based Oracle Database 11g Release 2 RAC nodes advanced, journaled file systems such as ext3 provide the most flexible approach. Therefore, we recommend the minimal configuration provided by the default installation. When installing the Oracle software, the database software is usually located in a directory tree below the ORACLE_BASE directory. Under the Optimal Flexible Architecture (OFA), the first mount point is the /u01 directory. This means that, in the default configuration, the /u01 directory can be created in the root partition. Thus we recommend that you have at least 10GB of disk space available for the Oracle software installation.

The space required for the boot device is dependent on the number of kernels compiled and configured for booting. Oracle Database 11g Release 2 RAC nodes only require the standard precompiled kernel, so space utilization rarely exceeds 30MB. The actual space used is dependent on the hardware architecture. However, if you're customizing the partition layout, then the 100MB allocated by the default configuration is sufficient.

Oracle's recommendations for Linux vary. For example, it recommends setting the swap size anywhere from equal to the RAM to four times the RAM. Consider the following documented example for Oracle 11g Release 2, which is typical. For systems with between 1GB and 2GB of RAM, Oracle recommends swap space of 1.5 times the size of RAM. For systems with between 2GB and 16GB of RAM, Oracle recommends that you set the swap size to equal the size of RAM. And for systems with more than 16GB of RAM, Oracle recommends 16GB of swap space. The size of a single swap partition in Oracle Enterprise Linux 5 is determined by the limits on the partition or the logical volume size, as discussed previously.

Swap space can be regarded as a disk-based component of virtual memory. In Linux, unlike some other UNIX operating systems, virtual memory is the sum total of RAM plus swap space. Recommendations to make the swap size one or two times the size of the RAM are often based on the requirements of traditional UNIX operating systems. This swap configuration was required because an executable program had been loaded into both swap space and RAM at the same time. Oracle's guidelines on swap allocation tend to reflect these traditional requirements. Linux, on the other hand, maintains swap space for demand paging. When an executable is loaded on Linux, it is likely that a portion of the executable will not be required, such as the error-handling code. For efficiency, Linux only loads virtual memory pages when they are used. If RAM is fully utilized and one process requires loading a page into RAM, Linux will write out the page to swap space from another process, according to a Least Recently Used (LRU) algorithm.

The performance of disk-based storage is orders of magnitude slower than memory. Therefore, you should plan to never be in the position of having pages written to swap space due to demand for RAM from competing processes. Oracle should always run in RAM, without exception. As a basic rule, you do not need a significant amount of swap space for Oracle on Linux, assuming there is enough memory installed. Every rule has an exception, however. Linux tends to be reasonably aggressive in managing swap space, maintaining as much free RAM as possible. It does this by writing out memory pages to swap space that have not been recently used, even if there is no current demand for the memory at the time. This approach is designed to prevent a greater impact on performance than doing all of the maintenance work when the space is required. This approach also frees up RAM for other potential, such as the disk buffer. The behavior of the swapping activity can be modified with the kernel parameter, vm.swappiness. A higher value means the system will be more active in unmapping mapped pages, while a lower value means that swapping activity will be reduced. However, the lower amount of swap activity comes at the expense that RAM that could potentially be freed.. The default value is 60, which you can see in /proc/sys/vm/swappiness. Note that memory allocated as huge pages is not swapped under any circumstances. The vm.panic_on_oom kernel parameter also affects virtual memory. By default this parameter is set to 0. If all virtual memory is exhausted (including the swap space), the operating system will recover memory by terminating processes with high levels of memory consumption, but low CPU activity. If the preceding kernel parameter is set to 1, a kernel panic will occur, terminating all system activity. This is the preferred approach under a RAC environment. However, you should endeavor to configure the virtual memory to ensure that the condition that provokes a kernel panic is never reached.

With this information, you can intelligently size the swap space for Oracle on Linux. You can also evaluate the correct choice between what may appear to be conflicting recommendations. First, any requirement for swap space that is two to four times the size of the RAM is an over specification based on traditional UNIX requirements. Second, although one given recommendation is for swap of 1.5 times the size of the RAM with installed RAM of 1GB to 2GB, we recommend that you install at least 2GB or RAM per installed CPU socket. Therefore, if you're meeting these guidelines, you should not plan for more swap space than you have RAM.

For the basic operating environment, swap space should not be utilized during normal usage. With enough memory, even a single 2GB swap partition will often suffice. The exception occurs in the case where many Oracle user processes or other processes are created and intersperse active periods with idle time. In this scenario, it is necessary to allocate swap space to account for cases where these processes are temporarily paged out of memory. This type of application profile may actively use more than 2GB of swap space without having any impact on overall performance; also, it can potentially increase performance by freeing RAM that is not being used. However, it is also likely under this application profile that an Oracle shared server configuration would be more efficient.

The swap space needed will depend on application requirements. Given sufficient disk space, we recommend adopting the simplest single starting point for swap space allocation for Oracle on Linux. That is, you should plan for an amount of swap space equal to the amount of RAM installed in the system. If you know that you will not create a large number of processes on the system, then you may reduce the allocation accordingly. If you do have a large number of processes but sufficient RAM, then you can evaluate using vm.swappiness to reduce the usage of swap space. If additional swap space is required, you can use the mkswap command at a later point in time. One benefit of the mkswap command: It can also be used when the system is operational. Conversely, if you have a large RAM configuration, then resizing the swap space by many tens or hundreds of gigabytes will not be detrimental to performance, as long as your swap space is equivalent to your installed RAM. However, this approach may use disk space unnecessarily. If this is the case, and you wish to conserve disk space, then you may size a swap partition smaller than the installed amount of RAM. Oracle advises a 16GB swap partition for configurations of more than 16GB of RAM. However, this does not mean that you cannot have a swap partition of more than 16GB if you wish. For example, you might use more than 16GB of swap space if you have enough disk capacity that conserving the disk space between 16GB and your available RAM size is not a concern.

If you do not have a system with onboard RAID, you may wish to consider protecting your Linux operating systems disk drives with an add-in RAID adapter. We recommend a hardware RAID solution wherever possible because it is far superior solution to its software equivalent. If you cannot configure a hardware RAID solution and wish to create a partitioning scheme with some level of redundancy, then there are two ways in which this can be achieved. First, you can use software RAID, and then use the configured RAID device as the underlying device for an LVM configuration. Second, you can use LVM to mirror devices. As previously noted, the /boot partition cannot reside on an LVM volume. When mirroring drives with a RAID 1 configuration (see Chapter 4 for more information on RAID), LVM reads from one side of the mirror only. In a software-based RAID solution, a RAID 1 configuration reads from both mirrored devices, which results in higher levels of performance.

It is also possible to configure the /boot partition on a RAID configuration with RAID 1 that uses at least one of the first two drives on the system. If the /boot directory is not a separate partition from the / (root) directory, then these conditions apply to the / (root) directory instead. For these reasons, we recommend software RAID over LVM if you're manually configuring redundancy. However—and this cannot be overemphasized—you should opt for hardware RAID wherever possible. Hardware-based RAID can protect against a drive failure transparently, but software-based software RAID may require rebooting the server if a drive fails. The advantage of the scenario just described is this: if the boot drive is in a software RAID configuration, then downtime is restricted to the time required for the reboot to take place, as opposed to the time required to restore from a backup.

The following example shows a software RAID 1 equivalent of the default partitioning scheme on the first two drives in the system. To begin configuring software RAID on the disk partitioning page, select Create custom layout from the dropdown menu and press Next. On the disk setup page, check whether there are any existing partitions or logical volumes. If so, use the Delete button to remove the disk configuration, so that the chosen drives can display free space (see Figure 6-3).

Figure 6.3. Setting up software-based RAID 1

Next, select the RAID button. This will present you with a single option: Create a software RAID partition. Press OK to display the Add Partition window. As noted previously, it is not possible for the /boot partition to reside on a logical volume. For that reason, if the / (root) and swap are to utilize LVM, then you need to independently create four software RAID partitions. Create the first two partitions on /dev/sda and /dev/sdb and make them 100MB each. Next, create two more partitions, one on each drive. This enables you to utilize the rest of the maximum allowable size. Figure 6-4 shows the created RAID partitions.

Figure 6.4. Creating the RAID partitions

Press the RAID button again to make the following option available: Create a RAID device. This screen also shows that you now have four software RAID partitions you can use. Select the

Create a RAID device option to display the Make RAID Device window. Next, choose the following options: /boot for the Mount Point, ext3 for the File System Type, md0 for the RAID Device, and RAID 1 for the RAID Level. For the RAID members, select the two 100MB software RAID partitions created previously, and then press OK. /boot is created on /dev/md0. Repeat the process to display the Make RAID Device window. However, this time you should select a File System Type of physical volume (LVM) from the RAID Device md1 with RAID Level 1. Also be sure to select the two remaining RAID Members. Click OK to create the physical volume. Now press the LVM button to display the Make LVM Volume Group window, and then press the Add button to create the logical volumes for the / (root) and swap (see Figure 6-5).

Figure 6.5. Creating the LVM Volumes

Press OK to complete the disk partitioning. At this point, you have created the default configuration with RAID 1 across the first two drives in the system (see Figure 6-6).

Figure 6.6. The software RAID configuration

After installation, the partitioning layout will look something like this:

[root@london2 ˜]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      433G  1.3G  409G   1% /
/dev/md0               99M   13M   82M  14% /boot
tmpfs                 7.9G     0  7.9G   0% /dev/shm

After completing your chosen partitioning scheme, click Next to save the partitioning map and advance to the Boot Loader Configuration page on x86 and x86-64 systems. The disks will not be physically partitioned at this point. This won't occur until after all of the installation information has been collected, but before the installation of packages has commenced.

A YUM server provides a repository for RPM packages and their associated metadata. This makes installing the packages and their dependencies straightforward. Oracle provides a public YUM server at http://public-yum.oracle.com, but its server provides only the packages you have already downloaded on the installation media. Subscribers to the Unbreakable Linux Network can access additional security updates and patches on top of the content available on the public YUM server. If you do not have access to the Unbreakable Linux Network or you do not wish to use the public YUM server, it is a simple enough process to configure your own from the installation media.

Previously in this chapter, we explained how to configure VSFTP to make the installation media available across the network. Fortunately, this installation media also contains the YUM repository metadata. For example, the metadata for the Server RPMs is located under the Server/repodata directory in the file repomd.xml. This means you do not have to create additional metadata by building a custom repository with the createrepo command. You can test the readiness of the installation media for use as a YUM repository with the wget command. This should enable you to retrieve a selected file without any additional steps, as shown in this example:

[root@london1 tmp]# wget ftp://ftp1.example.com/pub/el5_4_x86_64/Server/oracle-validated*
...
2009-09-17 13:54:06 (6.10 MB/s) - `oracle-validated-1.0.0-18.el5.x86_64.rpm.1' saved [15224]

If you prefer, your local YUM repository can also be based on a mounted DVD-ROM. However, this repository will be local to an individual server. With the repository available, the next step is to configure YUM on the client. To do this, edit the file /etc/yum.conf and add the following section to the end of the file:

[Server]
name=Server
baseurl=ftp://ftp1.example.com/pub/el5_4_x86_64/Server/
gpgcheck=0
enabled=1

Setting the gpgcheck option to the value of 1 ensures that all packages are signed and that YUM will verify the signatures. As the preceding example shows, it is not strictly necessary to set this value if the YUM server is private and based on the Oracle installation media. However, you should set the value to 1 when using public YUM servers. The baseurl property specifies the location of the server RPMs. In the preceding example, baseurl property points to the FTP Server. If a local DVD is used, then baseurl takes the following form: baseurl=file:///media/disk/Server/. In this case, baseurl reflects the location of the mounted DVD. It is also possible to copy the DVD installation media to the local server, mount it with the loop option, and specify this as the file location. However, this approach is likely to be considerably more time consuming than simply configuring the FTP Server. To check the configuration, run the command yum list. If correctly configured, this command will list the RPMs shown as installed, as well as the ones listed under the Server category.

The system is now ready for you to install the Oracle Validated RPM with the yum install oracle-validated command. Once all the dependencies have been resolved, it is necessary to confirm the installation at the Is this ok prompt, as shown in the following truncated output:

[root@london1 etc]# yum install oracle-validated
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
...
Is this ok [y/N]: y
...
Installed:
  oracle-validated.x86_64 0:1.0.0-18.el5

Dependency Installed:
  compat-db.x86_64 0:4.2.52-5.1

compat-gcc-34.x86_64 0:3.4.6-4
  compat-gcc-34-c++.x86_64 0:3.4.6-4
  elfutils-libelf-devel.x86_64 0:0.137-3.el5
  elfutils-libelf-devel-static.x86_64 0:0.137-3.el5
  gcc.x86_64 0:4.1.2-46.el5
  gcc-c++.x86_64 0:4.1.2-46.el5
  gdb.x86_64 0:6.8-37.el5
  glibc-devel.i386 0:2.5-42
  glibc-devel.x86_64 0:2.5-42
  glibc-headers.x86_64 0:2.5-42
  kernel-headers.x86_64 0:2.6.18-164.el5
  libXp.i386 0:1.0.0-8.1.el5
  libaio-devel.x86_64 0:0.3.106-3.2
  libgomp.x86_64 0:4.4.0-6.el5
  libstdc++-devel.x86_64 0:4.1.2-46.el5
  sysstat.x86_64 0:7.0.2-3.el5
  unixODBC.x86_64 0:2.2.11-7.1
  unixODBC-devel.x86_64 0:2.2.11-7.1

Complete!

Once the Oracle Validated RPM installation completes, all the RPM packages and system configuration steps required for an Oracle Database 11g Release 2 RAC installation have also been completed. For example, the required user and groups have been created, and the necessary kernel parameters have been set. You can find the installed packages listed in /var/log/yum.log. In some combinations of the Oracle Validated RPM and Oracle Database 11g Release 2 on 64-bit systems, the OUI will report that some of the required packages are missing. For example, packages commonly listed as missing include libaio-devel, unixODBC, and unixODBC-devel. The error typically looks something like this:

This is a prerequisite condition to test whether the package
"libaio-devel-0.3.106" is available on the system. (more details)
  Check Failed on Nodes: [london2, london1]

This warning occurs because, on x86-64 architecture systems, the OUI checks for the presence of both the x86-64 and i386 packages, whereas some versions of the Oracle Validated RPM install only the 64-bit versions. The package versions can be verified with the following snippet:

[root@london1 ˜]# rpm -q --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})
" 
libaio-devel 
unixODBC 
unixODBC-devel
libaio-devel-0.3.106-3.2 (x86_64)
unixODBC-2.2.11-7.1 (x86_64)
unixODBC-devel-2.2.11-7.1 (x86_64)

You can prevent 32-bit RPMs from being reported as missing by the OUI by installing them directly with the rpm command or with the yum command. Doing so ensures that the requested architecture is fully specified by the yum install command, as in this example:

[root@london1 ˜]# yum install unixODBC-2.2.11-7.1.i386 unixODBC-devel-2.2.11-7.1.i386 libaio-devel-0.3.106-3.2.i386
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package libaio-devel.i386 0:0.3.106-3.2 set to be updated
---> Package unixODBC.i386 0:2.2.11-7.1 set to be updated
---> Package unixODBC-devel.i386 0:2.2.11-7.1 set to be updated
--> Finished Dependency Resolution
...
Installed:
  libaio-devel.i386 0:0.3.106-3.2           unixODBC.i386 0:2.2.11-7.1
  unixODBC-devel.i386 0:2.2.11-7.1

Complete!

You can now verify the subsequent presence of the correct architectures for the packages:

[root@london1 ˜]# rpm -q --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})
" 
> libaio-devel 
> unixODBC 
> unixODBC-devel
libaio-devel-0.3.106-3.2 (x86_64)
libaio-devel-0.3.106-3.2 (i386)
unixODBC-2.2.11-7.1 (x86_64)
unixODBC-2.2.11-7.1 (i386)
unixODBC-devel-2.2.11-7.1 (x86_64)
unixODBC-devel-2.2.11-7.1 (i386)

On NUMA based systems you may also wish to install the numactl-devel RPM package to enable full NUMA support, see chapter 4 for more details. The preceding YUM configuration provides only the Server RPM packages required by the Oracle Validated RPM. However, if you wish to continue to using YUM as a method for RPM installation, then it is also necessary to add additional categories in /etc/yum.conf under the Cluster, ClusterStorage, and VT headings (and their corresponding RPM directories). Adding these categories lets you ensure the full availability of all of the RPM packages for YUM that are present on the installation media. Alternatively, you may use the Oracle public YUM server.

If you are a customer of the ULN, then once you have registered your system, you may also use the up2date command to install and run the Oracle Validated RPM and maintain the operating system. However, we recommend that you avoid using this command because it has been superseded by YUM. The up2date command provides functionality similar to YUM, albeit with different syntax. Specifically, this command can automatically resolve dependencies of the installed RPM packages.

Although the up2date command is available in Oracle Enterprise Linux 5, it no longer implements the original up2date configuration. For the sake of backward compatibility, it has essentially been re-implemented as a wrapper around YUM commands by using YUM repositories. Consistency in configuration across nodes is essential for a clustered environment, so we recommend that you standardize on using only YUM commands for installing packages and resolving dependencies. To implement this approach, we also recommend that you use a local YUM repository (as described previously in this chapter) and synchronize this repository with the systems provided by the ULN. The ULN's YUM Repository Setup explains how to do this. You can then use your own YUM repository as the direct source for updating the packages on the nodes in your cluster. The benefit of this approach: You won't need to rely on an external repository where an interruption in access could render the operating system configurations on the nodes in your cluster inconsistent.

In some environments, one group of users will install and maintain Oracle software, while a separate group of users will utilize the Oracle software to maintain the database. Maintaining separate groups requires the creation of user groups to draw a distinction between two sets of permissions. The first set lets a group install Oracle software, which requires access to the Oracle Universal Installer (OUI) and oraInventory. The second set grants permissions for a group to accomplish general database administration tasks. These two groups are typically named oinstall and dba.

In most RAC environments, the DBA installs, maintains, and uses the Oracle software. In such cases, the oinstall group may appear superfluous. However, the Oracle inventory group is included in some preinstallation system checks; therefore, using this group for Oracle software installation is a good practice. The oracle-validated-verify script creates the dba and oinstall groups using the commands groupadd dba and groupadd oinstall, respectively. For a new installation, these groups will have the group IDs of 500 and 501, respectively. If you create these groups manually, you may specify the group ID on the command line:

[root@london1 root] # groupadd -g 500 dba

Any group ID may be used, but the same group ID should be used for the dba and oinstall groups on all nodes in the cluster. For this reason, you should ensure that no other groups are created on all the nodes in the cluster until the oracle-validated-verify script has been run.

After creating the groups, the script uses the following command to create the oracle user as a member of the dba and oinstall groups:

useradd -g oinstall -G dba -d /home/oracle -p $encpasswd oracle

Also, note how the preceding line sets the encrypted password environment variable to oracle.

As when creating a group, there is no cluster awareness of the user ID when creating a user. If this is the first user created after installation, then the user ID will be 500. Again, you should ensure that this is the same on all the nodes in the cluster. You can run the following command to create the oracle user manually:

[root@london1 root] # useradd -u 500 -g oinstall -G dba -d /home/oracle oracle

The preceding line creates a specific user ID, where the user is also a member of the dba and oinstall groups. The oracle user will be appended to the /etc/passwd file, and it will have the group ID of the dba group. The user will also be added to the dba and oinstall groups in the /etc/group file, as well as to the default home directory location of /home/oracle. The oinstall group is the primary group, while the dba group is a supplementary group. The /home directory can be in any location, but it's good practice to keep it distinct from the ORACLE_HOME directory, the default location for Oracle software. By default, the user is created with the default bash shell, which can be modified by using the useradd command with the -s option.

You can verify that the oracle user and dba group have been correctly configured using the id command, as in this example:

[oracle@london1 ˜]$ id oracle
uid=500(oracle) gid=501(oinstall) groups=501(oinstall),500(dba)

After the oracle-validated-verify script creates the account, you should use the passwd command to change the default, unsecured password for the oracle user to something more secure:

[root@london1 root]# passwd oracle
Changing password for user oracle
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

To complete the user configuration, you must also manually configure the environment variables for the oracle user; you'll learn how to do this later in this chapter.

After configuring the Oracle user and groups, the Oracle Validated RPM configures the Linux kernel parameters to meet Oracle's recommendations. The updated kernel parameters are set in the file /etc/sysctl.conf; you can find a copy of the original at /etc/sysctl.conf.orabackup. Some of the kernel parameters are set according to value of the total memory in the system, which is retrieved from/proc/meminfo. Existing parameters are modified in place, while additional parameters are added to the end of the /etc/sysctl.conf. These additions are based on the values contained in oracle-validated.params. You can also manually edit the /etc/sysctl.conf file, save its contents, and then apply those settings using the following command:

[root@london1 root]# sysctl -p

The kernel parameters and their correct values for 11g Release 2 are shown in Table 6-6.

The recommended Linux kernel parameter settings are very much general guidelines, but understanding these parameters and how applicable they are to your particular system is essential. You may still need to change some of the updated parameters for your configuration. These parameters can be loosely grouped into five classifications: shared memory, semaphores, network settings, message queues and open files. We will examine each of these classifications next.

When an Oracle instance starts, it allocates shared memory segments for the SGA. This allocation applies to RAC instances in exactly the same way as a single instance. The SGA has fixed and variable areas. The fixed area size cannot be changed, but the variable area, as the name implies, will vary according to the Oracle database parameters set in the SPFILE or init.ora file.

Within Oracle Database 11g Release 2 RAC, the memory configuration of the SGA and PGA—and consequently, the appropriate kernel parameters—are dependent on whether Automatic Memory Management (AMM), Automatic Shared Memory Management (ASMM), or even manual memory management is used. If AMM is used, then Oracle manages both SGA and PGA from one memory allocation set, according to the Oracle parameters MEMORY_TARGET and MEMORY_MAX_TARGET. If ASMM if used, however, the two most significant Oracle parameters to be aware of for sizing the SGA are SGA_TARGET and SGA_MAX_SIZE. And you must also know that session-level memory set with the PGA_AGGREGATE_TARGET parameter is not allocated from the SGA.

Setting the SGA_TARGET parameter enables Oracle to automatically set the parameters for the variable components of the SGA, such as db_cache_size and shared_pool_size; doing this does not necessarily require manually sizing these components individually. However, manually sized components such as the redo log buffer, streams pool, and nondefault database caches (e.g., the keep, recycle, and nondefault block size caches) are given priority for the allocated shared memory. Subsequently, the balance of the memory is allocated to the automatically sized components.

If you do choose to set these parameters manually, the kernel parameters required are the same as when ASMM is used. You can view the allocated shared memory segments using the ipcs -m command. The following example shows a single shared memory segment created for an SGA_TARGET and SGA_MAX_SIZE that is set to 14GB. The additional shared memory segment in this case is a single 4KB page for the ASM instance, where shmid identifies the memory mapped files in /dev/shm for the use of AMM:

[oracle@london1 ˜]$ ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x6422c258 32768      oracle    660        4096       0
0x10455eac 98305      oracle    600        15034482688 31

You can also view the defined shared memory limits with the ipcs -lm command:

[root@london1 ˜]# ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 47996078
max total shared memory (kbytes) = 32388608
min seg size (bytes) = 1

As introduced in Chapter 4, the choice of Oracle memory management determines the operating system call to allocate memory. The shmget() system call is used for ASMM, while the mmap() system call is used for AMM. Failure to set the appropriate memory-related kernel parameters correctly could result in the Oracle instance failing to start with the correct kernel.shmall, kernel.shmmax, and kernel.shmmni parameters for ASMM. The section on Open Files should be reviewed for the parameters relevant to using AMM.

kernel.shmmax = 2147483648

kernel.shmmax =  10737418240

kernel.shmall = 1073741824

Semaphores are used by Oracle for resource management. They serve as a post/wait mechanism by enqueues and writers for events, such as free buffer waits. Semaphores are used extensively by Oracle, and they are allocated to Oracle at instance start-up in sets by the Linux kernel. Processes use semaphores from the moment they attach to an Oracle instance waiting on semop() system calls. Semaphores are set with the sysctl command or in the /etc/sysctl.conf file. This behavior differs from that of other Unix operating systems, where the different values are often allocated individually. You can view the semaphore limits on your system with the command ipcs -ls:

[root@london1 ˜]# ipcs -ls

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 100
semaphore max value = 32767

You use the kernel.sem parameter to control the number of semaphores on your system. The Oracle Validated RPM sets the value of kernel.sem to the following values:

kernel.sem = 250 32000 100 142

This setting allocates values for the semmsl, semmns, semopm, and semmni parameters, respectively.

Semaphores can be viewed with the operating system command ipcs -ls. The semmsl parameter defines the total number of semaphores in a semaphore set. This parameter should be set to the Oracle Validated RPM value of 250. Setting semmsl should always be considered in terms of semmns because both parameters impact the same system resources.

The semmns parameter sets the total number of semaphores permitted in the Linux system, and 32000 is the recommended default high-range value. The previously discussed semmsl value sets the maximum number of semaphores per set, while semmni sets the maximum number of semaphore sets. This means that the overall number of semaphores that can be allocated will be the minimum value of semmns or semmsl, multiplied by the value of semmni. A default value of 32000 for semmns ensures that this value is used by the system for its total semaphore limit.

The semopm parameter defines the maximum number of semaphore operations that can be performed by the semop() system call. In Linux, semop() can set multiple semaphores within a single system call. The recommended setting for this value is 100.

The semmni parameter sets the maximum number of total semaphore sets defined, and it should have a minimum value of 128. Multiplying this value by a semmsl value of 250 precisely equals the semmns setting of 32000. Although the Oracle Validated RPM sets this value to 142, it is overridden by semmns.

Unless an extremely large number of Oracle user connections are required on each node, the Oracle Validated RPM semaphore settings should be sufficient.

Setting network parameters correctly is especially important in a RAC environment because of its reliance on a local high-performance network interconnect between the Oracle instances. The next section describes the kernel parameters modified by the Oracle Validated RPM.

net.core.rmem_default, net.core.wmem_default,
net.core.rmem_max, net.core.wmem_max

net.ipv4.tcp_rmem = 4096 262144 4194304
net.ipv4.tcp_wmem = 4096 262144 4194304

Processes that include Oracle processes can communicate in an asynchronous manner with interprocess communication (IPC) messages. These messages are placed into a message queue where they can be read by another process. An example of this in an Oracle database environment is the communication that takes place between the foreground and background processes visible through the Oracle wait events, rdbms ipc message and rdbms ipc reply. The rdbms ipc message is sent by a background process when idle and waiting for a foreground process to send a message. You can view the message queue limits on your system with the command ipcs -lq, as shown:

[root@london1 ˜]# ipcs -lq

------ Messages: Limits --------
max queues system wide = 16
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536

The ipcs -lq command provides information on three message queue parameters:

The parameter that affects the number of open files is fs.file-max, and the Oracle Validated RPM sets this parameter to 327679. The fs.file-max parameter is used to set the maximum limit of open files for all processes on the system. This value is likely to be lower than the default value, which is determined dynamically based on the system memory. For example, this value is 1545510 on a x86-64 system with Oracle Enterprise Linux 5 and 16GB RAM; for 18BG of RAM, this value is 1773467. On some releases of Oracle 11g Release 2, this value is also lower than the minimum value checked by the OUI of 6815744, although you may ignore this warning if the Oracle Validated RPM setting meets your requirements. (You may also choose to change the parameter it to the OUI required value to prevent the warnings.)

The number of open and available file handles on the system can be seen in /proc/sys/fs/file-nr. In an ASM environment, using ASMLIB will reduce the total number of file descriptors across the system. Conversely, in an AMM environment, using ASMLIB increases demand on the required file descriptors across the system.

In contrast to ASMM, using AMM means that both SGA and PGA memory are managed in a unified manner by setting the memory-related parameters, MEMORY_TARGET and MEMORY_MAX_TARGET. To manage the SGA and PGA memory together, the Oracle 11g Database on Linux uses the tmpfs file system mounted at /dev/shm for a POSIX implementation of shared memory, using the mmap() and shm_open() system calls instead of shmget().

By default, tmpfs is allocated from virtual memory. Thus it can include both physical RAM and swap space, and it is sized to half of physical memory without swap space. However, unlike huge pages, which you will learn more about later in this chapter, tmpfs memory allocation is dynamic. Therefore, if tmpfs is not used, it does not take any allocation of memory, and it does not require a change in settings if AMM is not used. When AMM is used, Oracle allocates a number of memory mapped files in tmpfs. These files are allocated with at a file, or granule, size ranging from 4MB to MEMORY_MAX_TARGET of 1GB and 16MB when it is greater than 1GB. The allocated files can be identified by the shmid of the corresponding single page shared memory segment. The following example shows single page shared memory segments for both an ASM instance and a database instance:

[oracle@london1 ˜]$ ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x6422c258 557056     oracle    660        4096       0
0x10455eac 819201     oracle    660        4096       0

The allocated memory itself can be viewed in the mounted directory of /dev/shm. The zero-sized segments are mapped by Oracle processes to constitute a memory allocation up to MEMORY_MAX_TARGET, but these are not yet allocated to any of the dynamic components identified within V$MEMORY_DYNAMIC_COMPONENTS; thus they do not consume any physical memory. By default, the tmpfs file system allocates half of physical memory. Therefore, if your MEMORY_TARGET parameter exceeds this value, then the instance will fail to start with the error:

SQL> ORA-00845: MEMORY_TARGET not supported on this system

You may want to increase the size of the memory allocation in a manner that is persistent across reboots. To do this, begin by unmounting the /dev/shm file system, and then modifying the entry in /etc/fstab relating to tmpfs:

[root@london1 ˜]# more /etc/fstab
/dev/VolGroup00/LogVol00 /                      ext3    defaults          1 1
LABEL=/boot             /boot                   ext3    defaults          1 2
tmpfs                   /dev/shm                tmpfs   defaults,size=15g 0 0
devpts                  /dev/pts                devpts  gid=5,mode=620    0 0
sysfs                   /sys                    sysfs   defaults          0 0
proc                    /proc                   proc    defaults          0 0
/dev/VolGroup00/LogVol01 swap                   swap    defaults          0 0

Next, remount the file system to view the newly allocated space. Note that, until the memory is used, allocating a larger amount of memory to tmpfs does not actually result in the memory being utilized immediately:

[root@london1 ˜]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      433G  147G  264G  36% /
/dev/sda1              99M   13M   82M  14% /boot
tmpfs                  15G     0   15G   0% /dev/shm

You should keep the following in mind when working with the kernel parameter settings of the Oracle Validated RPM: when using AMM limits on memory, memory allocation is partly determined by the number of file descriptors, but it is not dependent on the shared memory parameters previously discussed in this section. Oracle recommends that the number of file descriptors be set to a value of at least 512*PROCESSES. This means that the Oracle Validated RPM setting of fs.file-max is sufficient for an Oracle processes value of up to 640. Failure to have a sufficient number of file descriptors will result in the following error: ORA-27123: unable to attach to shared memory segment. There is also a per user limit for the number of open files, in addition to the system-wide value, which by default is set to 1024. The Oracle Validated RPM modifies this per user value in the file /etc/security/limits.conf; you'll learn more about this in this chapter's section on PAM limits configuration.

When using AMM, there are some distinctions that you should be aware of in comparison to ASMM. First, if you're operating in a virtualized environment such as the Oracle VM (see Chapter 5), then MEMORY_MAX_TARGET cannot be allocated to a value greater than the memory currently allocated to the system. This limits the levels of memory that can be allocated and de-allocated to the operating system dynamically. On the other hand, SGA_MAX_SIZE can be set to larger than the available memory, which offers more flexibility in this case. Additionally, AMM cannot be configured in conjunction with huge pages. You should consider all of these factors when weighing the relative merits of using huge pages or AMM.

The parameter fs.aio-max-nr sets the maximum number of concurrent asynchronous I/O operations permitted on the system. By default, this parameter is set to 65536, and the Oracle Validated RPM increases this to 3145828. The actual system value can be seen in /proc/sys/fs/aio-nr. If /proc/sys/fs/aio-nr is equal to fs.aio-max-nr, then asynchronous I/O operations will be impacted. However, if you are using ASMLIB, then the value of /proc/sys/fs/aio-nr does not exceed a value of greater than 0. In this case, setting fs.aio-max-nr to any value has no effect. Changing the value of aio-max-nr does not allocate additional memory to kernel data structures, so it can be maintained at the default value.

When the Alt+SysRq+Ctr key combination is pressed on the system console, it enables direct communication with the kernel. This functionality is primarily enabled for debugging purposes. This feature is termed the Magic SysRq key, and it is enabled in the Linux kernel of Oracle Enterprise Linux 5. By default, however, the parameter kernel.sysrq is set to 0 to disable this functionality. The Oracle Validated RPM sets this parameter to 1 to enable all of the command key combinations, although a bitmask of values between 2 and 256 can be set to limit the number of commands available. An example key combination is Alt+SysRq+b, which immediately reboots the system. The value in kernel.sysrq only enables the system request functionality when the key combination is entered on the console keyboard. The same functionality is enabled in /proc/sysrq-trigger, regardless of the value of this setting. For example, the following command from the root user will also reboot the system, even if kernel.sysrq is set to 0:

echo b > /proc/sysrq-trigger

Clusterware uses the preceding command to reset a node in the cluster.

Other notable Magic SysRq values include t, which provides a detailed debugging report on the activity of all processes in the system; and m, which provides a report on memory usage. Both of these reports can be viewed in /var/log/messages or with the dmesg command. However, because the Magic SysRq commands communicate directly with the kernel, these debugging reports should be used infrequently and not as part of commands or scripts that you run regularly for system monitoring purposes.

Setting the parameter kernel.sysrq can be a matter of choice of security policies. However, this enabled functionality requires access to the console keyboard, so it will typically mean that the user has access to the physical system, such as the power supply.

Linux Pluggable Authentication Modules (PAM) are responsible for a number of authentication and privilege tasks on the system. You can find a number of modules to deal with activities such as password management, and the Oracle Validated RPM in particular sets values for the pam_limits module that determines which resources on the system can be allocated by a single user session, as opposed to across the system as a whole; that is, it determines the limits that are set at the login time of a session and not cumulative values for all sessions combined. For this reason, the important session is the one that is used to start the Oracle database. These limits are set for the oracle user in the file /etc/security/limits.conf, as in the following example, which shows the settings for an x86-64 system:

oracle  soft  nofile   131072
oracle   hard   nofile    131072
oracle   soft   nproc     131072
oracle   hard   nproc     131072
oracle   soft   core      unlimited
oracle   hard   core      unlimited
oracle   soft   memlock   50000000
oracle   hard   memlock   50000000

The values can be observed at the oracle user level; use the command ulimit -a to see all parameters:

[oracle@london1 ˜]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 155648
max locked memory       (kbytes, -l) 15360000

max memory size         (kbytes, -m) unlimited
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 131072
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Alternatively, you can use the ulimit command with the relevant argument in the preceding example to see the value of an individual parameter. For all of the relevant parameters, the Oracle Validated RPM sets both the soft and hard limits to the same value. Typically, the hard value is set higher than the soft value, and the user is able to increase or decrease the session level soft value up to the hard value. However, when these values are the same, the hard limit is the single enforceable value that the user cannot exceed.

The following four parameters affect PAM limits:

For AMD x86-64 architecture processors, the Oracle Validated RPM modifies the kernel boot parameters in the file /boot/grub/grub.conf to set the parameter numa=off. This parameter setting disables NUMA features (see Chapter 4 for more information); however, NUMA is not disabled on recent generations of Intel x86-64 architecture processors, which are also based on NUMA. Therefore, we recommend reviewing the NUMA section in Chapter 4 to determine the potential impact that this setting may have on different systems. We also recommend testing your configuration for its impact upon performance.

In addition to modifying kernel boot parameters, the Oracle Validated RPM also sets individual kernel module parameters. In particular, the script examines the file /etc/modprobe.conf for the presence of the e1000 driver, which is the device driver for Intel PRO 1000 Network Adapters. The script adds the option FlowContro=l where this driver is present. Flow control is a method of using standard Ethernet pause frames to manage communication and prevent buffer overruns in cases where the sender is sending data faster than the receiver can handle it. For the e1000 driver, the FlowControl option determines whether these pause frames are both generated and responded to. The possible values are 0, which disables the feature; 1, which means receive only; 2, which means transmit only; and 3, which enables both transmitting and receiving. The Oracle Validated RPM sets this parameter for receive only. The impact of the setting can be viewed with the ethtool application with the -a argument. Note that this setting only applies to the e1000 driver for PCI and PCI-X devices. For later PCI-Express devices, the e1000e or igb drivers will be used; therefore, this setting does not apply to the later devices.

An additional kernel parameter that can bring performance benefits to an Oracle Database 11g Release 2 RAC on Linux configuration is vm.nr_hugepages, which implement a feature known variously as large pages, huge pages, and hugetlb.

Setting vm.nr_hugepages with sysctl configures the number of huge pages available for the system; the default value is 0. The size of an individual huge page is dependent on the Linux operating system installed and the hardware architecture. The huge page size on your system can be viewed at the end of /proc/meminfo.

The following example for an x86-64 system with Oracle Enterprise Linux 5 system shows a default huge page size of 2MB:

[root@london1 root]# cat /proc/meminfo
...
HugePages Total:  0
HugePages_Free:   0
Hugepagesize:     2048kB

Some systems also enable the setting of a 1GB huge page size. Therefore, this page size value should always be known before setting the vm.nr_hugepages value.

Huge pages are allocated as much larger contiguous memory pages than the standard system pages. Such pages improve performance at the Translation Lookaside Buffer (TLB) level. As discussed in greater detail in Chapter 4, a larger page size enables a large range of memory addresses to be stored closer to the CPU level, increasing the speed of Oracle SGA access. This approach increases access speed by improving the likelihood that the virtual memory address of the pages that Oracle requires is already cached.

The vm.nr_hugepages parameter can be set at runtime. However, the pages are required to be contiguous. Therefore, we advise setting the required number in /etc/sysctl.conf, which is applied at boot time. Once allocated, huge pages are not pageable. This guarantees that an Oracle SGA using them will always remain resident in main memory.

The vm.nr_hugepages parameter must be set to a multiple of the huge page size to reach a value greater than the required Oracle SGA size. In addition, the Oracle user must have permission to use huge pages; this permission is granted by the memlock parameter discussed previously in this chapter.

For testing purposes, the huge page allocation can be mounted as a file system of type hugetlbfs, and memory can be utilized from this allocation using mmap(). However, at the time of writing, this mounted file system does not support AMM, nor does it support shmget() when using Oracle. Thus, there is no requirement for the hugetlbfs file system to be mounted during normal operations.

When the Oracle 11g instance starts, the shmget() call to allocate the shared memory for the SGA includes the SHM_HUGETLB flag to use huge pages, if available. If you're using ASMM or if the SGA required, then the SGA_MAX_SIZE is greater than the huge pages available. In this case, Oracle will not use any huge pages at all. Instead, it will use standard memory pages. The huge pages will still remain allocated to their pool; this will reduce the overall amount of available memory for general use on the system because huge pages will only be used for shared memory. The shmmax parameter is still applicable when using huge pages, and it still must be sized greater than the desired Oracle SGA size.

The values in the file /proc/meminfo can be viewed again to monitor the successful use of huge pages. After an Oracle instance is started, viewing /proc/meminfo will show the following:

HugePages_Total:  7500
HugePages_Free:    299
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

This output shows that 7500 huge pages were created, and the Oracle instance took 14GB of this huge page allocation for its SGA. If the instance has been started and the total remains the same as the free value, then Oracle has failed to use huge pages, and the SGA has been allocated from the standard pages. You should ensure that this is the case when starting the instance with both sqlplus and the srvctl command. It is also possible to reduce the value of the vm.nr_hugepages value after the instance has been started and re-run the sysctl -p command to release remaining unallocated and unreserved huge pages.

When setting the shmall parameter, the huge pages are expressed in terms of standard system memory pages, and one single huge page taken as shared memory will be translated into the equivalent number of standard system pages when allocated from the available shmall total. The shmall value needs to account for all of the huge pages used expressed in the number of standard pages, not just the finite Oracle SGA size. setting shmall to the number of huge pages Failing to account for all of the huge pages would result in the following Oracle error when starting the instance:

ORA-27102: out of memory
Linux-x86-64 Error: 28: No space left on device

When correctly managed, huge pages can both provide performance benefits and reduce the memory required for page tables. For example, the following snippet from /proc/meminfo shows the memory consumed by page tables for a huge pages configuration on x86-64:

[root@london1 ˜]# cat /proc/meminfo
...
PageTables:      24336 kB
...

HugePages_Total:  7500
HugePages_Free:    331
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

This allocation is not fixed because page tables are built dynamically for all processes as required. However, if the parameter PRE_PAGE_SGA is set to true, then these page tables are built at the time of process creation. PRE_PAGE_SGA can impact both instance and session startup times, as well as memory consumption. Therefore, we only recommend setting this parameter in an environment where its impact has been assessed. For example, it may be applicable in a system where performance is being analyzed and the number of sessions created is fixed for the duration. However, this parameter is less applicable in cases where there a number of sessions that connect to and disconnect from the database in an ad-hoc manner. The following example shows a system without huge pages, where the page tables have already grown to more than 1GB in size:

[oracle@london1 ˜]$ cat /proc/meminfo
PageTables:    1090864 kB
...
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

However, there are caveats that should be kept in mind when using huge pages. As noted previously, huge pages are only used with shared memory; therefore, they should not be allocated to more than is required by the SGA. Additionally, at the time of writing, huge pages are not compatible with Oracle VM or AMM, as discussed earlier. Therefore using huge pages requires more manual intervention by the DBA for memory management. For these reasons, we recommend that you test the use of huge pages to assess whether the additional management delivers performance and memory efficiency benefits for your environment.

In a RAC environment, Oracle needs to rapidly detect and isolate any occurrence of node failure within the cluster. It is particularly important to ensure that, once a node has entered an uncertain state, it is evicted from the cluster to prevent possible disk corruption caused by access that is not controlled by the database or cluster software. With Oracle RAC at 10g and 11.1, eviction at the Oracle software level was managed by Oracle Clusterware's Process Monitor Daemon (OPROCD). On the Linux operating system, eviction was ensured by the hangcheck-timer kernel module.

When Clusterware was operational, both the oprocd and the hangcheck-timer were run at the same time. Although the hangcheck-timer remains a requirement for RAC at Oracle 11g Release 1, it is no longer a requirement for RAC at Oracle 11g Release 2. At this release, there is also no oprocd daemon, and the functionality has been implemented within Clusterware's cssdagent and cssdmonitor processes, without requiring the additional hangcheck-timer kernel module. The timeout value for resetting a node remains equal to Oracle Clusterware's misscount value. This configurable value is given in the file crsconfig_params and set when the root.sh script is run as part of the Grid Infrastructure software installation. By default, the value of CLSCFG_MISSCOUNT is left unset in Oracle 11g Release 2, and the parameter takes the default value of 30 seconds as confirmed with the crsctl command after installing the Grid Infrastructure software:

[oracle@london1 bin]$ ./crsctl get css misscount
30

If you do wish to configure an additional timer with respect to this misscount value in Oracle 11g Release 2, then we recommend evaluating and testing the optional functionality provided by IPMI, rather than using the hangcheck-timer kernel module (you'll learn more about this later in this chapter). An IPMI-based timer can provide functionality above and beyond Oracle Clusterware. This is because it can be implemented in a hardware-independent fashion from the operating system itself. In this case, it is possible to reset a system undergoing a permanent hang state. If activating an IPMI-based timer, we recommend a timeout value of at least double the configured misscount value. This will both complement Clusterware's functionality, but also enable Clusterware to serve as the primary agent in evicting failed nodes from the cluster.

As noted previously in this chapter, the oracle user is created by the Oracle Validated RPM. This user can be the owner of both the Oracle database and Grid Infrastructure software. If you do not need to explicitly create additional users for your environment, then we recommend maintaining this default ownership. However, if you wish to have a different owner for the Grid Infrastructure software, then you may also create the grid user using the methods detailed previously for the oracle user. You must also ensuring that the oinstall group is specified for the initial group. Next, we will focus on the configuration of the environment of the oracle user only as owner of all installed Oracle software, including the database software, the Grid Infrastructure software of Clusterware, and ASM.

Before installing the Oracle software, you need to create the directories to install Oracle 11g Release 2. These directories correspond to the environment variables you will set for the oracle user. The directories you create will depend on whether you wish to implement a configuration adhering to the OFA; unless you have a policy in place that prevents you from being OFA-compliant, then we recommend that you follow the OFA guidelines for directory configuration. For an OFA-compliant configuration, you will create a parent directory of /u01 or later or a separate /u01 disk partition under the root file system. All other directories containing Oracle software will be created under this directory.

When the Oracle software is installed, the OUI will search for an OFA directory configuration the oracle user has permission to write to, such as /u01/app. The OUI will use this directory for the Oracle software configuration. The OUI will create the oraInventory directory in this location with permissions set to the oinstall group. This ensures that if multiple users are configured, all users will be able to read and modify the contents of oraInventory. If the ORACLE_BASE environment variable is set, then this takes preference over the OFA-compliant directory structure. Of course, if ORACLE_BASE is set to the OFA-compliant directory of /u01/app/oracle, then the outcome is the same. If ORACLE_BASE is not set and an OFA directory structure is not available, then the oracle user directory is used for software installation. However, the default permissions in this configuration do more than prevent the oracle user having ownership of the Oracle software.

If basing your installation on OFA, then your directory structure will be based on an ORACLE_BASE for the oracle user, but on a separate ORACLE_HOME directory also termed the ORA_CRS_HOME for the Grid infrastructure software. For the Grid Infrastructure software, the subdirectory structure must be owned by the root user once it's operational. Therefore, the directory must be installed directly under ORACLE_BASE. The/u01/app should also be owned by root, but with a group of oinstall to permit access to the oraInventory directory. Similarly, installations owned by the oracle user should be under the ORACLE_BASE directory. For the database ORACLE_HOME, the directory structure should be owned by the oracle user.

When configuring an OFA-compliant structure for both the Grid Infrastructure software and Oracle database software, we recommend that you create the ORACLE_BASE directory and allow the OUI to create the ORACLE_HOME and ORA_CRS_HOME to default specifications.

Issue the following commands as root to implement this minimum requirement:

[root@london1 root]# mkdir -p /u01/app/oracle
[root@london1 root]# chown -R root:oinstall /u01
[root@london1 root]# chmod -R 775 /u01

Using mkdir, the directories specified by ORA_CRS_HOME and ORACLE_HOME can also be pre-created to use the preferred directory structure instead of the default values from the OUI, which are /u01/app/11.2.0/grid/ for the Grid Infrastructure software and /u01/app/oracle/product/11.2.0/dbhome_1/ for the Database software. Initially, the permissions on these directories should be set to 775. After a successful default Grid infrastructure software installation, however, some of the directory ownership and permissions will have been changed to an owner of root, group of oinstall, and permissions of 755.

[root@london1 /]# ls -ld /u01
drwxr-xr-x 3 root oinstall 4096 Sep 17 14:03 /u01
[root@london1 /]# ls -ld /u01/app
drwxr-xr-x 5 root oinstall 4096 Sep 17 15:25 /u01/app
[root@london1 /]# ls -ld /u01/app/11.2.0/grid/
drwxr-xr-x 64 root oinstall 4096 Sep 17 15:32 /u01/app/11.2.0/grid/

Under the grid directory itself, some of the directories will be owned by root, such as bin, crs, css, and gns. However, the majority of the directory structure will retain ownership under the oracle user (or grid user, if you have chosen a separate owner for the Grid Infrastructure software). The default ORACLE_HOME and oraInventory directory will retain the following permissions after the Database software has also been installed:

[root@london1 /]# ls -ld /u01/app/oracle/product/11.2.0/dbhome_1/
drwxr-xr-x 72 oracle oinstall 4096 Sep 17 17:07
[root@london1 /]# ls -ld /u01/app/oraInventory/
drwxrwx--- 5 oracle oinstall 4096 Sep 17 17:10

With the oracle user created and the password supplied by the Oracle Validated RPM, it is possible to log in with the newly created oracle user account. The oracle user account will be configured with a default ˜/.bash_profile in the oracle user's home directory, which will resemble the following:

[oracle@london1 oracle]$ cat ˜/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ˜/.bashrc ]; then
        . ˜/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
unset USERNAME

If you are using the OFA directory structure, you do not need to update the default ˜/.bash_profile file with any environment variables before installing the Oracle Grid Infrastructure or Oracle database software. The installation will create the default directory structure detailed in the previous section. In fact, Oracle explicitly recommends not setting the environment variables ORA_CRS_HOME, ORACLE_HOME, ORA_NLS10, or TNS_ADMIN. And should you set the ORACLE_BASE environment variable only if you wish to change the installation locations from the default OFA.

Oracle has previously asserted that the ORACLE_BASE environment variable will become mandatory in future versions; however, at version 11g, it is not required if you are satisfied with the default such as /u01/app/oracle. Oracle recommends that you not set the additional environment variables because they are used extensively in the installation scripts. Therefore, not setting the variables ensures that there are no conflicting locations defined. If you do set the ORACLE_HOME environment variable, then your settings will override the default installation location for both the Grid Infrastructure software and Oracle database software. Thus, you should be aware of the correct relationship between the ownership and permissions of the different directory structures. You should also ensure that your settings comply with the appropriate guidelines.

After installation, you should set the required environment variables for the operation and administration of your system. Within the ˜/.bash_profile file, you should also set the following set the value of umask to 022.

The umask command applies a default mask to the permissions on newly created files and directories. The value 022 ensures that these are created with permissions of 644; this prevents other users from writing to the files created by the Oracle software owner.

For Oracle 11g Release 2 RAC, the following environment variables should be set for the oracle user: PATH, ORACLE_SID, ORACLE_HOME, and ORACLE_BASE. Environment variables must be configured identically on all nodes except for ORACLE_SID, which is instance specific. The correct setting of the ORACLE_SID variable is dependent upon the chosen approach for workload management; you can learn more about this in Chapter 11. The following sections detail the purpose of the most relevant environment variables applicable to an oracle user installing and administering 11g Release 2 RAC on Linux.

crsctl pin css -n
...

/u01/app/11.2.0/grid

[oracle@london1 ˜]$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"

LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin;

Now let's look at an example ˜/.bash_profile file that shows how to configure several environment variables and settings for a post-installation Oracle 11g Release 2 RAC node. Note that the export command is used on each environment variable line both to set the value of the variable and to ensure that this value is passed to all child processes created within the environment:

[oracle@london1 oracle]$ cat ˜/.bash_profile
# .bash_profile

if [ -t 0 ]; then
stty intr ^C
fi

# Get the aliases and functions
if [ -f ˜/.bashrc ]; then
        . ˜/.bashrc
fi


# User specific environment and startup programs
umask 022
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1

export ORA_CRS_HOME=/u01/app/11.2.0/grid
export ORACLE_SID=PROD1
export PATH=$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:$PATH

These environment variables would be set automatically on login or directly with the following command:

[oracle@london1 oracle]$ source .bash_profile

After installation, the hostname of the system is set in the file /etc/sysconfig/network. The file will also contain content for enabling the network and for defining the gateway.

If you need to change the system hostname for any reason, then the recommended method is to do this through the graphical user interface. Begin by selecting Network from the System and then Administration menu, and the hostname can be modified under the DNS tab of this utility.

If you provided the IP address of your working DNS Server during the operating system installation, then some of the hostname and name resolution information will have already been configured for you. If this is the case, you should verify the contents of the file /etc/resolv.conf. Alternatively, you could configure the file manually for a DNS Server added after the operating system has been installed on the cluster nodes. In both cases, the file should reference the search path for the domain in use. The following example for a GNS configuration shows both the domain and subdomain:

[root@dns1 named]# cat /etc/resolv.conf
search example.com grid1.example.com
nameserver 172.17.1.1
options attempts: 2
options timeout: 1

You should also use the file /etc/nsswitch.conf to verify the order of name services that are queried. In particular, you want to ensure that references to NIS are not found before the DNS:

[root@dns1 ˜]# vi /etc/nsswitch.conf
#hosts:     db files nisplus nis dns
hosts:      files dns

You should use the nslookup command to verify that the DNS name resolution, as explained earlier in this chapter's "Drilling Down on Networking Requirements" section.

For both GNS and manual IP configurations, we recommend that the /etc/hosts file be kept updated with the public and private interconnect addresses of all the nodes in the cluster. This file should also include the domain name extension to be included as an alias in the host file for the fixed public IP addresses.

Keeping the file updated ensures that there is no break in operations during any temporary failure of the DNS service. Again, the preferred method is to modify the settings using the graphical user interface. Navigate to the same utility used to set the hostname and select the Hosts tab. At this point, you have the option to add, edit, or delete the address, hostname, and aliases of hosts relevant to the cluster. You should also modify the default hostnames, removing the configured hostname from the 127.0.0.1 loopback address to prevent potential name resolution errors in the Oracle software. For example, these kinds of errors can occur when using the srvctl utility. Only the local hostname should be associated with this loopback address. It is particularly important that you do not remove this loopback address completely; removing it will result in errors in Oracle network components, such as the listener. Therefore, you should also remove or comment out all of the special IPv6 addresses. The following /etc/hosts file shows the details for a two-node cluster for a manual configuration:

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost
172.17.1.101            london1.example.com london1
172.17.1.102            london2.example.com london2
172.17.1.201            london1-vip.example.com london1-vip
172.17.1.202            london2-vip.example.com london2-vip
192.168.1.1             london1-priv
192.168.1.2             london2-priv

For a GNS configuration, the VIP addresses should not be included in the /etc/hosts file. As noted previously in this chapter, SCAN addresses should not be configured in /etc/hosts for either GNS or a manual IP configuration.

All nodes in an Oracle cluster must have exactly the same system time settings. If the system clocks are not synchronized, you may experience unpredictable behavior in the cluster. For example, you might fail to successfully register or evict a node as required. Also, manually adjusting the clock on a node by a time factor of minutes could cause unplanned node evictions. Therefore, we strongly advise that you synchronize all systems to the same time and make no major adjustments during operation. ("Major adjustments" do not include local time adjustments, such as regional changes for daylight saving time.)

Within Linux, the most common method to configure time synchronization is to use the Network Time Protocol (NTP). This protocol allows your server to synchronize its system clock with a central server. Your preferred time servers will be available from your Internet service provider. Alternatively, if these are not available, you can choose from a number of open access public time servers.

In Oracle 11g Release 2 RAC, the Oracle Cluster Time Synchronization Service Daemon is always operational. However, if NTP is active, CTSS only monitors the status of time synchronization. If NTP is not available, then CTSS will synchronize the time between the nodes of the cluster itself without an external time source. For this reason, operating with NTP is preferable. After installing the Grid Infrastructure software, you can monitor the operation of CTSS in the octssd.log in the $ORA_CRS_HOME/log/london2/ctssd directory, as in this example:

[cssd(26798)]CRS-1601:CSSD Reconfiguration complete. Active nodes are london1 london2 .
2009-09-17 15:34:32.217
[ctssd(26849)]CRS-2403:The Cluster Time Synchronization Service on host london2 is in observer mode.
2009-09-17 15:34:32.224

A good practice within a network environment is to configure a single, local dedicated time server to synchronize with an external source. You should also set all internal servers to synchronize with this local time server. If this method is employed, you can use the configuration detailed momentarily to set the local time server to be the preferred server. This approach configures additional external time servers in case of the local one fails. If you do not have NTP available, then you should ensure that the NTP service is disabled. You should not run NTP without an external time source available because the CTSS service will remain in observer mode for potentially unsynchronized times, as reported in the octssd.log.

You can instruct the Oracle server to use NTP on Oracle Enterprise Linux systems from the graphical user interface by right-clicking the Date and Time tab displayed on your main panel, and then selecting Adjust Date & Time. Next, select the second tab on this page and add your NTP servers using the process detailed previously during the installation process. You can also configure NTP manually using the chkconfig and service commands to modify the /etc/ntp.conf and /etc/ntp/step-tickers files. To do this, begin by first ensuring that the NTP service has been installed on the system by using the chkconfig command:

[root@london1 root]# chkconfig  --list ntpd
ntpd   0:off  1:off  2:off  3:off  4:off  5:off  6:off

Next, manually edit the /etc/ntp.conf file and add lines specifying your own time servers, as in the following examples:

server ntp0.uk.uu.net
server ntp1.uk.uu.net
server ntp2.uk.uu.net

If you have a preferred time server, add the keyword prefer to ensure synchronization with this system, if available:

server ntp0.uk.uu.net prefer

To use open-access time servers, enter the following information:

server 0.rhel.pool.ntp.org
server 1.rhel.pool.ntp.org
server 2.rhel.pool.ntp.org

You will also have a default restrict line at the head of your configuration file. However, another good practice is to include server-specific security information for each server. This prevents that particular NTP server from modifying or querying the time on the system, as in the following example:

restrict ntp0.uk.uu.net mask 255.255.255.255 nomodify notrap noquery

Next, modify your /etc/ntp/step-tickers and add the same servers listed in your /etc/ntp.conf file:

server ntp0.uk.uu.net
server ntp1.uk.uu.net
server ntp2.uk.uu.net

Now use the chkconfig command to make sure that the NTP daemon will always start at boot time at run levels 3 and 5:

[root@london1 root]# chkconfig --level 35 ntpd on

You're now ready to verify the configuration, which you accomplish using chkconfig --list:

root@london1 root]# chkconfig  --list ntpd
ntpd  0:off  1:off  2:off  3:on  4:off  5:on  6:off

The start-up options for the NTP daemon are detailed in the /etc/sysconfig/ntpd file. In normal circumstances, the default options should be sufficient. You can now start the service to synchronize the time using the following command:

[root@london1 root]# service ntpd start

Next, the date command can be used to query the system time to ensure that the time has been set correctly. The NTP daemon will not synchronize your system clock with the time server if they differ significantly. If the system clock and time server differ too much, the NTP daemon will refer to the systems in the /etc/ntp/step-tickers files to set the time correctly. Alternatively, the time can be set manually using the ntpdate command:

[root@london1 root]# ntpdate -u -b -s ntp0.uk.uu.net

During the installation of Oracle 11g Release 2 RAC software, a secure shell (ssh) configuration is required on all nodes. This ensures that the node on which the installer is initiated can run commands and copy files to the remote nodes. The secure shell must be configured, so no prompts or warnings are received when connecting between hosts. During the installation of the Grid Infrastructure software, the OUI provides the option to both test and automatically configure a secure shell between the cluster nodes. Therefore, we recommend that you not configure a secure shell in advance, but instead permit the OUI to complete this part of the configuration.

You should review the following steps to troubleshoot and verify the actions taken by the installer only if the connectivity test following automatic configuration fails. Additionally, these steps for the manual configuration of ssh are required for the Oracle Cluster Health Monitor user called crfuser (you will learn more about this in Chapter 12).

To configure a secure shell on the cluster nodes, first run the following commands as the oracle user to create a hidden directory called ˜/.ssh if the directory does not already exist (we use the standard tilde character [˜] here to represent the location of the oracle user's home directory):

[oracle@london1 oracle]$ mkdir ˜/.ssh
[oracle@london1 oracle]$ chmod 755 ˜/.ssh

Now create private and public keys using the ssh-keygen command. Next, accept the default file locations and enter an optional passphrase, if desired:

[oracle@london1 ˜]$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
a7:4d:08:1e:8c:fa:96:b9:80:c2:4d:e8:cb:1b:5b:e4 oracle@london1

Now create the DSA version:

[oracle@london1 ˜]$ /usr/bin/ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
dd:14:7e:97:ca:8f:54:21:d8:52:a9:69:27:4d:6c:2c oracle@london1

These commands will create four files in ˜/.ssh called id_rsa, id_rsa.pub, id_dsa, and id_dsa.pub. These files contain the RSA and DSA private and public keys.

In the .ssh directory, copy the contents of the id_rsa.pub and id_dsa.pub files to a temporary file. This file will be copied to all other nodes, so you use the hostname to differentiate the copies:

[oracle@london1 oracle]$ cat id_rsa.pub id_dsa.pub > london1.pub

Repeat this procedure for each host in the cluster, and then copy the public key file to all other hosts in the cluster:

[oracle@london1 oracle] scp london1.pub london2:/home/oracle/.ssh

Next, concatenate all the public key files into /ssh/authorized_keys on each host in the cluster:

cat london1.pub london2.pub > authorized_keys

Finally, set the permissions of the authorized keys file on all nodes:

[oracle@london1 oracle]$ chmod 644 authorized_keys

If no passphrase was specified, ssh and scp will now be able to connect across all nodes. If a passphrase was used, then these two additional commands should be run in every new bash shell session to prevent a prompt being received for the passphrase for every connection:

[oracle@london1 oracle]$ ssh-agent $SHELL
[oracle@london1 oracle]$ ssh-add

Enter the passphrase, and the identity will be added to the private key files. You can test the ssh and scp commands by connecting to all node combinations, remembering to check the connection back to the node you are working upon. Connections should be tested across both public and private networks:

[oracle@london1 oracle]$ ssh london1
[oracle@london1 oracle]$ ssh london2
[oracle@london1 oracle]$ ssh london1-priv
[oracle@london1 oracle]$ ssh london2-priv

All combinations should be tested. On the first attempted connection, the following warning will be received, and the default of answer yes should be entered to add the node to the list of known hosts. Doing so prevents this prompt from stalling the Oracle installation:

[oracle@london1 .ssh]$ ssh london1
The authenticity of host 'london1 (172.17.1.101)' can't be established.
RSA key fingerprint is 6c:8d:a2:13:b1:48:03:03:74:80:38:ea:27:03:c5:07.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'london1,172.17.1.101' (RSA) to the list of known hosts.

If the ssh or scp command is run by the oracle user on a system running an X Windows–based desktop from another user, then the session will receive the following warning unless X authority information is available for the display:

Warning: No xauth data; using fake authentication data for X11 forwarding.

This warning is received because, by default, ssh is configured to do X11 forwarding, and there is no corresponding entry for the display in the ˜/.Xauthority file in the home directory of the oracle user on the system where the command was run. To prevent this warning from occurring during the Oracle software installation, edit the file /etc/ssh/ssh_config and change the line ForwardX11 yes to ForwardX11 no. Next, restart the sshd service, as shown here:

[root@london1 root]# servce sshd restart

X11 forwarding will now be disabled, and the warning should not be received for ssh or scp connections. Alternatively, you can use the following entries to disable X11 forwarding at a user level. Do this by creating a file called config in the .ssh directory of only the oracle user:

[oracle@london1 .ssh]$ cat config
Host *
ForwardX11 no

If X11 forwarding is disabled at the system or user level, you can still use the ssh or scp command with the -X option request to forward X Window display information manually, as in the following example:

[oracle@london1 oracle]$ ssh -X london2

If X11 forwarding is re-enabled after installation, then using ssh with the -x option specifies that X information is not forwarded manually:

[oracle@london1 oracle]$ ssh -x london2

The first step in preparing the disks for both the Clusterware and Database files irrespective of how they are to be used is to first create partitions on the disks. The example commands used to illustrate the creation of partitions on these disks is fdisk and parted.

The command fdisk -l displays all of the disks available to partition. The output for this command should be the same for all of the nodes in the cluster. However, the fdisk command should be run on one node only when actually partitioning the disks. All other nodes will read the same partition information, but we recommend adopt one of the following pair of approaches. First, you can run the partprobe command on the other nodes in the cluster to update the partition table changes. Second, you can reboot these nodes for the partition information, so they are fully updated and displayed correctly.

The following example shows the corresponding disk for one of the SCSI devices detailed previously. In this example, /dev/sdd is the external disk to be partitioned:

[root@london1 ˜]# fdisk -l /dev/sdd

Disk /dev/sdd: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders

Units = cylinders of 2074 * 512 = 1061888 bytes

   Device Boot      Start         End      Blocks   Id  System

To partition the drives, use fdisk with the argument of the disk device you want to partition:

[root@london1 root]# fdisk /dev/sdd

In the preceding example, the shared device /dev/sdd has been selected as the disk where a copy of the OCR and Clusterware voting disk will reside under an ASM diskgroup. We will create the single required partition of 1GB in size, passing it to the ASM configuration.

At the fdisk prompt, enter option n to add a new partition; p, to make it a primary partition; and 1, the number of the next available primary partition. Next, accept the default value for the first cylinder and enter the size of the partition in the form of the number of cylinders. In this case, that number is half of the available value. The reported number of cylinders may vary even for exactly the same disks connected to different architecture systems. Therefore, you can also specify the partition size, such as +500M for a 500MB partition. For these selections, the fdisk dialog will resemble the following:

[root@london1 Desktop]# fdisk /dev/sdd

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1011, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1011, default 1011):
Using default value 1011

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

The newly created partition can now be displayed as follows:

[root@london1 Desktop]# fdisk -l /dev/sdd

Disk /dev/sdd: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1        1011     1048376+  83  Linux

Now either run partprobe or reboot the remaining nodes in the cluster, and then use fdisk -l to verify that all of the nodes can view the partition tables written to the disk.

We noted during Linux installation that MBR partitioning stores data in hidden sectors at the start of the disk, whereas GPT partitioning does not do this. Therefore, when using MBR partitioning either with fdisk or parted, it is important to take this data into account when creating disk partitions, especially on the disks where the Oracle database files will reside. The modifications required are dependent upon the storage type and configuration, and the aim here is to ensure that, for a RAID striped configuration, disk I/O is aligned with the storage RAID stripe size. Depending on the I/O characteristics, failure to do this may lead to an increase in disk or stripe crossing in some cases. For example, this may occur in cases where a single logical read or multiple write operations from the system will result in multiple stripe operations on the storage, thus proving detrimental to performance. In MBR formatting, the hidden sectors occupy the first 63 sectors of the disk. In a case where this storage stripe size is 64kB, it is necessary to realign the partition boundary to sector 128, where 128 sectors of 512 bytes align with the 64kB stripe size. This should be done for each disk on which the database data files will reside, as well as for the first partition on a disk. All subsequent partitions should align with the new boundary. The following example illustrates the additional fdisk commands (run in expert mode) required to realign the partition boundary for the partitioning dialog shown previously:

Command (m for help): x

Expert command (m for help): b
Partition number (1-4): 1
New beginning of data (63-1044224, default 63): 128

fdisk can still be used to format disks with the MBR formatting scheme; however, parted must be used to take advantage of GPT partitioning. Like fdisk, parted is invoked with the argument of the disk to partition. On starting, it prints the version of the software and licensing information to the screen. Next, it presents the (parted) prompt:

[root@london1 ˜]# parted /dev/sdd
GNU Parted 1.8.1
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)

If the disk label type is not already specified as gpt (the default), then use the mklabel command to specify the new disk label type:

(parted) mklabel
Warning: The existing disk label on /dev/sdd will be destroyed and all data on
this disk will be lost. Do you want to continue?
Yes/No? Yes
New disk label type?  [msdos]? gpt

The print command is used to display the partition table. We have just created the partition table, so this example shows that the disk label type is gpt and no partitions have yet been created:

parted) print

Model: DGC RAID 0 (scsi)
Disk /dev/sdd: 1074MB
Sector size (logical/physical): 512B/512B

Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

To create a partition, use the mkpart command. At the prompts, enter a value for the partition name and accept the default file system type. Although mkpart can be used for creating a local file system at the same time as the partition, it does not support any clustered file systems that can be shared between nodes or ASM. For the partition sizes, enter the starting value for the partition in megabytes. The first value will be 0, and the ending value will be in megabytes. The following example creates a 1074MB partition:

(parted) mkpart
Partition name?  []? crs1
File system type?  [ext2]?
Start? 0
End? 1074
(parted)

Use mkpart to create the subsequent partitions, and then enter the values for partition and file system type. Next, enter the ending point of the previous partition as the starting point of the new one; for this partition's ending point, enter the value (in MBs) of the disk size detailed in the disk geometry section. Printing the partition table now displays the created partition. You do not need to call an additional write command, and the partition table will remain on exiting the parted application:

(parted) print

Model: DGC RAID 0 (scsi)
Disk /dev/sdd: 1074MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size   File system  Name   Flags
 1      17.4kB  1074MB 1074MB               crs1

GPT partitions are not visible with the fdisk command, and one partition will be displayed, regardless of the number of partitions created:

[root@london1 ˜]# fdisk -l /dev/sdd

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdd: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         131     1048575+  ee  EFI GPT
...

Therefore, if you have created multiple partitions, you should view /proc/partitions to see whether the partitions have been successfully created and are available for use.

If you're using an OCFS2 cluster file system method to store the OCR and Clusterware voting disk, then the Grid Infrastructure software installation will create the OCR and Clusterware voting disk as files during the installation process. Thus the only requirement is to format the disk partitions you created previously with OCFS2, as discussed in Chapter 5. You may then specify suitable file names to be created at the time of the Grid Infrastructure installation process (see Chapter 7 for more information on this). If you wish to use ASM, there are additional configuration steps required to prepare the partitions either with ASMLIB or manually.

I/O multipathing is a concept similar to network channel bonding, which is covered later in this chapter. Like channel bonding, I/O multipathing support requires at least two storage HBAs per server that connect to the target storage to display the LUNs through two independent paths. To completely eliminate any single point of failure, each path should also be accessed through a separate storage switch. In the event of an HBA, cable, or switch failure, a path to the storage is maintained. During regular operations, such a configuration provides load balancing in addition to redundancy.

Typically storage vendors have provided multipathing software solutions dedicated for their range of storage products. In contrast, device-mapper provides a generic solution for I/O multipathing across a range of vendors' products, including Oracle Enterprise Linux 5. Notwithstanding the generic nature of multipathing with device-mapper, you should check with your storage vendor to determine its support for this form of multipathing. You should also inquire about the specific software and configuration settings for a complete multipathing solution at both the server and storage levels.

As its name implies, device-mapper provides a method to redirect I/O between block devices. It also provides the foundation for a number of storage configurations that you have already encountered in this chapter, such as LVM and RAID.

The device-mapper-multipath is included in a default installation, or it can be installed from the installation media with the rpm or yum commands. The module is loaded with the command modprobe dm-multipath. Before proceeding with multipath configuration, it is necessary to install and configure the storage and HBA, so that the two paths to the disk devices are visible to the Linux operating system. The following example illustrates the same storage approach shown in the previous example, but with the additional, visible path to the same devices:

[root@london1 ˜]# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: Hitachi HDT72505 Rev: V56O
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: Hitachi HDT72505 Rev: V56O
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: DGC      Model: RAID 0           Rev: 0324
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 01
  Vendor: DGC      Model: RAID 0           Rev: 0324
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 02
  Vendor: DGC      Model: RAID 0           Rev: 0324

Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: DGC      Model: RAID 0           Rev: 0324
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 01
  Vendor: DGC      Model: RAID 0           Rev: 0324
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 02
  Vendor: DGC      Model: RAID 0           Rev: 0324
  Type:   Direct-Access                    ANSI SCSI revision: 04

The additional devices will also be visible with the fdisk command; depending on the storage, however, they may not be accessible.

The multipathing setup is configured in the file /etc/multipath.conf. The following example is specific to the EMC CLARiiON storage; other storage vendors can supply examples specific to their products. Begin by removing or commenting out the following section:

#blacklist {
#        devnode "*"
#}

Next, add the following to the end of the file:

defaults {
path_grouping_policy failover
user_friendly_names yes
}
multipaths {
multipath {
wwid 360060160a5b11c00420e82c83240dc11
alias       crs
mode        640
uid         500
gid         500
 }
}
devices {
    device {
        vendor "DGC"
        product "*"
        product_blacklist "(LUNZ|LUN_Z)"
        path_grouping_policy group_by_prio
        getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout "/sbin/mpath_prio_emc /dev/%n"
        path_checker    emc_clariion
        path_selector   "round-robin 0"
        features        "0"
        hardware_handler        "1 emc"
        features "1 queue_if_no_path"
        no_path_retry 300
        hardware_handler "1 emc"
        failback immediate

}

The device-mapper creates a number of devices to support multipathing. A device name such as /dev/dm-* indicates the kernel device name, and it should not be accessed directly:

[root@london1 ˜]# ls -l /dev/dm-*
brw-rw---- 1 root root 253, 2 Aug  5 15:32 /dev/dm-2
brw-rw---- 1 root root 253, 3 Aug  5 15:32 /dev/dm-3
brw-rw---- 1 root root 253, 4 Aug  5 15:32 /dev/dm-4
brw-rw---- 1 root root 253, 5 Aug  5 15:32 /dev/dm-5
brw-rw---- 1 root root 253, 6 Aug  5 15:32 /dev/dm-6
brw-rw---- 1 root root 253, 7 Aug  5 15:32 /dev/dm-7

Devices names such as /dev/mpath/mpath2 are created as symbolic links, according to udev rules defined in /etc/udev/rules.d/40-multipath.rules. It is possible to modify these rules to set the correct permissions (as detailed in the previous section). However, these modified rules will only impact these symbolic links. Instead, the device to use for access is listed under the directory /dev/mapper/, as in this example: /dev/mapper/mpath2. For this reason, the syntax for the section to define persistence is based on the WWID; the LUN unique ID; and the alias, mode, and ownership. The upcoming example illustrates the approach for explicitly naming and changing the ownership of the OCR and Clusterware voting disk. At the same time, it leaves the other devices with their default naming, which means you can apply these changes to as many devices as you wish. This is the preferred method for configuring the correct permissions of multipathed devices, as opposed to using udev rules directly, an approach you'll learn about later in this chapter. You should ensure that the ownership and permission is set equally in /etc/multipath.conf for all of the nodes in the cluster.

To verify your configuration, run the multipath command with the -d option, as in this example: multipath -v3 -d. The verbose output gives considerable detail. In particular, the paths list section is useful for identifying the devices, their paths, and their current status. If the output is satisfactory, you can run the same command without the -d option to active the configuration. Checking the /dev/mapper directory shows the created multipath devices available for use:

[root@london1 ˜]# ls -l /dev/mapper*
total 0
crw-r----- 1 root   dba       10, 63 Aug  5 15:31 control
brw-r----- 1 oracle dba      253,  3 Aug  5 15:32 crs
brw-r----- 1 oracle dba      253,  6 Aug  5 15:32 crsp1
brw-rw---- 1 root   disk     253,  2 Aug  5 15:32 mpath2
brw-rw---- 1 root   disk     253,  5 Aug  5 15:32 mpath2p1
brw-rw---- 1 root   disk     253,  4 Aug  5 15:32 mpath3
brw-rw---- 1 root   disk     253,  7 Aug  5 15:32 mpath3p1
brw-rw---- 1 root   disk     253,  0 Aug  5 15:32 VolGroup00-LogVol00
brw-rw---- 1 root   disk     253,  1 Aug  5 15:31 VolGroup00-LogVol01

Note the updated name and permission for the OCR and Clusterware voting disk in the preceding example. The multipath command also shows the active paths to the devices:

root@london1 ˜]# multipath -l1
mpath2 (360060160a5b11c006840e1703240dc11) dm-2 DGC,RAID 0
[size=102G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=0][active]
 \_ 3:0:0:0 sdc 8:32  [active][undef]

\_ round-robin 0 [prio=0][enabled]
 \_ 2:0:0:0 sdf 8:80  [active][undef]
crs (360060160a5b11c00420e82c83240dc11) dm-3 DGC,RAID 0
[size=1.0G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=0][active]
 \_ 3:0:0:1 sdd 8:48  [active][undef]
\_ round-robin 0 [prio=0][enabled]
 \_ 2:0:0:1 sdg 8:96  [active][undef]
mpath3 (360060160a5b11c0046fd4df53240dc11) dm-4 DGC,RAID 0
[size=299G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=0][active]
 \_ 3:0:0:2 sde 8:64  [active][undef]
\_ round-robin 0 [prio=0][enabled]
 \_ 2:0:0:2 sdh 8:112 [active][undef]

You can test the multipath configuration by disabling access to the active path:

[root@london1 ˜]# echo offline > /sys/block/sdd/device/state

In the next example, the failing path is identified by the major and minor device number combination. In this case, the system logs the path failed over in response to the path being disabled:

device-mapper: multipath: Failing path 8:48.
device-mapper: multipath emc: emc_pg_init: sending switch-over command

The multipath command shows the currently active path:

[root@london1 ˜]# multipath -l
...
crs (360060160a5b11c00420e82c83240dc11) dm-3 DGC,RAID 0
[size=1.0G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=0][enabled]
 \_ 3:0:0:1 sdd 8:48  [failed][faulty]
\_ round-robin 0 [prio=0][active]
 \_ 2:0:0:1 sdg 8:96  [active][undef]
...

The preceding example includes more actions than simply changing paths to the devices. In EMC CLARiiON terminology, the LUN is automatically trespassed between storage processors on the storage itself in response to the switch over command to enable access through the alternate path. This additional functionality reiterates the importance of ensuring that multipathing with device-mapper is a supported configuration throughout the entire system, including the HBA and storage.

To return the device back to an active state, you should echo running instead of offline to the device state.

Partitioning on multipath devices can be performed with the fdisk command, exactly as detailed previously on the /dev/mapper/mpath0 device. However, an additional step is required to then register the created partitions with the command kpartx.

After presenting the shared storage to the operating system and partitioning the disks, no additional software is required to use them for ASM. If you're using ASM either for the OCR and Clusterware voting disk partition or for database storage, then you may also wish to configure ASMLIB on the underlying devices beforehand. One of the main benefits of ASMLIB is that it provides persistence in the naming and permissions of the disks to be used for ASM. And while udev can also provide this functionality without requiring additional package installations beyond the default operating installation, ASMLIB was always intended to provide additional benefits for Oracle environments. As noted previously in this chapter, the use of ASMLIB reduces the requirement for file descriptors across the system. ASMLIB also implements asynchronous I/O, bypassing the standard Linux implementation of asynchronous I/O. However, the main benefits of ASMLIB are in the area of manageability and flexibility.

Once configured, ASMLIB provides a standard method for identifying the presence of ASM disks on the system. It is also possible to add additional nodes to the cluster or transfer the disks between different architecture systems. Subsequently, scanning the disks will detect the ASM configuration and maintain the configured permission for ASMLIB installed locally on each node. This stands in contrast to udev, which requires manual identification and configuration of the ASM disks for each individual system. Therefore, the main benefits of ASMLIB are ease of use and the ability to identify ASM disks. If you are comfortable with the benefits of having udev implement this solution for the OCR and Clusterware voting disk partitions, then ASMLIB is not an essential requirement. However, ASMLIB may provide additional identification benefits for the ASM diskgroups that contain database files. This is especially true in cases where disks are often transferred between systems, thereby reducing the possibility of errors resulting in data loss.

ASMLIB requires the installation of three RPM packages: the ASM library, tools, and driver. The RPM packages for the tools and driver are included with the installation media; however, the ASM library is not. Customers of the Unbreakable Linux Network can retrieve the library from there. The library (and the other packages) can also be downloaded from the following location:

www.oracle.com/technology/software/tech/linux/asmlib/rhel5.html

You can install all three ASMLIB RPM packages, as follows:

[root@london1 ˜]# rpm -ivh oracleasm*
Preparing...                ########################################### [100%]
   1:oracleasm-support      ########################################### [ 33%]
   2:oracleasm-2.6.18-164.el########################################### [ 67%]
   3:oracleasmlib           ########################################### [100%]

Your first action is to configure the driver. This action sets the ownership of the device as a functional equivalent of udev configuration:

[root@london1 asmlib]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

Next, you use the createdisk command to mark the disks as ASM disks:

[root@london1 ˜]# /etc/init.d/oracleasm createdisk VOL1 /dev/sdc1
Marking disk "VOL1" as an ASM disk:                        [  OK  ]

The configured disks can be deleted with the deletedisk command. In the process of deleting the disk metadata, this command will also delete access to any data configured on the disk. The listdisks command shows the configured volumes, while the querydisk command shows the mapping between the ASM disk and the system disk device:

[root@london1 asmlib]# /etc/init.d/oracleasm querydisk /dev/sdc1
Device "/dev/sdc1" is marked an ASM disk with the label "VOL1"

After configuring the first node in the cluster, you can use the scandisks command to detect the configured disks on the other nodes:

[root@london1 asmlib]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]

If you have configured multipath devices as explained previously in this chapter, then an additional configuration step is required. A multipath configuration will present three views of the same disk device. These views represent both channels and the multipath device itself. By default, on scanning ASMLIB will select the first channel scanned, which may not be the multipath device. To ensure that it is the multipath device that ASMLIB uses, you can add a section that excludes scanning the non-multipath devices to the /etc/sysconfig/oracleasm configuration file. For example, you can exclude scanning the disk devices that represent the channels for the OCR and Clusterware voting disk partitions given in the previous example, as shown here:

ORACLEASM_SCANEXCLUDE="sdd sdg"

Explicitly excluding the channels from direct use means that ASMLIB will be configured with the multipath device, thereby ensuring that ASM will benefit from the underlying redundancy of the I/O configuration.

Due to Linux's Unix-based heritage, devices in Linux are traditionally files created by the mknod command. This command directs input and output to the appropriate device driver identified by the major device number. One of the challenges with the static configuration of devices is that the number of devices pre-created to support a wide range of hardware connectivity is significant. This makes it difficult to identify which devices are connected to the system and active. With the 2.6 kernel, an in-memory virtual file system implementation called sysfs was introduced to present a rationalized view of the connected hardware devices to processes running in user space. The sysfs virtual file system is mounted on /sys; a driver loaded as a module registers with sysfs when the module is loaded, and the udevd event management daemon dynamically creates the corresponding devices under /dev. This implementation reduces the created devices down from many thousands of pre-created static devices to only the devices connected to the system. The device configuration enables device creation and naming, according to defined rules in the main configuration file, /etc/udev/udev.conf. This file includes an entry for the directory that defines the rules for device creation; by default, it is located under the directory /etc/udev/rules.d.

Familiarity with udev is important because previously, attributes such as device ownership and permissions could be modified and would be persistent across reboots due to the static nature of the configuration. Under udev's dynamic configuration, the device is created every time the corresponding driver module is loaded. Therefore, rules must be defined to ensure this configuration sets the desired attributes each time. Configuring devices with udev means that operating system standards can be used for correctly preparing a partition for ASM, but without requiring ASMLIB.

# There are a number of modifiers that are allowed to be used in some of the
# fields.  See the udev man page for a full description of them.
#
# default is OWNER="root" GROUP="root", MODE="0600"
#
KERNEL=="sdd[1-9]", OWNER="oracle" GROUP="dba", MODE="0660"

[root@london1 rules.d]# udevcontrol reload_rules
[root@london1 rules.d]# start_udev
Starting udev:                                             [  OK  ]

[root@london1 rules.d]# ls -l /dev/sdd1
brw-r----- 1 oracle dba 8, 49 Jul 29 17:20 /dev/sdd1

Enabling Udev Persistence

Here's something else to consider if you're not using ASMLIB. In addition to using udev to set permissions, you should also consider using it to configure device persistence of the OCR and Clusterware voting disk partition, as well as the database storage partitions used for ASM. Similar to the setting of permissions, the dynamic nature of device discovery with udev means that there is no guarantee that devices will always be discovered in the same order. Therefore, it is possible that device names may change across reboots, especially after adding new devices. For this reason, we recommend that you also configure udev to ensure that the same devices are named consistently whenever udev is run.

The first step is to identify the physical devices that correspond to the logical disk names configured by udev. The physical device names are derived on the storage itself, and these typically correspond to the Network Address Authority (NAA) worldwide naming format to ensure that the identifier is unique. Figure 6-9 illustrates the identifier for a LUN under the heading of Unique ID.

Figure 6.9. The LUN Unique ID

The identifier can also be determined with the commands udevinfo or /sbin/scsi_id when run against the devices in the /sys directory. With the /sbin/scsi_id command, the arguments -p 0x80 and -s return the SCSI Vendor, model, and serial number to cross reference against the storage. However it is the optionally specified -p 0x83 and the -s arguments that return the unique identifier. In this case, that identifier is prefixed by the number 3, which identifies the device as an NAA type 3 device; this identifier also corresponds to the unique ID from the storage:

[root@london1 ˜]# /sbin/scsi_id -g -s /block/sdd
360060160a5b11c00420e82c83240dc11

The identifier being determined by the storage is based on the entire LUN, as opposed to the individual disk partitions configured by the systems. Therefore, the identifier is the same for multiple partitions on the same device. Additionally, the identifier will also be the same for the same device shared between nodes in the cluster:

[root@london2 ˜]# /sbin/scsi_id -g -s /block/sdd
360060160a5b11c00420e82c83240dc11

The file /etc/scsi_id.config enables you to set options to modify the default behaviour of the scsi_id command. In particular, the option -g is required for the scsi_id command to produce output. Therefore, we recommend that you add the line options=-g to this file, as follows:

# some libata drives require vpd page 0x80
vendor="ATA",options=-p 0x80
options=-g

The following example for setting udev rules assumes that this addition has been made. If you do not wish to make the preceding addition, then you should be aware that the -g option must be given to the scsi_id command when specified in the udev rules. If not, the command will generate no output, causing the rule to fail.

You can also retrieve the device information using the command udevinfo in conjunction with the identifier shown under the heading, ID_SERIAL:

[root@london1 ˜]# udevinfo -q all -p /block/sdd
P: /block/sdd
N: sdd
S: disk/by-id/scsi-360060160a5b11c00420e82c83240dc11
S: disk/by-path/pci-0000:0b:00.0-fc-0x5006016b41e0613a:0x0001000000000000
E: ID_VENDOR=DGC
E: ID_MODEL=RAID_0
E: ID_REVISION=0324
E: ID_SERIAL=360060160a5b11c00420e82c83240dc11
E: ID_TYPE=disk
E: ID_BUS=scsi
E: ID_PATH=pci-0000:0b:00.0-fc-0x5006016b41e0613a:0x0001000000000000

Once you have the unique identifier, you can edit the file created previously for setting device permissions, 55-udev.rules. The following entry corresponds to the device name identified previously and uses /sbin/scsi_id with the -g option that was set in the /etc/scsi_id.config file. Also, the %n substitution variable is specified so that the partitions for this device are correctly named in numerical order. At this point, the owner, group, and permissions are preserved, as previously described:

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id",
RESULT=="360060160a5b11c00420e82c83240dc11", NAME="crs%n", OWNER="oracle", GROUP="dba",
MODE="0660"

You can verify the validity of the rules with the command, udevtest:

[root@london1 rules.d]# udevtest /block/sdd
main: looking at device '/block/sdd' from subsystem 'block'
run_program: '/bin/bash -c '/sbin/lsmod | /bin/grep ^dm_multipath''
run_program: '/bin/bash' (stdout) 'dm_multipath           55257  0 '
run_program: '/bin/bash' returned with status 0
run_program: '/lib/udev/usb_id -x'
run_program: '/lib/udev/usb_id' returned with status 1
run_program: '/lib/udev/scsi_id -g -x -s /block/sdd -d /dev/.tmp-8-48'
run_program: '/lib/udev/scsi_id' (stdout) 'ID_VENDOR=DGC'
run_program: '/lib/udev/scsi_id' (stdout) 'ID_MODEL=RAID_0'
run_program: '/lib/udev/scsi_id' (stdout) 'ID_REVISION=0324'
run_program: '/lib/udev/scsi_id' (stdout) 'ID_SERIAL=360060160a5b11c00420e82c83240dc11'
run_program: '/lib/udev/scsi_id' (stdout) 'ID_TYPE=disk'
run_program: '/lib/udev/scsi_id' (stdout) 'ID_BUS=scsi'
...

Finally, reload the udev rules and restart udev. The devices that can be specified for the OCR and Clusterware voting disk have been created as devices, and they are now available for use. This ensures that a given name will preserved for a particular device across reboots:

[oracle@london1 ˜]$ ls -l /dev/crs*
brw-r----- 1 oracle dba 8, 48 Jul 31 15:44 /dev/crs
brw-r----- 1 oracle dba 8, 49 Jul 31 15:44 /dev/crs1

You can also use the command udevinfo to query the created device based on its name, as in this example:

udev udevinfo -q all -n /dev/crs