The 64-bit extension of x86 is called x86-64 but can also be referred to as x64, EM64T, Intel 64, and AMD64; however, the minor differences in implementation are inconsequential, and all of these names can be used interchangeably.

Two fundamental differences exist between x86, 32- and x86-64, 64-bit computing. The most significant is in the area of memory addressability. In theory, a 32-bit system can address memory up to the value of 2 to the power of 32, enabling a maximum of 4GB of addressable memory. A 64-bit system can address up to the value of 2 to the power of 64, enabling a maximum of 16 exabytes, or 16 billion GB, of addressable memory—vastly greater than the amount that could be physically installed into any RAC cluster available today. It is important to note, however, that the practical implementations of the different architectures do not align with the theoretical limits. For example, a standard x86 system actually has 36-bit physical memory addressability behind the 32-bit virtual memory addressability. This 36-bit physical implementation gives a potential to use 64GB of memory with a feature called Page Addressing Extensions (PAE) to translate the 32-bit virtual addresses to 36-bit physical addresses. Similarly, x86-64 processors typically implement 40-bit or 44-bit physical addressing; this means a single x86-64 system can be configured with a maximum of 1 terabyte or 16 terabytes memory respectively. You'll learn more about the practical considerations of the impact of the different physical and virtual memory implementations later in this chapter.

In addition to memory addressability, one benefit from moving to 64-bit registers is the processors themselves. With 64-bit registers, the processor can manipulate high-precision data more quickly by processing more bits in each operation.

For general-purpose applications x86-64 processors can operate in three different modes: 32-bit mode, compatibility mode, or 64-bit mode. The mode is selected at boot time and cannot be changed without restarting the system with a different operating system. However, it is possible to run multiple operating systems under the different modes simultaneously within a virtualized environment (see Chapter 5). In 32-bit mode, the processor operates in exactly the same way as standard x86, utilizing the standard eight of the general-purpose registers. In compatibility mode, a 64-bit operating system is installed, but 32-bit x86 applications can run on the 64-bit operating system. Compatibility mode has the advantage of affording the full 4GB of addressability to each 32-bit application. Finally, the processor can operate in 64-bit mode, realizing the full range of its potential for 64-bit applications.

This compatibility is indispensable when running a large number of 32-bit applications developed on the widely available x86 platform while also mixing in a smaller number of 64-bit applications; however, Oracle's published certification information indicates that 32-bit Oracle is not supported on a 64-bit version of Linux on the x86-64 architecture. That said, the architecture may be used for 32-bit Linux with 32-bit Oracle or for 64-bit Linux with 64-bit Oracle. The different versions cannot be mixed, though, and compatibility mode may not be used. To take advantage of 64-bit capabilities, full 64-bit mode must be used with a 64-bit Linux operating system, associated device drivers, and 64-bit Oracle—all must be certified specifically for the x86-64 platform.

The single most important factor for adopting 64-bit computing for Oracle is the potential for memory addressability beyond the capabilities of a 32-bit x86 system for the Oracle SGA. Additionally, the number of users in itself does not directly impact whether a system should be 32- or 64-bit. However, a significantly large number of users also depend on the memory handling of the underlying Linux operating system for all of the individual processes, and managing this process address space also benefits from 64-bit memory addressability. For this reason, we recommend standardizing on an x86-64 processor architecture for Oracle RAC installations. In addition, memory developments such as NUMA memory features are only supported in the 64-bit Linux kernel; therefore, the advantages of 64-bit computing are significantly enhanced on a server with a NUMA architecture as discussed later in this chapter.

For nearly all of the architectures available to run Linux, server I/O will be based on the Peripheral Component Interconnect (PCI). PCI provides a bus-based interconnection with expansion slots to attach additional devices, such as Network Interface Cards (NIC) or Fibre Channel host bus adapters (HBAs). For Oracle RAC in an enterprise environment, common requirements include the following: one external network interface; one backup network interface; two teamed network interconnect interfaces; and two teamed storage-based interfaces, which must be either network- or Fibre Channel–based. Of the six connections required, some may be satisfied by dual or quad cards. In some environments, such as a blade-based system, the connections are shared between servers. The expansion slots within a particular server may also have different specifications. Therefore, DBAs need to know whether the connections and bandwidth available from a particular environment will meet their requirements. Also note whether the PCI slots available are hot pluggable, which means that a failed card may be replaced without requiring server downtime.

The original implementation of PCI offered a 32-bit bus running at frequency of 33MHz. The following calculation shows that this presents 133 MB/s of bandwidth:

33 × 4 bytes (32 bits) = 133 MB/s

The bandwidth shown for PCI is shared between all of the devices on the system bus. In a RAC environment, the PCI bus would be saturated by a single Gigabit Ethernet connection. For this reason, the original 32-bit, 33MHz PCI bus has been extended to 64 bits at 266MHz. However, the 64-bit bus has also been extended to 100MHz and 133MHz, and it is now referred to as PCI-X. The configurations of PCI are summarized in Table 4-7.

PCI-X has sufficient capability to sustain a RAC node with gigabit-based networking and 1Gb- or 2Gb-based storage links. However, like PCI, PCI-X is a shared-bus implementation. With Ethernet, Fibre Channel, and Infiniband standards moving to 10Gb-based implementations and beyond, sufficient bandwidth is likely to be unavailable for all of the connections that an Oracle RAC node requires with PCI-X.

The consideration of the bandwidth requirements of 10Gb connections should be made in conjunction with selecting a platform that supports the generation of PCI called PCI-Express. PCI-Express should not be confused with PCI-X. In contrast to PCI-X's shared bus, PCI-Express implements a high-speed point-to-point serial I/O bus.

When reviewing a hardware specification for PCI connectivity, the relevant section will explicitly reference PCI-Express or its abbreviation of PCIe, as follows:

Six (6) available PCI-Express Gen 2 expansion slots

In contrast to PCI, a PCI-Express link consists of dual channels, implemented as a transmit pair and a receive pair, to enable bi-directional transmission simultaneously. The bandwidth of a PCI-Express link can be scaled by adding additional signal pairs for multiple paths between the two devices. These paths are defined as x1, x4, x8, and x16, according to the number of pairs. Like memory configuration, the desired PCIe configuration should also be cross-referenced against the supported configuration of the system chipset. For example, if the chipset is described as supporting 36 PCIe lanes, this means it will support two x16 links and an x4 link; one x16, two x8 links, and an x4 link; or any supported combination that is also dependent on the configuration of the physical slots available on the motherboard.

The unencoded bandwidth is approximately 80% of the encoded data transfer rate PCI-Express uses a form of encoding that utilizes the remaining 20%. The encoding enables better synchronization, error detection, and error resolution. For this reason, the transfer rate is often written as GT/s (Gigatransfers per second) to distinguish the actual Gb/s transfer rate from the rate at which the data itself is transmitted. Each link consists of dual channels and is bi-directional. In the original implementation, each lane supported 250MB/s. However, the revised PCI Express Base 2.0 specification released in 2007 doubles the clock frequency from the maximum of 1.25GHz at the original implementation to a maximum of 2.5Ghz at the revised specification, which is commonly known as Gen 2. This effectively doubles the bandwidth per lane to 500MB/s. Table 4-8 illustrates the combined bandwidth of both directions with an x1 link that supports a total bandwidth of 1Gb/s.

At the time of writing, PCI Express Gen 3 is under development, this specification increases the frequency to 4GHz and the bandwidth per lane to 1GB/s.

Because it is based on a point-to-point architecture, PCI-Express will support 10Gb and higher links to the interconnect and storage without sharing bandwidth. PCI-Express adapters also natively support features that are usually managed at the system-board level with PCI and PCI-X, such as hot plug, hot swap, and advanced power management. You can use the command dmidecode as described previously for memory lspci to query your platform configuration for PCI-Express support. Do so using this format:

[root@london1 ˜]# /sbin/lspci | grep -i express
00:01.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port 1 (rev 12)
00:02.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port 2 (rev 12)
00:03.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port 3 (rev 12)
...

When considering Infiniband as an interconnect technology, it is important to note that Infiniband also implements a serial architecture-based protocol in its own right. Therefore, Infiniband does not, by definition, require PCI-Express connectivity with the option of Infiniband Landed on Motherboard (LOM) providing a direct system connection. That said, such configurations are not particularly common and tend to be focused more in the direction of HPC computing. More common Infiniband implementations feature Infiniband implemented with PCI-Express HBAs, which are also known as Host Channel Adapters (HCAs). The latter approach enables the selection of an industry standard server platform, while also adding Infinband as an additional technology. For this reason, we recommend systems with expansion slots that support PCI and PCI-derived technologies with bandwidth sufficient for your Oracle RAC requirements.

The most popular choice for an Oracle RAC cluster interconnect is a Gigabit Ethernet switched network. Gigabit Ethernet connectivity is available through dedicated HBAs inserted into the PCI slots on a server. However, almost without exception, industry-standard servers will include at least two Gigabit Ethernet ports directly on the server motherboard. These industry-standard servers provide Ethernet connectivity without requiring additional hardware, thus leaving the additional PCI slots free for further connectivity.

Gigabit Ethernet run in full duplex mode with a nonblocking, data center class switch should be the minimum interconnect standard applied.

Gigabit Ethernet theoretically supports transfer rates of 1,000Mb/s (1000 Megabits are equivalent to 100 Megabytes) and latencies of approximately 60 to 100 microseconds for shorter packet sizes. A clear difference exists between this latency and the two to four milliseconds we previously specified for the average receive time for a consistent read or current block for Cache Fusion. In addition to the Oracle workload, processing the network stack at the Linux operating system level is a major contributing factor to the latency of Cache Fusion communication.

For standard TCP/IP networking, reading from or writing to a network socket causes a context switch, where the data is copied from user to kernel buffers and processed by the kernel through the TCP/IP stack and Ethernet driver appropriate to the NIC installed. Each stage in this workload requires a degree of CPU processing, and tuning kernel network parameters (you can learn more about this in Chapter 6) can play a role in increasing network throughput. To minimize the overhead of network processing, the default protocol for Oracle 11g RAC on Linux is the User Datagram Protocol (UDP), as opposed to Transmission Control Protocol (TCP). UDP is a non-connection-oriented protocol that doesn't guarantee data ordering. It also places the responsibility for verifying the data transmitted on the application itself—Oracle, in this case. The benefit of this approach is this: UDP reduces the kernel networking requirements.

At the time of writing, 10 Gigabit Ethernet (10GbE) technology is becoming more widely available and has recently been supported within the RAC Technologies Matrix. As 10GbE adoption increases, driven by reduction in cost and 10GbE ports available directly on server motherboards, it is likely to become the standard implementation over time, offering improved levels of bandwidth in addition to lower latencies compared to gigabit Ethernet. Therefore, we recommend monitoring the status of 10GbE closely. Given the required levels of cost, availability, and support for 10GbE, its adoption should be viewed as a progressive development of Ethernet, and it should be adopted where reasonably possible. When implementing 10 Gigabit Ethernet over copper cabling, it is important to also use the correct cabling for such an implementation; Cat 5 or 5e is commonly used for Gigabit Ethernet, while 5e or 6 is commonly used for 10 Gigabit Ethernet.

In a standard environment with a two-node cluster and no resiliency, each node connects to a single interconnect (Oracle does not recommend running the Clusterware and Database interconnect on separate networks) with a single NIC, in addition to the public network and any additional networks, such as the backup network. Figure 4-4 shows the interconnect network for simplicity.

Figure 4.4. Nonredundant interconnect

If a single interconnect NIC fails, Oracle Clusterware will attempt to reconnect for a user-defined period of time before taking action to remove the node from the cluster. By default, the amount of time it will wait is defined by the 11g Clusterware CSS Misscount value as 30 seconds. When this happens, the master node directs the Oracle database Global Cache Service to initiate recovery of the failed instance. However, if the interconnect network fails because, for example, the network switch itself fails, then a scenario will result that is equivalent to the failure of every single node in the cluster, except for the designated master node. The master node will then proceed to recover all of the failed instances in the cluster before providing a service from a single node. This will occur irrespective of the number of nodes in the cluster.

Because the master node must first recover all instances, this process will result in a significant reduction in the level of service available. Therefore, we recommend the implementing a fully redundant interconnect network configuration.

A common conisdieration is to implement the Oracle CLUSTER_INTERCONNECTS parameter. This parameter requires the specification of one or more IP addresses, separated by a colon, to define the network interfaces that will be used for the interconnect. This network infrastructure is configured as shown in Figure 4-5.

Figure 4.5. Clustered interconnects

The CLUSTER_INTERCONNECTS parameter however is available to distribute the network traffic across one or more interfaces, enabling you to increase the bandwidth available for interconnect traffic. This means it is most relevant in a data warehousing environment. The parameter explicitly does not implement failover functionality; therefore, the failure of any interconnect switch will continue to result in the failure of the entire interconnect network and a reduced level of service. Additionally, Oracle does not recommend setting the CLUSTER_INTERCONNECTS parameter, except in specific circumstances. For example, the parameter should not be set for a policy-managed database (you will learn more about workload management in Chapter 11).

To implement a fully redundant interconnect configuration requires the implementation of software termed NIC bonding at the operating system level. This software operates with the network driver level to provide two physical network interfaces that operate as a single interface. In its simplest usage, this software provides failover functionality, where one card is used to route traffic for the interface, and the other remains idle until the primary card fails. When a failure occurs, the interconnect traffic is routed through the secondary card. This occurs transparently to Oracle—its availability is uninterrupted, the IP address remains the same, and the software driver also remaps the hardware or MAC address of the card, so that failover is instantaneous. The failover is usually only detectable within Oracle by a minimal IPC time-out wait event. It is also possible in some configurations to provide increased bandwidth. This would deliver a similar solution to the CLUSTER_INTERCONNECTS parameter, with the additional protection of redundancy.

When implementing a bonding solution, understanding the implications of NIC failure, as opposed to failure of the switch itself, is important. Consider the physical implementation shown in Figure 4-5, where two nodes are connected separately to two switches. In this scenario, if a primary NIC fails on either of the nodes, that node will switch over to use its secondary NIC. However, this secondary NIC is now operating on a completely independent network from the primary card that still remains operational on the fully functional node. Communication will be lost, and the Oracle Clusterware will initiate cluster reconfiguration to eject the non-master node from the cluster.

Now considering the physical implementation shown in Figure 4-6, where the two nodes are connected separately to the two switches. In this case, the two switches are also connected with an interswitch link or an external network configuration, enabling traffic to pass between the two interconnect switches. As with the previous scenario, when a NIC fails, the network traffic is routed through the secondary interface. However, in this case, the secondary interface continues to communicate with the primary interface on the remaining node across the interswitch link. In a failover mode when no failures have occurred, only the primary switch is providing a service. After a failure, both switches are active and routing traffic between them.

Figure 4.6. Fully redundant clustered interconnects

Crucially, this failover scenario also guards against the failure of a switch itself. While operational, if the secondary switch fails, the network traffic remains with the primary interconnect, and no failover operations are required. However, if the primary switch fails, the driver software reacts exactly as it would if all of the primary NICs on all of the nodes in the cluster had failed at exactly the same time. All of the NICs simultaneously switch from their primary to their secondary interface, so communication continues with all of the network traffic now operating across the secondary switch for all nodes. In this solution, there is no single point of failure because either a single NIC, network cable, switch, or interswitch link can fail without impacting the availability of the cluster. If these components are hot-swappable, they can also be replaced, and the fully redundant configuration can be restored without impacting the availability of the clustered database.

This form of teaming requires interaction with the network driver of the NICs used for bonding. Therefore, this approach is best implemented with channel bonding in an active-backup configuration in the widest range of cases (see Chapter 6 for more information on this subject). The implementation of NIC bonding in a broadcast configuration is not recommended for the high availability demands of an Oracle RAC configuration. In addition to a high availability configuration, a load balancing configuration may be implemented, depending upon the support for such a configuration at both the network hardware and driver levels.

The specifications of Infiniband describe a scalable, switched, fabric-based I/O architecture to standardize the communication between the CPU and peripheral devices. The aim of Infiniband was to unify and replace existing standards such as Ethernet, PCI, and Fibre Channel, so as to move beyond static server configurations to a dynamic, fabric-based data center environment where compute power can be added separate from the devices that provide the data to be processed.

The evolution of technologies such as PCI-Express and 10 Gigabit Ethernet testify to the fact that Infiniband did not wholly succeed in its initial design goal; however, the technology has become established as a standard for the interconnect between compute nodes in high performance computing clusters, and more recently, as the connectivity technology utilized by the Oracle Exadata Storage Server.

Infiniband is similar to PCI-Express in the way it implements a serial architecture that aggregates multiple links. Infiniband supports 2.5Gb/s per link. It also includes support for double data rate (DDR), which increases this to 5Gb/s; and quad data rate (QDR), which increases its throughput to 10 Gb/s at the same frequency. The most common system interconnect implementation uses 4x links, which results in bandwidth of 20Gb/s and 40Gb/s; however, also like PCI-Express encoding, this reduces the available bandwidth to 16 Gb/s and 32 Gb/s respectively. In addition to its bandwidth benefits, Infiniband latencies are also measured in microseconds, with a typical value of 10 microseconds. This means Infiniband offers significant latency gains over Ethernet, as well as a reduction in the CPU utilization required for I/O.

To implement Infiniband in an Oracle RAC environment, IP over Infiniband is supported. However, the optimal solution with Oracle 11g on Linux is to use the Reliable Datagram Sockets (RDS) protocol for its lower utilization of CPU, rather than Infiniband over IB. It is important to note that the supported implementation and version of RDS for Linux is the Open Fabrics Enterprise Distribution (OFED) 1.3.1 for RDS version 3, which is not included in either Red Hat or Oracle Enterprise Linux by default. This means it must be downloaded from Oracle as patch 7514146 and installed separately. It is also necessary to relink the binary when both the ASM and database instance are shutdown to use RDS, as shown in the following lines:

[oracle@london1 ˜]$ cd $ORACLE_HOME/rdbms/lib
[oracle@london1 lib]$ make -f ins_rdbms.mk ipc_rds ioracle

Once relinked, Oracle RDS must be successfully configured and used. Note that both RDS and UDP cannot be configured at the same time. Therefore, returning to the default UDP settings requires relinking the binary again:

[oracle@london1 lib]$ make -f ins_rdbms.mk ipc_g ioracle

Infiniband fabrics are typically implemented with the focus on providing bandwidth in a clustered environment. For this reason, they are often configured in what is known as a Fat Tree Topology or Constant Bisectional Bandwidth (CBB) network to support a large number of nodes in a non-blocking switch configuration with multiple levels. This is in contrast to the hierarchical configuration of a typical Ethernet-based interconnect. However, it is important to note that the bandwidth requirements of Oracle RAC are typically considerably lower than the HPC supercomputing environments where Infiniband is typically deployed. Therefore, we recommend verifying the benefits of such an approach before deploying Infiniband as your interconnect technology.

In the absence of compelling statistical evidence that you will benefit from using a higher performance interconnect, we recommend selecting the default UDP protocol running over Ethernet at a gigabit or 10 gigabit rating for a default installation of an Oracle RAC cluster. Ethernet offers a simpler, more cost-effective interconnect method than Infiniband; and where cost is a decisive factor, implementing Infiniband should be compared against the alternative of 10 Gigabit Ethernet, coupled with high-performance server processors and the RAM to provide an adequate buffer cache on all server nodes. In other words, you need to consider your entire solution, rather than focusing on your method of interconnect entirely in isolation. To that end, we recommend comprehensive testing of your application in a RAC environment with multiple node and interconnect configurations as the basis for moving beyond a default configuration. As you have seen, Ethernet also offers greater potential for building redundancy into the interconnect configuration—and at a significantly lower cost.

There are a handful of exceptions where an Infiniband interconnect may prove a valuable investment. For example, it might make sense when implementing large data warehousing environments where high levels of inter-instance parallel processing capabilities are required. Additionally, Infiniband may prove worthwhile when looking to consolidate both the interconnect and storage fabric into a single technology with sufficient bandwidth to accommodate them both. However, in the latter circumstance, 10 Gigabit Ethernet should also be considered, to see if it can meet the increased bandwidth requirements.

In this section, we have used the terms random reads and sequential reads from a storage perspective. However, the Oracle wait event usually associated with a sequential read realized on the storage is db file scattered read, whereas the Oracle wait event usually associated with a random read on the storage is db file sequential read. Therefore, clarifying the differing terminology seems worthwhile.

Within Oracle, the reading of data blocks is issued from the shadow process for a user's session. A scattered read is a multiblock read for full-table scan operations where the blocks, if physically read from the disk storage, are normally accessed contiguously. The operations required to read the blocks into the buffer cache are passed to the operating system, where the number of blocks to fetch in a single call is determined by the Oracle initialization parameter, DB_FILE_MULTIBLOCK_READ_COUNT. The long-standing methodology for UNIX multiple buffer operations is termed scatter/gather. In Linux, the SCSI generic (sg) packet device driver introduced the scatter/gather I/O in the 2.2 kernel. For Oracle on Linux, a similar methodology is employed, where scattered reads are issued with the pread() system call to read or write multiple buffers from a file descriptor at a given offset into the user session's program global area (PGA). Subsequently, these buffers are written into noncontiguous buffers in the buffer cache in the SGA. The key concept to note is that the data may be retrieved from the disk in a contiguous manner, but it is distributed into noncontiguous, or scattered, areas of the SGA. In addition to db file scattered read, in some circumstances you may also see the wait event direct path read associated with a profile of multiblock reads on the storage. The ultimate destination of the data blocks in a direct path read is the PGA, as opposed to the SGA. This is most evident when utilizing Parallel Query; hence, the typical profile on the storage is one of multiple, contiguous reads with the requests issued asynchronously by the Parallel Execution Servers. Parallel Execution in a RAC environment is discussed in more detail in Chapter 14.

A db file sequential read event, on the other hand, is associated with index-based reads and often retrieves a single block or a small number of blocks that are then stored contiguously within the SGA. A single block read, by definition, is stored in contiguous memory. Therefore, this form of I/O is termed sequential if it's physically read from the disk storage, despite the fact that the file locations for these blocks are accessed randomly.

For the reasons previously described in this chapter, the important aspect to note in terms of read storage activity for RAC is that accessing data blocks through Cache Fusion from other nodes in the cluster is preferable to accessing them from disk. This holds true, no matter which method you employ. However, the buffer cache should always be sufficiently sized on each node to minimize both Cache Fusion and physical disk reads, as well as to ensure that as much read I/O as possible is satisfied logically from the local buffer cache. The notable exception occurs with data warehouse environments, where in-memory parallel execution is not being used. In that case, the direct path reads are not buffered in the SGA, so sufficient and usually considerable read I/O bandwidth must be available to satisfy the combined read demands of the cluster.

In a transactional environment, the most important aspect of storage performance for RAC in an optimally configured system will most likely be the sequential writes of the redo logs. The online redo logs will never be read as part of normal database operations, with a couple of exceptions.

These exceptions are as follows: when one node in the cluster has failed, so the online redo logs are being read and recovered by another instance; and when the archiver (ARCn) process is reading an online redo log to generate the archive log. These activities, however, will not take place on the current active online redo log for an instance.

To understand why these sequential writes take prominence in a transactional environment, it's important to look at the role of the LGWR process. The LGWR process writes the changes present in the memory resident redo log buffer, which itself comprises a number of smaller individual buffers to the online redo logs located on disk. LGWR does not necessarily wait until a commit is issued before flushing the contents of the log buffer to the online logs; instead, by default, it will write when the log buffer is 1/3 full, _LOG_IO_SIZE is modified, every three seconds, or if posted by the DBWRn process—whichever occurs first, as long as LGWR is not already writing. Additionally, if a data block requested by another instance through Cache Fusion has an associated redo, then LGWR must write this redo to disk on the instance currently holding the block before it is shipped to the requesting instance.

Note that there are two important aspects of this activity in terms of storage utilization. First, in a high-performance OLTP environment, storage performance for redo logs is crucial to overall database throughput. Also, with Oracle 11g Release 2, the log buffer is sized automatically, so it cannot be modified from the default value. Second, issuing a rollback instead of a commit does not interrupt the sequential write activity of the LGWR process to the online redo logs. The rollback operation is completed instead from the information stored in the undo tablespace. Because these undo segments are also stored in the database buffer cache, the rollback operation will also generate redo, resulting in further sequential writes to the online redo logs.

When an Oracle client process ends a transaction and issues a commit, the transaction will not be recoverable until all of the redo information in the log buffer associated with the transaction and the transaction's system change number (SCN) have been written to the online redo log. The client will not initiate any subsequent work until this write has been confirmed; instead, it will wait on a log file sync event until all of its redo information is written.

Given sufficient activity, LGWR may complete a log file sync write for more than one transaction at a time. When this write is complete, the transaction is complete, and any locks held on rows and tables are released.

It is important to note that a log file sync is not totally comprised of I/O related activity, and it requires sufficient CPU time to be scheduled to complete its processing. For this reason, it is true that poorly performing storage will inevitably result in significant times being recorded on log file sync wait events. However, the converse is not necessarily true, that high log file sync wait events are always indicative of poorly performing storage. Scenarios associated with high CPU utilization moving to a system with a more advanced CPU or, to a lesser extent, increasing the scheduling priority of the LGWR process, can significantly increase the redo throughput of an apparently disk-bound environment.

Note that the modified blocks stored in the database buffer will not necessarily have been written to the storage at this point. In any event, these will always lag behind the redo information to some degree. The redo information is sufficient for database recovery; however, the key factor is the time taken to recover the database in the event of a failure. The more that the SCN in the online redo logs and archive logs is ahead of the SCN in the database datafiles, the longer the time that will be taken to recover.

Oracle RAC has multiple redo log threads—one for each instance. Each instance manages its own redo generation, and it will not read the log buffer or online redo logs of another instance while that instance is operating normally. Therefore, the redo log threads operate in parallel.

For RAC, all of the redo log threads should be placed on storage with the same profile to maintain equilibrium of performance across the cluster. The focus for redo should also be placed on the ability of the storage to support the desired number of I/O operations per second (IOPS) and the latency time for a single operation to complete, with less emphasis placed on the bandwidth available.

In addition to the sequential write performance associated with the redo log threads, other important storage aspects for transactional systems include the random reads and writes associated with the buffer cache and the writes of the DBWRn process. For 11g, the actual number of DBWR processes configured is either 1 or CPU_COUNT/8, whichever is greater. Like the LGWR process, DBWRn has a three-second time-out when idle. And when active, DBWRn may write dirty or modified blocks to disk. This may happen before or after the transaction that modified the block commits. This background activity will not typically have a direct impact on the overall database performance, although the DBWRn process will also be active in writing dirty buffers to disk, if reading more data into the buffer cache is required and no space is available.

For example, when a redo log switch occurs or the limits set by the LOG_CHECKPOINT_TIMEOUT or LOG_CHECKPOINT_INTERVAL parameter are reached for a particular instance, a thread checkpoint occurs, and every dirty or modified block in the buffer cache for that instance only will be written to the datafiles located on the storage. A thread checkpoint can also be manually instigated on a particular instance by the command alter system checkpoint local. A thread checkpoint ensures that all changes to the data blocks from a particular redo log thread on an instance up to the checkpoint Number, the current SCN (System Change Number), have been written to disk. This does not necessarily mean that the resulting DBWRn activity is limited to the local instance. As discussed in Chapter 2, when another instance requires the data block already in use, that data block is shipped via the interconnect through Cache Fusion to the requesting instance, while the original instance maintains a copy of the original block called a past image. The past image must be held until its block master signals whether it can be written to disk or discarded. In this case, it is the DBWRn process of the instance with the most recent past image prior to the checkpoint Number that does the write to disk. From a storage perspective, however, this is not additional disk write activity in a RAC environment. Instead, DBWR Fusion writes are a result of the transfer of current data blocks between instances through Cache Fusion, without the modified blocks having been written to disk beforehand. Therefore, the most recent committed changes to a data block may reside on another instance. Without Cache Fusion, writes of modified blocks to disk to be read by another instance are known as DBWR Cross-Instance writes and involves an increased DBWRn workload. With DBWR Fusion writes, any additional overhead is in interconnect related messaging, as opposed to the storage level.

A full checkpoint on all instances, known as a Global Checkpoint, writes out all changes for all threads and requires the command alter system checkpoint global, which is also the default if local or global is not specified. Checkpoint details can be reported in the alert log by setting the parameter log_checkpoints_to_alert to the value of TRUE. Doing so reports that a checkpoint occurred up to a particular redo byte address (RBA) in the current redo log file for that instance.

Setting the parameter FAST_START_MTTR_TARGET to a time value in seconds of between 0 and 3600, while unsetting LOG_CHECKPOINT_TIMEOUT and LOG_CHECKPOINT_INTERVAL, calculates the correct checkpoint interval based on the desired recovery time in a process called fast start checkpointing. Fast start checkpointing has been available since Oracle 9i; therefore, setting LOG_CHECKPOINT_TIMEOUT and LOG_CHECKPOINT_INTERVAL is not recommended. Additionally, since release 10g, automatic checkpoint tuning is enabled if the FAST_START_MTTR_TARGET parameter is not set or is set to a sufficiently large value, which means that potentially no checkpoint related parameters are required to be set. Note that setting the FAST_START_MTTR_TARGET is not mandatory; nevertheless, we recommend setting this parameter in a RAC environment to enable fast start checkpointing. Doing so is preferable to relying entirely upon automatic checkpoint tuning because, in a RAC environment, checkpoint tuning also impacts recovery from a failed instance, in addition to crash recovery from a failed database.

It is important to be aware that the parameter FAST_START_MTTR_TARGET includes all of the timing required for a full crash recovery, from instance startup to opening the database datafiles. However, in a RAC environment, there is also the concept of instance recovery, whereby a surviving instance recovers from a failed instance. To address this issue, the _FAST_START_INSTANCE_RECOVERY_TARGET parameter was recommended by Oracle for RAC at version Oracle 10g Release 2 as a more specific way to determine the time to recover from the failed instance. However, this parameter was only relevant if one instance in the cluster failed, and it's no longer available for Oracle 11g Release 2. Therefore, we recommend setting the parameter FAST_START_MTTR_TARGET in a RAC environment. If you set this value too low for your environment and checkpointing is frequent, then the workload of the DBWRn process will be high, and performance under normal workloads is likely to be impacted. At the same time, the time to recover a failed instance will be comparatively shorter. If the FAST_START_MTTR_TARGET parameter is set too high, then DBWRn activity will be lower, and the opportunity for throughput will be higher. That said, the time taken to recover from a failed instance will be comparatively longer. FAST_START_MTTR_TARGET can be set dynamically, and the impact of different values can be observed in the column ESTIMATED_MTTR in the view V$INSTANCE_RECOVERY. For recovery from a failed instance, the instance startup time and total datafile open time should deducted from the ESTIMATED_MTTR value and can be read in the columns INIT_TIME_AVG and the number of datafiles multiplied by FOPEN_TIME_AVG in X$ESTIMATED_MTTR respectively.

During a recovery operation, the FAST_START_PARALLEL_ROLLBACK parameter determines the parallelism of the transactional component recovery operation after the redo apply. By default, this parameter is set to a value of LOW, which is twice the CPU_COUNT value. Given sufficient CPU resources, this parameter can be set to a value of HIGH to double the number of processes compared to the default value; the aim of changing this parameter is to improve the transaction recovery time.

The differing profiles of the LGWR and DBWRn processes on the nodes of the cluster will dictate the activity observed on the storage. There is more emphasis on LGWR activity in a transactional, as opposed to a data warehouse, environment. Some of the balance in activity between LGWR and DBWRn processes lies in the mean time to recover. But given redo logs of a sufficient size, a checkpoint on each node will most likely occur at the time of a log switch. This checkpoint will allow sufficient time for DBWRn to flush all of the dirty buffers to disk. If this flushing does not complete, then the Checkpoint Not Complete message will be seen in the alert.log of the corresponding instance, as follows:

Thread 1 cannot allocate new log, sequence 2411
Checkpoint not complete
  Current log# 4 seq# 2410 mem# 0: +REDO/prod/onlinelog/group_4.257.679813563
Sun Feb 27 05:51:55 2009
Thread 1 advanced to log sequence 2411 (LGWR switch)
  Current log# 1 seq# 2411 mem# 0: +REDO/prod/onlinelog/group_1.260.679813483
Sun Feb 27 05:52:59 2009
Thread 1 cannot allocate new log, sequence 2412
Checkpoint not complete
  Current log# 1 seq# 2411 mem# 0: +REDO/prod/onlinelog/group_1.260.679813483
Sun Feb 27 05:53:28 2009
Thread 1 advanced to log sequence 2412 (LGWR switch)
  Current log# 2 seq# 2412 mem# 0: +REDO/prod/onlinelog/group_2.259.679813523

When this error occurs, throughput will stall until the DBWRn has completed its activity and enabled the LGWR to allocate the next log file in the sequence. When the FAST_START_MTTR_TARGET parameter is set to a non-zero value, the OPTIMAL_LOGFILE_SIZE column in the V$INSTANCE_RECOVERY view is populated with a value in kilobytes that corresponds to a redo log size that would coincide a checkpoint with a log file switch, according to your MTTR target. Consequently, a longer recovery target will coincide with the recommendation for larger redo log files. This value varies dynamically; therefore, we recommend using this value for general guidance, but mainly to help ensure that the log files are not undersized in conjunction with expected DBWRn performance to the extent that Checkpoint Not Complete messages are regularly viewed in the alert log. Although the precise redo log size is dependent on the system throughput, typically a range from 512MB to 2GB is an acceptable starting value for a RAC environment.

Without asynchronous I/O, every I/O request that Oracle makes to the operating system is completed singly and sequentially for a particular process, with the pread() system call to read and corresponding pwrite() system call for writes. Once asynchronous I/O has been enabled, multiple I/O requests can be submitted in parallel within a single io_submit() system call, without waiting for the requests to complete and subsequently be retrieved from the queue of completed events. Utilizing asynchronous I/O potentially increases the efficiency and throughput for both DBWRn and LGWR processes. For DBWRn, it also minimizes the requirement that you increase the values of DB_WRITER_PROCESSES beyond the default settings that may benefit synchronous I/O.

In an Oracle RAC on Linux environment, I/O processing is likely to benefit from enabling asynchronous I/O both in regular operations and for instance recovery. Asynchronous I/O support at the operating system level is mandatory for an Oracle 11g Release 2 installation, and it is a requirement for using ASM.

To utilize asynchronous I/O for Oracle on Linux, you need to ensure that the RPM package libaio has been installed according to the process described in Chapter 4:

[oracle@london1 ˜]$ rpm -q libaio
libaio-0.3.106-3.2

The presence of the libaio package is mandatory, even if you do not wish to use asynchronous I/O, because the Oracle executable is directly linked against it, and it cannot be relinked with the async_off option to break this dependency. The additional libaio-devel RPM package is checked for under package dependencies during the Oracle installation; and in x86-64 environments, both the 32-bit and 64-bit versions of this RPM are required.

Asynchronous I/O can be enabled directly on raw devices. However, with Oracle 11g Release 2, raw devices are not supported by the Oracle Universal Installer (OUI) for new installs. That said, raw devices are supported for upgrades of existing installations, and they may be configured after a database has been installed. Asynchronous I/O can also be enabled on file systems that support it, such as the OCFS2, and it is enabled by default on Automatic Storage Management (ASM) instances, which we discuss in Chapters 6 and 9, respectively. For NFS file systems, the Direct NFS Client incorporated within Oracle 11g is required to implement asynchronous I/O against NFS V3 storage devices.

Where asynchronous I/O has not been explicitly disabled, by default the initialization parameter DISK_ASYNCH_IO is set to TRUE. Thus, it will be used on ASM and raw devices when the instance is started. In fact, it is important to note that ASM is not itself a file system (see Chapter 9 for more details on this topic), so the compatible I/O is determined by the underlying devices, which are typically raw devices. For this reason, setting asynchronous I/O for ASM and raw devices implies the same use of the technology.

If the DISK_ASYNCH_IO parameter is set to FALSE, then asynchronous I/O is disabled. The column ASYNCH_IO in the view V$IOSTAT_FILE displays the value ASYNC_ON for datafiles where asynchronous I/O is enabled.

If you're using asynchronous I/O on a supporting file system such as OCFS2, however, you also need to set the parameter FILESYSTEMIO_OPTIONS, which by default is set to NONE. Setting FILESYSTEMIO_OPTIONS to anything other than the default value is not required for ASM. Oracle database files are stored directly in ASM; they cannot be stored in an Automatic Storage Management Cluster File System (ACFS), so FILESYSTEMIO_OPTIONS is not a relevant parameter for ACFS.

On a file system, this parameter should be set to the value ASYNCH by executing the following commands on one instance in the cluster:

[oracle@london1 lib]$ srvctl stop database -d PROD
SQL> startup nomount;

SQL>  alter system set filesystemio_options=asynch scope=spfile;

SQL> shutdown immediate;

[oracle@london1 lib]$ srvctl start database -d PROD

Once asynchronous I/O is enabled, there are a number of ways to ascertain whether it is being used. For example, if ASMLIB is not being used when running iterations of the command cat /proc/slabinfo | grep kio, changing values under kioctx and kiocb show that that asynchronous I/O data structures are being used. Additionally, during testing, the output of the operating system strace command run against the LGWR and DBWRn processes can be viewed for the use of the io_submit() and io_getevents() system calls. It is important to note, however, that if you're using ASMLIB on 2.6 kernel-based systems, only the read() system call is evident, even for write events. For example, the following snippet determines the process id of the LGWR process:

[oracle@london1 ˜]$ ps -ef | grep -i lgwr
oracle    5484     1  0 04:09 ?        00:00:00 asm_lgwr_+ASM1
oracle    5813     1  1 04:11 ?        00:02:32 ora_lgwr_PROD1
oracle   28506 13937  0 06:57 pts/2    00:00:00 grep -i lgwr

This process can then be traced with the output directed into a file until it is interrupted with Ctrl-C :

[oracle@london1 ˜]$ strace -af -p 5813 -o asynctest
Process 5813 attached - interrupt to quit
Process 5813 detached

Viewing this output in the file shows the use of the read() system call, even for the sequential writes of LGWR:

read(22,"MSA210P260306v3230017q202
327*"..., 80) = 80
read(22, "MSA210P260306v32"
..., 80) = 80

For this reason, when using ASMLIB the only determinant of the use of asynchronous I/O is by checking the value of the column ASYNCH_IO in the view V$IOSTAT_FILE. With Linux 2.6-based kernels, the size of the asynchronous I/O operations can no longer be modified with the kernel parameter, fs.aio-max-size. Instead, the size must be determined automatically.

Asynchronous I/O should not be confused with direct I/O. Direct I/O is an Oracle parameter set to avoid file system buffering on compatible file systems, and depending on the file system in question, it may be used either independently or in conjunction with asynchronous I/O. Remember: If you're using ASM, direct I/O is not applicable.

Direct I/O can be enabled in the initialization parameter file by setting the parameter FILESYSTEMIO_OPTIONS to DIRECTIO. If enabling both asynchronous I/O and direct I/O is desired on a compatible file system, then this parameter can alternatively be set to the value of SETALL, as in this example:

SQL>  alter system set filesystemio_options=directIO scope=spfile;

Alternatively, you can set the value like this:

SQL>  alter system set filesystemio_options=setall scope=spfile;

This means a combination of the parameters DISK_ASYNCH_IO and FILESYSTEMIO_OPTIONS can be used to fine-tune the I/O activity for the RAC cluster, where appropriate. However, adding such support should always be referenced against the operating system version and storage.

RAID 0 implements striping across a number of disks by allocating the data blocks across the drives in sequential order, as illustrated in Table 4-10. This table shows a stripe set of eight disks. The blocks from A to X represent, not Oracle blocks, but logical blocks of contiguous disk sectors, such as 128KB. In this example, we have one stripe against eight disks, which makes our total stripe size 1MB (8 × 128KB).

This implementation does not cater to redundancy; therefore, by the strictest definition, it is not RAID. The loss of a single drive within the set results in the loss of the entire RAID group. So, on one hand, this implementation offers a lower level of protection in comparison to the presentation of individual drives.

What it does offer, on the other hand, is improved performance in terms of operations per second and throughput over the use of individual drives for both read and write operations. The example in the table would in theory offer eight times the number of operations per second of an individual drive and seven times the average throughput for hard disks. The throughput is marginally lower with hard-disk drives due to the increased latency to retrieve all of the blocks in a stripe set because the heads must move in parallel on all drives to a stripe. On solid state-based storage, the average throughput is also a factor of eight.

For RAC, this implementation of RAID delivers the highest-performing solution; however, the lack of resilience makes it unsuitable for any production implementation. RAID 0 is heavily utilized, mainly for commercial database benchmarks. The combination of RAID 0 and destroking on hard-disk drives is ideal for an environment where data loss or the cost of unused disk space is not a significant consideration.

In RAID 1, all data is duplicated from one disk or set of disks to another, resulting in a mirrored, real-time copy of the data (see Table 4-11).

RAID 1 offers a minor performance improvement over the use of single drives because read requests can be satisfied from both sides of the mirror. However, write requests are written to both drives simultaneously, offering no performance gains. The main benefits of this form of RAID are that, in the event of a single drive failure or possibly multiple drive failures, the mirror is broken, but data availability is not impacted. This means that performance is only marginally impaired, if at all, until the drive is replaced. At this point, however, performance is impacted by the resilvering process of copying the entire contents of the good drive to the replacement, a lengthy and intensive I/O operation.

When implemented in software using a volume manager, the CPU load from mirroring is the lowest of any form of RAID, although the server I/O traffic is doubled, which should be a consideration in comparing software- to hardware-based RAID solutions.

The most significant cost of this form of RAID comes in using exactly double the amount of storage as that which is available to the database.

RAID 10 offers the advantages and disadvantages of both RAID 0 and RAID 1, first by mirroring all of the disks onto a secondary set, and then by striping across these mirrored sets. Table 4-12 shows this configuration.

This form of RAID is usually only available with hardware-based RAID controllers. It achieves the same I/O rates that are gained by striping, and it can sustain multiple simultaneous drive failures in which the failures are not experienced on both sides of a mirror. In this example, four of the drives could possibly fail with all of the data, and therefore the database, remaining available. RAID 10 is very much the implementation of SAME that Oracle extols; however, like RAID 1, it comes at the significant overhead of an additional 100% requirement in storage capacity above and beyond the database.

RAID 0+1 is a two-dimensional construct that implements the reverse of RAID 10 by striping across the disks and then mirroring the resulting stripes. Table 4-13 shows the implementation of RAID 0 across disks 1, 2, 5, and 6, mirrored against disks 3, 4, 7, and 8.

RAID 0+1 offers identical performance characteristics to RAID 10, and it has exactly the same storage-capacity requirements. The most significant difference occurs in the event of a drive failure. In the RAID 0+1 configuration, if a single drive fails—for example, Disk 1—then you lose access to the entire stripe set on disks 1, 2, 5, and 6. At this point, you only have to lose another disk on the stripe set on the other side of the mirror to lose access to all of the data and, thus, the database it resides on.

Therefore, you might reasonably question where a RAID 0+1 configuration would be used instead of RAID 10. If implementing RAID in hardware on a dedicated-storage array, then you would never use this approach; RAID 10 should always be used. However, if using software RAID, such as with Oracle ASM, in combination with any number of low-cost modular storage arrays, a RAID 0+1 configuration can be used to stripe the disks at the storage level for performance and then mirrored by software between multiple arrays for resilience.

RAID 5 introduces the concept of parity. In Table 4-14, you can see the eight disks striped similarly to RAID 0, except that you have a parity block at the end of each stripe. This parity block is the same size as the data blocks, and it contains the results of the exclusive OR (XOR) operation on all of the bits in every block in the stripe. This example shows the first three stripes; and if it were to continue across all of the disks, you would have seven disks of data and one disk of parity.

In this RAID configuration, the data can be read directly, just as it can in RAID 0. However, changing a data block requires writing the data block, and then reading, recalculating, and subsequently rewriting the parity block. This additional overhead for writes on RAID 5 is termed the write penalty. Note that, from the properties of the XOR operation for a write operation, touching the data block and the parity block is only necessary for a write operation; the parity can be calculated without it being required to access all of the other blocks in the stripe. When implemented on a hardware-based RAID controller, the impact of the parity calculation is negligible compared to the additional read and write operations, and the write penalty will range from 10% to 30%, depending on the storage array used. RAID 5 is less effective when implemented as a software solution because of the requirement to read all of the data, including the parity information, back to the server; calculate the new values; and then write all of the information back to the storage, with the write penalty being approximately 50%.

Recalling our discussion of storage fundamentals for RAC, this RAID configuration may appear completely unsuited to LGWR activity from a redo thread, presenting the system with a large number of sequential writes and no reads. From a theoretical standpoint, this unsuitability is true. However, good RAID 5-storage systems can take this sequential stream of writes and calculate the parity block without first needing to read the parity block from disk. All of the blocks and the parity block are written to disk in one action, similar to RAID 0; hence, in the example shown, eight write operations are required to commit blocks A to G to disk, compared to the fourteen required for RAID 10, although these can be completed in parallel.

The primary attraction of RAID 5 is that, in the event of a disk failure, the parity block means that the missing data for a read request can be reconstructed using the XOR operation on the parity information and the data from the other drives. Therefore, to implement resiliency, the 100% overhead of RAID 10 has been significantly reduced to the lower overhead of the parity disk. Unlike RAID 10, however, the loss of another drive will lead to a total data loss. The loss of a single drive also leads to the RAID 5 group operating in a degraded mode, with the additional load of reading all of the blocks and parity in the stripe to calculate the data in the failed block increasing the workload significantly. Similarly, when all of the drives are replaced, all of the blocks and the parity need to be read to regenerate the block, so it can be written to the replaced drive.

With parity-based RAID, an important area to look at is the impact of significantly larger disk media introduced since the original RAID concepts were defined. Despite the increase in disk size, the likelihood of a disk failure remains approximately the same. Therefore, for a RAID 5 configuration of a given capacity, the chances of operating in degraded mode or even losing data are comparatively higher. This risk has led to the emergence of RAID 50–based systems with mirrored RAID 5 configurations. However, RAID 50 offers further challenges in terms of the impact on performance for RAC.

In using the term cache, we are referring to the RAM set aside to optimize the transfer of the Oracle data from the storage, and we are primarily concerned with the storage-array controller cache. For data access, other levels of cache exist at the hardware level through which the Oracle data will pass, for example, the multiple levels of CPU cache on the processor discussed previously in this chapter, as well as the buffer cache on the hard drive itself. Note that both are usually measured in a small number of megabytes.

As we have seen previously in this chapter, for a transactional system, the storage will experience a number of random reads and writes, along with a continual stream of shorter sequential writes. Data warehouse activity will either buffer much of the data in the SGA for single threaded queries, or request long, continual streams of reads for parallel queries with minimal write and redo log activity outside of data loading times.

For write operations, the effect of cache on the storage controller can be significant to a point. If operating as a write-back cache, the write request is confirmed as complete as soon as the data is in cache, before it has been written to disk. For Oracle, this cache should always be mirrored in RAM and supplied with a backup battery power supply. In the event of failure, this backup enables the data within cache to be written to disk before the storage array shuts down, reducing the likelihood of database corruption. Write cache is also significant in RAID 5 systems because it enables the calculation of parity for entire stripes in cache, and it stores frequently used parity blocks in cache, thus reducing the read operations required.

This benefit from write cache, however, is valuable only to a certain extent in coping with peaks in throughput. In the event of sustained throughput, the data will be written to cache faster than the cached data can be written to disk. Inevitably, the cache will fill and throughput will operate at disk speed. Storage write cache should be seen as an essential buffer to enable the efficient writing of data back to disk. Allocating a large amount of cache, however, will never be a panacea for a slow disk layout or subsystem.

Of the generally available RAID configurations discussed, a popular choice is RAID 5. The reasons are straightforward: RAID 0 is not practical in terms of redundancy, and RAID 1 is not suitable for performance. Although RAID 10 and RAID 0+1 offer the best performance and reliability, they have the significant cost of requiring a 100% overhead in storage compared to usable capacity. In theory, RAID 5 offers a middle ground, sacrificing a degree of performance for resilience, while also maintaining most of the storage capacity as usable.

In practice, the costs and benefits are not as well defined. With RAID 1, RAID 0, and combinations of these levels, throughput calculations tend to be straightforward. With RAID 5, however, benchmarking and tests are required to clearly establish the performance thresholds. The reason for these testing requirements is that the advantages for RAID 5 tend to be stated from the viewpoint of a single sequential write stream, combined with random reads where performance is predictable. In practice, hosting more than one database environment on a storage system is highly likely; when adding RAC into the equation with multiple redo log streams, the I/O activity tends to be less identifiable than the theory dictates.

For example, as the RAID 5 system increases in activity, you are less likely to be taking direct advantage of storing and calculating your stripes in cache and writing them together to disk in one action. Multiple, single-logical-block updates will take at least four I/O operations for the parity updates, and the increased number of I/O operations will be compounded in the event of stripe crossing; that is, where the cluster file system or ASM stripe is misaligned with the storage stripe. Stripe crossing will result in more than one parity operation per write compounding the effect on throughput still further.

As more systems and activity are added to the RAID 5 storage, the impact becomes less predictable, meaning that RAID 10 is more forgiving of the practice of allocating storage ad hoc, rather than laying it out in an optimal manner. This difference between the theory and practice of RAID 5 and RAID 10 tends to lead to polarity between Oracle DBAs and storage administrators on the relative merits of RAID 5 and RAID 10. Both approaches are, in fact, correct from the viewpoints of their adherents.

In summary, when determining a RAID specification for RAC, RAID 10 with hardware is the optimal choice for fault tolerance, read performance, and write performance. Where RAID 5 is used, the careful planning, layout, and testing of the database across the storage can deliver a cost-effective solution, especially where the workload is predominantly read-only, such as a data warehouse. For a transactional system, RAID 10 for the redo log threads and RAID 5 for the datafiles can provide a practical compromise.

Some of the confusion regarding I/O in Linux stems from the fact that SCSI defines both a medium—that is, the physical attachment of the server to the storage—and the protocol for communicating across that medium. To clarify, here we refer to the SCSI protocol operating over a standard copper SCSI cable.

The SCSI protocol is used to define the method by which data is sent from the host operating system to peripherals, usually disk drives. This data is sent in chunks of bits—hence, the term block I/O–in parallel over the physical medium of a copper SCSI cable. Because this SCSI data is transmitted in parallel, all bits must arrive in unison.

The original implementation of SCSI in 1986 utilized an 8-bit data path at speeds of up to 5MB/s, and it enabled up to eight devices (including the host adapter itself) to connect to a single host adapter. Because of the signal strength and the deviation from the original source being transmitted, called jitter, the maximum distance a SCSI device could be from the host system using a high voltage differential was effectively limited to under 25m.

SCSI has subsequently been revised and updated to speeds of 320MB/s with up to 16 devices, although at a lower bus length of 12m using a low-voltage differential.

Each SCSI device has an associated target ID, and this target can be further divided into subdevices identified by LUNs. Because a server can have several host adapters, and each one may control one or more SCSI buses, uniquely identifying a SCSI device means that an operating system must account for the controller ID, the channel (or bus) ID, the SCSI ID, and the LUN. This hierarchy is precisely the one implemented in Linux for addressing SCSI devices, and it can be viewed with the following command:

[root@london1 ˜]# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: ST3500320NS      Rev: SN05
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi5 Channel: 00 Id: 00 Lun: 00
  Vendor: Slimtype Model: DVD A  DS8A1P    Rev: C111
  Type:   CD-ROM                           ANSI SCSI revision: 05

Host: scsi6 Channel: 00 Id: 00 Lun: 00
  Vendor: DGC      Model: RAID 0           Rev: 0324
  Type:   Direct-Access                    ANSI SCSI revision: 04
Host: scsi6 Channel: 00 Id: 00 Lun: 01
  Vendor: DGC      Model: RAID 0           Rev: 0324
  Type:   Direct-Access                    ANSI SCSI revision: 04

From this output, you can see that on SCSI host adapter 6, there are two drives on SCSI ID 0 and SCSI ID 1. This, this output indicates that the system has a single HBA, from which the two external disks are presented. Although /proc/scsi/scsi can still be used to view the SCSI disk configuration from the Linux 2.6 kernel, the SCSI configuration is in fact maintained under the /sys directory. For example, you can view the configured block devices using this snippet:

[root@london1 ˜]# ls -ld /sys/block/sd*
drwxr-xr-x 7 root root 0 Mar  8 23:44 /sys/block/sda
drwxr-xr-x 6 root root 0 Mar  8 23:44 /sys/block/sdb
drwxr-xr-x 6 root root 0 Mar  8 23:44 /sys/block/sdc

Similarly, you can use this snippet to determine the details of SCSI host adapter 6, through which the SCSI disks are connected:

[root@london1 ˜]# cat /sys/class/scsi_host/host6/info
Emulex LPe11000-M4 4Gb 1port FC: PCIe SFF HBA on PCI bus 0e device 00 irq 169

Finally, you can also view the major and minor device numbers for a specific device--in this case, the major number 8 and minor number 32. This which will prove useful for cross-referencing the physical storage layout with the configured disks, as discussed later in this section:

[root@london1 ˜]# cat /sys/class/scsi_device/6:0:0:1/device/block:sdc/dev
8:32

Using the /sys interface, you can also initiate a bus rescan to discover new devices, without needling to reload the relevant driver module, as in this snippet:

[root@london1 ˜]# echo "- - -" > /sys/class/scsi_host/host6/scan

The names of the actual SCSI devices, once attached, can be found in the /dev directory. By default, unlike some UNIX operating systems that identify devices by their SCSI bus address, Linux SCSI devices are identified by their major and minor device numbers. In earlier static implementations, there were either 8 or 16 major block numbers for SCSI disks. The initial 8 major block numbers were as follows: 8, 65, 66, 67, 68, 69, 70, and 71. The second set of 8 major block numbers added the following list of major numbers to the initial 8: 128, 129, 130, 131, 132, 133, 134, and 135.

Each major block number has 256 minor numbers, some of which are used to identify the disks themselves, while some others are used to identify the partitions of the disks. Up to 15 partitions are permitted per disk, and they are named with the prefix sd and a combination of letters for the disks and numbers for the partitions.

Taking major number 8 as an example, the first SCSI disk with minor number 0 will be /dev/sda, while minor number 1 will be the first partition, /dev/sda1. The last device will be minor number 255, corresponding to /dev/sdp15. This letter and number naming convention continues with /dev/sdq for major number 65 and minor number 0; next comes /dev/sdz15 at major number 65 and minor number 159, and this continues with /dev/sdaa at major number 65 and minor number 160. Within 2.6 kernels, device configuration is dynamic with udev based on the major and minor number for a device available under /sys (we discuss udev configuration in more detail in Chapter 6). With dynamic configuration, there is no restriction when creating a large number of static devices in advance. For example, the following shows all of the created SCSI disk devices on a system:

[root@london1 ˜]# ls -l /dev/sd*
brw-r----- 1 root disk 8,  0 Mar  8 23:44 /dev/sda
brw-r----- 1 root disk 8,  1 Mar  8 23:44 /dev/sda1
brw-r----- 1 root disk 8,  2 Mar  8 23:44 /dev/sda2
brw-r----- 1 root disk 8, 16 Mar  8 23:44 /dev/sdb
brw-r----- 1 root disk 8, 17 Mar  8 23:44 /dev/sdb1
brw-r----- 1 root disk 8, 32 Mar  8 23:44 /dev/sdc
brw-r----- 1 root disk 8, 33 Mar  8 23:44 /dev/sdc1

It is possible to use any available major number once the original 16 reserved major numbers have been allocated; and, as there are 4,095 major numbers, a 2.6 kernel based Linux implementation can support many thousands of disks, with the actual total number dependant on the sum total of all devices on the system. Also, /proc/partitions can be used to show the configured devices and their corresponding major and minor device numbers that correspond with the preceding directory listing:

[root@london1 ˜]# cat /proc/partitions
major minor  #blocks  name

   8     0  488386584 sda
   8     1     104391 sda1
   8     2  488279610 sda2
   8    16  104857600 sdb
   8    17  104856223 sdb1
   8    32  313524224 sdc
   8    33  313516476 sdc1
 253     0  467763200 dm-0
 253     1   20512768 dm-1

Although the device-naming convention detailed is the default, one with udev device names can be changed according to user-defined rules and persistent bindings created between devices and names. In some respects, such an approach offers similar functionality to what is offered by ASMLIB.

SCSI, by itself, as a protocol and medium, is not generally associated with the term SAN. First, the number of devices that can be connected to a single SCSI bus is clearly limited, and a strict SCSI implementation restricts the access to the bus to one server at a time, which makes it unsuitable for RAC. Second, although using link extenders to increase the length of SCSI buses is possible, doing so is impractical in a data-center environment, especially when compared to alternative technologies.

These disadvantages have led to advancements in SCSI, most notably the development of Serial Attached SCSI (SAS) to overcome the limitations of the parallel transmission architecture and to include the support of I/O requests from more than one controller at a time. However, these developments need to be balanced against overcoming the disadvantages of the SCSI medium with alternative technologies, such as Fibre Channel (FC).

Despite these disadvantages, the SCSI protocol itself remains the cornerstone of block-based storage for Linux. Its maturity and robustness have meant that the additional technologies and protocols used to realize SANs are implemented at a lower level than the SCSI protocol; therefore, even though SCSI cabling will rarely be used, the device naming and presentation will remain identical, as far as the operating system and Oracle are concerned.

FC was devised as a technology to implement networks and overcome the limitations in the standard at the time, which was Fast Ethernet running at 100Mb/s. Although FC is used to a certain extent for networks, these ambitions were never fully realized because of the advent of Gigabit Ethernet. However, the development of FC for networks made it compatible with the requirements for overcoming the limitations of SCSI-based storage as follows: transmission over long distances with a low latency and error rate, and the implementation of a protocol at a hardware level to reduce the complexities and CPU overheads of implementing a protocol at the operating system level.

These features enabled the connection of multiple storage devices within the same network; hence, the terminology, Storage Area Network. Note that although FC and SAN are often used interchangeably, they are not synonymous—a SAN can be implemented without FC, and FC can be utilized for other reasons apart from SANs. Also, similar to SCSI, with FC it's important to distinguish between the medium and the protocol. Despite the name, FC can be realized over copper or fiber-optic cabling, and the name Fibre Channel is correctly used to refer to the protocol for communication over either.

The FC protocol has five levels: FC-0 to FC-4. Levels FC-0 to FC-3 define the protocol from a physical standpoint of connectivity all the way through to communication. FC-4 details how the Upper Layer Protocols (ULPs) interact with the FC network. For example, it defines how the FC Protocol (FCP) implements the SCSI protocol understood by the operating system, with the FCP functionality delivered by a specific device driver. Similarly, IPFC implements the Internet Protocol (IP) over FC. There are also three separate topologies for FC: fabric, arbitrated loop, and point-to-point. By far, the most common of these topologies for RAC is the fabric topology, which defines a switch-based network enabling all of the nodes in the cluster to communicate with the storage at full bandwidth.

Optical cabling for FC uses two unidirectional fiber-optic cables per connection, with one for transmitting and the other for receiving. On this infrastructure, the SCSI protocol is transmitted serially, which means distances of up to 10 km are supported at high transfer rates. As well as supporting greater distances, fiber-optic cabling is also insensitive to electromagnetic interference, which increases the integrity and reliability of the link. The server itself must be equipped with an FC HBA that implements the FC connectivity and presents the disk devices to the host as SCSI disks.

At the time of writing, existing FC products support transfer rates of up to 8Gb/s (800 MB/s). However, these available rates must always be considered in conjunction with the underlying disk performance. For example, you must have a minimum of three high specification hard-disk drives in the underlying storage to outperform SCSI alone.

When implementing a SAN environment in a fabric topology, a method must be in place to distinguish between the servers to present the storage to. RAC is the perfect example to illustrate that no restriction exists in the number of hosts that are presented an identical view of the same storage. To realize this lack of restriction, all devices connected to a fabric are identified by a globally unique 64-bit identifier called a World Wide Name (WWN). When the WWN logs into a fabric, it is assigned a 24-bit identifier that represents the device's position within the topology, and zoning is configured at the switch layer to ensure that only the designated hosts have a view of their assigned storage. This explanation is, to some extent, an oversimplification, because for resilience systems can be equipped with multiple HBAs and connected to the same storage by different paths through separate switches to guard against hardware failure. This configuration is realized at the host level by multipathing software that unifies the multiple paths into a single disk image view (see Chapter 6 for a detailed implementation of I/O Multipathing with a device-mapper).

With these advantages of performance and connectivity over SCSI, FC has become the dominant protocol for SAN. However, it does have some distinct disadvantages that can inhibit realizing these benefits. One of the most significant of these is cost. The components for FC—the server HBA, optical cabling, and infrastructure—are significantly more expensive than their copper equivalents. Concerns have also been voiced about the lack of security implemented within the FC protocol, and the supported distances of up to 10 km are still a significant constraint in some environments.

In addition, arguably the most problematic area for Fibre Channel SAN, especially for RAC, is that of interoperability and support. To Oracle and the operating system, FC is interacting with SCSI devices, possibly with a clustered-file system layered on top, and could be using asynchronous and/or direct I/O. The device driver for the server HBA implements the SCSI interaction with the storage using FCP, and it could be running multipathing software across multiple HBAs. The storage is also likely to be dealing with requests from multiple servers and operating systems at the same time. Coupled with RAC, all of the servers in the cluster are now interacting simultaneously with the same disks. For example, one node in the cluster recovering after a failure could be scanning the storage for available disks, while the other nodes are intensively writing redo log information. You can see an exponential increase in the possible combinations and configurations. For this reason, FC storage vendors issue their own compatibility matrices of tested combinations, of which clustering—Oracle RAC, in particular—is often an included category. These vendors often provided detailed information about supported combinations, such as the Oracle and Linux versions, the number of nodes in the cluster, multipathing software support, and HBA firmware and driver versions. In reality, this certification often lies behind the curve in Oracle versions, patches, and features; in Linux releases and support; in server architectures; and in many other factors, which make the planning and architecting of the entire stack for compatibility a crucial process when working on FC SAN-based systems. Storage vendors should always be consulted with respect to support at the planning stage of an FC-based RAC solution on Linux.

A recent development is the introduction of Fibre Channel over Ethernet (FCoE), which seeks to mitigate against some of the complexities of Fibre Channel, while also leveraging its advantages and dominance in SAN environments. FCoE replaces the FC-0 and FC-1 layers with Ethernet, thereby enabling Fibre Channel installations to be integrated into an Ethernet-based network. From an Ethernet-based technology perspective, 10GbE is the key driving technology, providing sufficient bandwidth for Ethernet networks to host both networking and storage traffic simultaneously—an architectural approach termed Unified Fabrics. This approach is of particular interest in RAC environments that require a single networking technology for both the cluster interconnect and SAN traffic, thereby increasing the flexibility of a clustered solution. A similar Unified Fabric approach can also be adopted with Infiniband, although with Ethernet as its underlying technology, FCoE integrates directly with the SAN at the network layer, as opposed to replacing both the network and SAN with a new protocol. At the time of writing, FCoE is still classed as an emerging technology, albeit one that, given sufficient levels of adoption, brings considerable potential to simplifying a Grid-based computing approach.

In addition to FCoE, there are three other competing protocols for transporting block-based storage over an IP-based network: Internet Small Computer Systems Interface (iSCSI), Internet FC Protocol (iFCP), and FC over TCP/IP (FCIP).

FCIP supports tunneling of FC over an IP network, while iFCP supports FCP level 4 implemented over the network. Whereas these protocols are often implemented in environments looking to interconnect or interoperate with existing FC-based SANs, the leader is iSCSI, which makes no attempt to implement FC. Instead, it defines a protocol to realize block-based storage by encapsulating SCSI into the Transmission Control Protocol (TCP) to be transmitted over a standard IP network. This native use of TCP/IP means that an existing Gigabit Ethernet infrastructure can be used for storage, unifying the hardware requirements for both storage and networking. The major benefits that stem from this unification include reducing cost of implementation, simplifying the administration of the entire network, and increasing the distance capabilities through the existing LAN or WAN—all while using well-understood IP-based security practices.

In iSCSI terminology, the client systems, which in this case are the RAC cluster nodes, require an iSCSI initiator to connect the storage target. Initiators can be realized in either software or hardware. The software initiator is, in effect, a driver that pairs the SCSI drivers with the network drivers to translate the requests into a form that can be transferred. This translation can be then be used with a standard Ethernet network interface card (NIC) connected to the storage by Category-5 cabling.

Like FC, once successfully implemented, iSCSI lets Oracle and Linux view and interact with the disks at a SCSI protocol level. Therefore, Oracle needs to be based on ASM-configured block devices or a cluster-file system on the storage presented. iSCSI software drivers for Linux are available as part of the defaultLinux install and the configuration of iSCSI is on Linux is detailed in Chapter 5, in the context of configuring storage for virtualization.

An alternative for iSCSI is the use of specialized iSCSI HBAs to integrate the entire iSCSI protocol stack of Ethernet, TCP/IP, and iSCSI onto the HBA, removing all of I/O protocol processing from the server. At the time of this writing, these HBAs are available at gigabit speeds over standard Category-5 cabling. Products supporting 10Gb speeds are available with fiber-optic cabling or the copper-based 10GBase-CX4 cable associated with InfiniBand. The downside: The costs approach those of the better-established FC, despite the improved performance.

Serial Advanced Technology Attachment (SATA) has received a degree of attention as a means of providing low-cost, high-capacity storage in comparison to SCSI- and FC-based disks. This lower cost for SATA, however, comes with some significant disadvantages when deployed with standard hard disks. SATA is based on a point-to-point architecture in which disk requests cannot be queued; therefore the maximum throughput is significantly lower compared to its higher-cost rivals, especially for random I/O. Rotational speeds are also lower, so latency tends to be higher for SATA. In its original specification, SATA supports throughput of 1.5Gb/s (150MB/s).

In addition, the maximum cable length for SATA is 1 m, introducing some physical limitations that mean, in practice, SATA–based arrays need to be connected to the target host systems by existing SAN or NAS technologies.

The most significant disadvantage cited for SATA is that the mean time between failures (MTBF) is up to three times that of SCSI- or FC-based disks. However, this measurement is based on SATA implementations of hard-disk drives. SATA is also a protocol commonly used for SSDs; therefore, the MTBF values should not be taken as applying to all SATA-based storage. It is also worth noting that, although SATA is commonly used for SSDs, it is not an exclusive requirement, and the relationship is primarily driven by the production of SSDs that are widely compatible with notebook, desktop, and server implementations. For example, SSDs are also available with a Fibre Channel interface.

In a RAID configuration with SATA, the maximum practical limit is 12 to 16 devices per controller. This means that database storage using this type of disk technology often requires the management of multiple storage arrays from the target-host level; hence, the applicability of volume-management software, such as ASM. As we noted earlier in our discussion of RAID 0 and RAID 1, using multiple storage arrays to achieve satisfactory levels of performance dictates that RAID 0+1 be used to mirror arrays at a host level of disks striped at the array level.

SATA has advanced to the SATA II specification, which has doubled the throughput to 3 Gb/s (300MB/s). There is also an additional draft specification that increases throughput to 6 Gb/s. The additional specification was introduced in 2008. When used in conjunction with SSDs, this means that SATA can also compete with SCSI and Fibre Channel disks in the performance sector. SSDs take advantage of some of the performance features available with SATA II—in particular, the Native Command Queuing (NCQ) that under the specification permits queuing up to 32 requests per device.

Once the block storage is available on all of the hosts in the cluster and visible in /proc/scsi/scsi, you have a choice about how to configure the devices in a form for use by Oracle on more than one system simultaneously. Standard file systems such as ext2 or ext3 are often used for single-instance environments. However, these file systems are not applicable for use in clustered environments where data may be cached and blocks modified on multiple nodes without synchronization between them, resulting in the corruption of the data.

From a protocol standpoint, the simplest approach is for Oracle to use the devices directly as raw character devices. However, from a technical standpoint, although raw devices are supported for upgrades and can be configured from the command line with SQLPLUS, they are no longer supported within OUI for new installs. Also, Oracle has indicated that raw devices will not be supported in releases subsequent to Oracle 11g. On the other hand, block devices are the storage underlying Oracle ASM, as discussed in Chapter 9, and these devices must be used by ASM for storing database files without an intervening file system, including ACFS.

The alternative to using the block devices configured with ASM is to use the corresponding block-based device with a file system designed specifically for operating in a clustered environment that also supports the storage of database files. In Chapter 5 we look at how to configure OCFS2 in the context of virtualization.

I/O scheduling governs how the order of processing disk I/O requests is determined. In Linux, I/O scheduling is determined by the Linux kernel; with 2.6 Linux kernel releases, four different I/O schedulers are available. Selecting the most appropriate scheduler may have an influence on the performance of your block-based storage. The schedulers available are as follows:

The scheduler can be selected by specifying the elevator option of the Linux boot loader. For example, if you are using GRUB, the following options would be added to the /boot/grub/grub.conf file: CFQ uses elevator=cfq, Deadline uses elevator=deadline, NOOP uses elevator=noop, and Anticipatory uses elevator=as. With 2.6 kernels, it is also possible to change the scheduler for particular devices during runtime by echoing the name of the scheduler into /sys/block/devicename/queue/scheduler. For example, in the following listing we change the scheduler for device sdc to deadline while noting that device sdb remains at the default:

root@london1 ˜]# cat /sys/block/sdc/queue/scheduler
noop anticipatory deadline [cfq]
[root@london1 ˜]# echo deadline > /sys/block/sdc/queue/scheduler
[root@london1 ˜]# cat /sys/block/sdc/queue/scheduler
noop anticipatory [deadline] cfq
[root@london1 ˜]# cat /sys/block/sdb/queue/scheduler
noop anticipatory deadline [cfq]

It is important to note that such dynamic scheduler changes are not persistent over reboots.

CFQ is the default scheduler for Oracle Enterprise Linux releases, and it balances I/O resources across all available resources. The Deadline scheduler attempts to minimize the latency of I/O requests with a round robin–based algorithm for real-time performance. The Deadline scheduler is often considered more applicable in a data warehouse environment, where the I/O profile is biased toward sequential reads. The NOOP scheduler minimizes host CPU utilization by implementing a FIFO queue, and it can be used where I/O performance is optimized at the block-device level. Finally, the Anticipatory scheduler is used for aggregating I/O requests where the external storage is known to be slow, but at the cost of latency for individual I/O requests.

Block devices are also tunable with the parameters in /sys/block/devicename/queue:

[root@london1 queue]# ls /sys/block/sdc/queue
iosched            max_sectors_kb  read_ahead_kb
max_hw_sectors_kb  nr_requests     scheduler

Some of the tunable parameters include the nr_requests parameter in /proc/sys/scsi, which is set according to the bandwidth capacity of your storage; and read_ahead_kb, which specifies the data pre-fetching for sequential read access, both of which have default values of 128.

CFQ is usually expected to deliver the optimal level of performance to the widest number of configurations, and we recommend using the CFQ scheduler, unless evidence from your own benchmarking shows another scheduler delivers better I/O performance in your environment.

With the CFQ scheduler, I/O throughput can be fine-tuned for a particular process using the ionice utility to assign a process to a particular scheduler class. The process can range from idle at the lowest to real-time at the highest, with best-effort in between. For example to change the scheduling process of the LGWR process, you need to begin by identifying the process id for the process:

[root@london1 ˜]# ps -ef | grep lgwr | grep -v grep
oracle   13014     1  0 00:21 ?        00:00:00 asm_lgwr_+ASM
oracle   13218     1  0 00:22 ?        00:00:00 ora_lgwr_PROD1

In this case, querying the process shows that no scheduling priority has been set:

[root@london1 ˜]# ionice -p 13218
none: prio 0

The scheduling classes are 1, 2, and 3 for real-time, best-effort, and idle, respectively. For real-time and best-effort, there are eight priority levels, ranging from 0-7, that determine the time allocated when scheduled. The following example shows how to set the LGWR process to a real-time class with a priority level of 4:

[root@london1 ˜]# ionice -c1 -n4 -p 13218
[root@london1 ˜]# ionice -p 13218
realtime: prio 4

In any case, we recommend testing your RAC storage configuration and chosen I/O scheduler. If, after testing, you conclude that you would benefit from using a non-default I/O scheduler, then you should ensure that you use the same scheduler on each and every node in the cluster.

So far we have covered the realization of block-based storage for Oracle RAC, as well as the technology underlying the implementation of SANs. In contrast to SAN, there is also NAS, which takes another approach in providing storage for RAC.

Although the approach is different, many storage vendors support both a SAN and/or NAS implementation from the same hardware. By definition, iSCSI is both SAN and NAS. To draw the distinction, we will use NAS to define storage presented over a network, but provided as a file-based system, as opposed to a block-based one. For Linux and RAC, the only file system that you need to be concerned with is the Network File System (NFS) developed by Sun Microsystems. Oracle supports RAC against solutions from EMC, Fujitsu, HP, IBM, NetApp, Pillar Data Systems, and Oracle Sun StorageTek. If you're using NFS with Oracle 11g, the Direct NFS client was introduced to implement NFS connectivity directly into the Oracle software itself, without requiring the use of the Linux NFS client.

As we have seen with block-level storage, Oracle RAC can either use the block devices directly or with ASM; or, for greater manageability, it can layer a file system such as OCFS2 on top of the block devices at the operating system level. The challenge for such a clustered-file system is to manage the synchronization of access between the nodes to prevent corruption of the underlying storage when a node is ejected from the cluster. Therefore, such an approach requires a degree of communication between the nodes. With a NAS approach, however, the file system itself is presented by the storage device and mounted by the RAC nodes across the network. This means that file-system synchronization is reconciled at the storage level. NAS storage also enables further file system features such as journaling, file-based snapshots, and volume management implemented and managed at the storage level for greater efficiency.

At the hardware level, NAS is realized by a file server, and the RAC nodes communicate with the file server through standard Ethernet connectivity. We want to stress that, For a RAC cluster, this storage network should be an absolutely private, dedicated network with strictly no other non-storage-related network traffic. The RAC cluster should not be connected simply by the local LAN, with NAS configuration requiring the same respect accorded to FC. Because file servers were originally designed for file sharing, their functionality is compatible with the RAC shared-all database approach. Also, because file servers are designed for this one function only, they tend to be highly optimized and efficient compared to a generic operating system in this role. A RAC cluster should not be based on an NFS file system served by anything but specialized hardware designed for that purpose and supported under the RAC Technologies Compatibility Matrix (RTCM) for Linux Clusters.

At the connectivity level, as discussed for iSCSI, generally available Ethernet technology currently operates at gigabit speeds below that of the commonly available bandwidth for FC, although increasing adoption of 10GbE provides a more level playing field in this respect. A commonly quoted restriction is that the data is transferred from the nodes through TCP/IP, which is implemented by the server CPU, and the communication between the CPU and the NIC results in a higher number of CPU interrupts. The use of the 11g Direct NFS client and server-based network acceleration technologies such as TOEs can negate some of this impact, but the additional overhead will impact the level of performance available to some degree.

As you have seen, there are a considerable number of factors that affect storage performance in an Oracle RAC environment, from the processor and memory to process the I/O requests down through the system, to the disk drives themselves. As such, evaluating the potential storage performance of a system from an entirely theoretical standpoint can only be accurate to a certain degree; therefore, the optimal way to understand the nuances of each approach is to test and compare the solution applicable to your environment.

To draw comparisons between various I/O configurations, we recommend testing the configurations in question. Ultimately, the best way of determining a storage configuration capable of supporting an Oracle RAC configuration is to measure that storage under an Oracle Database workload. However, while noting the relevance of measuring actual Oracle Database workloads, there is also a tool called Oracle Orion available with both the Grid infrastructure software and Oracle Database software installations in Oracle 11g Release 2. This tool enables the testing of storage performance in isolation.

If you're running against block devices, Orion should be run against the block devices themselves and not a file in the file system mounted upon them. Also, the user running the software should have permission to read and write to those devices, none of which should not be in use for any software or contain data that requires preserving. Set the names of your disks in a file with the extension .lun, as in this example:

[root@london1 bin]# cat prod.lun
/dev/sdd

When running Orion, you should ensure that the LD_LIBRARY_PATH environment variable is set to include either $ORACLE_HOME/lib or the equivalent directory in the Grid Infrastructure software home. You can find details for how to do this in Chapter 6. This snippet shows how to run the Orion tool with the -help argument:

[oracle@london1 bin]$ ./orion -help
ORION: ORacle IO Numbers -- Version 11.2.0.1.0

ORION runs IO performance tests that model Oracle RDBMS IO workloads.
It measures the performance of small (2-32K) IOs and large (128K+) IOs
at various load levels.
...

There are a number of options for configuring the type of I/O workload; however, the initial preliminary set of data can be collected, as follows:

[root@london1 bin]# ./orion -run simple -testname prod
ORION: ORacle IO Numbers -- Version 11.2.0.1.0
prod_20091113_1511
Calibration will take approximately 9 minutes.
Using a large value for -cache_size may take longer.

Within the working directory, a number of data files are produced, including a summary of performance data that corresponds to the storage-related values discussed previously in this chapter, including megabytes per second throughput, IOPS, and latency:

Maximum Large MBPS=91.09 @ Small=0 and Large=1
Maximum Small IOPS=1454 @ Small=5 and Large=0
Minimum Small Latency=2906.70 usecs @ Small=1 and Large=0

Other workloads in addition to simple include normal, oltp, dss with oltp reporting, IOPS, and latency and dss focusing on megabytes per second. Also, some advanced configurations give more opportunity to fine-tune the I/O workload, such as specifying different block sizes from the default.

I/O Calibration is also available within the Oracle 11g database in its DBMS_RESOURCE_MANAGER package. This procedure implements a read-only workload and is run as follows by specifying the input parameters of the number of physical disks on which the Oracle Database is installed, as well as the maximum tolerated latency in milliseconds:

SQL> set serveroutput on
declare
  max_iops              integer;
  max_mbps              integer;
  actual_latency        integer;
begin
  dbms_resource_manager.calibrate_io (
    num_physical_disks => 15,
    max_latency        => 10,
    max_iops           => max_iops,
    max_mbps           => max_mbps,
    actual_latency     => actual_latency);
  dbms_output.put_line ('IOPS = ' || max_iops);
  dbms_output.put_line ('MBPS = ' || max_mbps);
  dbms_output.put_line ('Latency = ' || actual_latency);
end;
/

The procedure may take up to 10 minutes to complete, and the status of the calibration process can be viewed as follows:

SQL> select inst_id, status from gv$io_calibration_status;

   INST_ID STATUS
---------- -------------
         1 IN PROGRESS
         2 IN PROGRESS

Note that the preceding runs on one instance in the cluster only, although Oracle recommends that all instances in the cluster remain open to calibrate the read workload across the entire cluster. Finally, the output parameters are reported by the procedure, as in this example:

IOPS = 2130
MBPS = 186
Latency = 9

PL/SQL procedure successfully completed.

This output can also be viewed subsequently in the view DBA_RSRC_IO_CALIBRATE and compared against the start and end of time of individual test runs.