Anaconda

Your system will then start Anaconda, the graphical configuration assistant to guide you through the rest of the installation.

The next steps are almost self-explanatory. After acknowledging the welcome screen by clicking next you have the option to select the language to be used during the installation. Alternative languages can always be installed for use with the system later. Once you are happy with the settings click on the “Next” button to select your keyboard layout to be used during the installation. The keyboard setting-just like the language setting made earlier-can be changed after the system has been installed. From a support point of view it is sensible to limit the installation to English. Mixing multiple languages in different regions makes a global support policy difficult. Translating error messages and troubleshooting can become a problem for non-native speakers.

Choice of storage devices

So far the installation process was very similar to the one used in Oracle Linux 5. The next screen though, titled “Storage devices” is new to Oracle Linux 6. It offers two choices:

1. To use “basic storage devices”
2. Alternatively use “specialized storage devices”

Specialized storage in this context allows you to install the operating system to a SAN disk or over Fibre Channel over Ethernet or iSCSI. It also (finally!) offers support for installation on hardware RAID and dm-multipathed devices. For all other needs, simply go with the first option.

In the following sections it is assumed that you are installing the operating system to internally attached disks. Most servers will use an internal RAID adapter to present a single view on internal disks. Consult your vendor’s manual for more information on how to configure these. In many cases, the array driver is already part of the Linux distribution and you don’t need to worry about it once the BIOS setup is completed. If not, then fear not-you have an option to add vendor drivers during the installation process. Whichever way you decide to install the operating system, it is important to have a level of resilience built in. Resilience can come in the form of multiple disks in a RAID 1 configuration, or in the form of multiple paths to the storage to prevent a single point of failure.

Following along the path set out by the example, select the “basic storage devices” click on the “Next” button. You may be prompted with a “storage device warning” before the next screen appears. Oracle Linux tries to read an existing partition table for all detected storage devices made available to the host. The warning is raised whenever no partition table can be found, or alternatively if the detected partition table cannot be read and understood.

If you are certain that the disk for which the warning appears is blank, unpartitioned, and not in use you should click on “Yes, discard my data.” Be careful not to check “Apply my choice to all devices with undetected partitions or filesystems!” You can exclude a disk from the further partitioning process by clicking on “No, keep my data” for that disk.

For cluster installations in particular you need to be careful at this stage when adding nodes to the cluster. If your system administrator has made the ASM LUNs available to the host you are about to install they must not be touched by the installer, otherwise you lose data! To prevent human errors, it is usually safer to not present the database LUNs to the host at installation time. Linux is very flexible and SAN storage can be made available to the host without rebooting. Be aware that the storage warning dialog will be repeated for each disk with an unknown partition table or file system.

Surprisingly the next screen allows you to configure the network. The storage configuration will be performed later.

Network configuration

You would have expected to be able to review the storage layout now but first you need to define the hostname of your new server. If you are using DHCP, supply the short hostname only, otherwise provide a fully qualified domain name (FQDN). The fully qualified hostname has been used in Figure 5-2. New to Oracle Linux 6 is the ability to define the network properties at this stage. Click on the “Configure Network” button in the lower-left corner of the screen to access the network configuration settings. The following dialog allows you to configure wired, wireless, mobile broadband, VPN, and DSL connections. Most enterprise systems will need to configure wired connections at this stage. Highlight your current network card by clicking on it, then chose “edit” from the right-hand menu. The resulting configuration options are shown in Figure 5-2.

This network configuration utility uses Network Manager under the covers. Network Manager is the replacement for the network administration tool (“system-config-network”) and one of the bigger changes in Oracle Linux 6 over its previous versions. This does not imply that the way network settings were configured in Oracle Linux go away, but you should be aware of the new way of doing things. The system settings for the selected device can be chosen in the tabs “wired,” “802.x Security,” “IPv4 Settings,” and “IPv6 Settings.” You have the option to make the following changes on these tabs:

Time zone settings and root password

Following the network configuration you need to set the system’s time zone. This setting is important as it allows determining the location of the server. If you are planning on clustering the Oracle system yet to be installed then please ensure that all the cluster nodes share the same time zone.

To pick a time zone you can either use the zoom controls to click on the city depicted by a yellow dot on the map nearest to your location to set the time zone. Another option is to use the dropdown list to find the most appropriate time zone.

Before finally making it to the partitioning wizard, you need to set a secure root password to be used with this server. As always, only system administrators should have access to the root account. Click “Next” to proceed with the installation.

Partitioning

An interesting detail of the Oracle Linux installation is the somewhat belated option to choose a partitioning layout. Again this section assumes you have chosen “basic storage devices.” The initial screen offers you a number of options to install Oracle Linux exclusively or in conjunction with other operating systems.

Oracle database servers are not normally configured for dual-booting with non-Linux systems. A planned upgrade of Oracle Linux 5 (or another Linux distribution) to Oracle Linux 6 however is a valid reason for a dual-boot configuration. Before beginning the partitioning of your server be sure to create backups of any data you want to preserve. Please bear in mind that mistakes do happen! After the changes you are making on the following screen are written to disk, previously existing partitions may irrevocably be lost!

Whichever option you select from the list of available options, it is recommended to enable the check box labeled “Review and modify partitioning layout” which takes you to the screen shown in Figure 5-3 to fine-tune the setup for a single-boot configuration with Oracle Linux 6.4 as the only installed operating system.

The partitioning screen has a great many options, and a complete description of all possible configuration combinations cannot be provided in this chapter. It is best to align the storage configuration with your current standards. That way you can’t really make a mistake. For all the other detail please refer to the online manual. To keep things simple and to prevent mistakes from happening it is often recommended not to present storage to the server, which is not immediately required to install the base operating system at this stage. The commands necessary to create storage mount points for the Oracle accounts will be described in a later section in this chapter.

The partitioning screen resembles the layout used in Oracle Linux 5, but the graphical representation in the top of how full an individual disk is will only appear after clicking on it. The user interface has been uncluttered in comparison with the previous version, but the functionality has remained the same. The “Create” button offers you to create the following entities, disk space permitting:

• Create a standard partition
• Create a (software) RAID partition
• Create a (software) RAID device
• Create a LVM physical volume
• Create a LVM volume group
• Create a LVM logical volume

The “Edit” button allows you to make changes to the highlighted partition, the “Delete” button unsurprisingly removes the selected entity. If you got stuck during the partitioning of your system and want to get back to square one, click on the “Reset” button to undo your changes to the original layout.

Oracle Linux has a wider set of options of file systems available to format your disks. Apart from ext3, which was the standard in Oracle Linux 5.x ext2, ext4, xfs, and Btrfs are available.

A possible partitioning scheme in Oracle Linux includes the partitions listed in Table 5-1.

It is strongly recommended not to install the operating system on a single hard disk. Whenever possible make use of a hardware RAID adapter inside the server. Otherwise the failure of a single disk will result in an outage of the server which can easily be prevented. If your server does not use a built-in RAID adapter for the internal disks you have the option to create a software RAID configuration. An example for a software-RAID is shown in Figure 5-4 below. Matching sized partitions are created on the internal disks and mirrored to form RAID 1 pairs. Of course, you have other software RAID levels as well but RAID 1 seems most practical.

As you can see the partition layout is symmetrical between the disks: three partitions are created on each device with a partition type of “software RAID” for this small Oracle SE database server. One gigabyte swap in /dev/sda1 and /dev/sdb1 each, 200 MiB boot partition on /dev/sda2 and /dev/sdb2 as well as a root partition spanning the rest of disks /dev/sda and /dev/sdb. In the next step RAID 1 pairs are created, resulting in RAID devices. The boot loader can be installed in the /boot partition. The resulting disk layout is shown here:

[root@server1 ∼]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              6.7G  787M  5.6G  13% /
tmpfs                 497M     0  497M   0% /dev/shm
/dev/md1              194M   39M  146M  21% /boot

An alternative approach would be to create two RAID 1 devices /dev/md0 and /dev/md1. The first one has to be a boot partition since the boot loader cannot reside in an LVM logical volume. The other RAID device can be converted into a physical device for use with LVM.

The Linux Logical Volume Manager (LVM) is an excellent choice to add flexibility to your disk layout. LVM is based on physical volumes, or LUNs. For example, a software RAID device can also be a physical volume. Multiple physical volumes can be aggregated into a volume group, which provides the cumulative amount of free space of all underlying physical volumes. You then carve space out of the volume group, creating logical volumes. On top of a logical volume you create the file system and the mount point. You can be guided through all of these steps in the installer, accessible by the “Create” button.

Now why would you use this complex sounding LVM at all? Because it gives you the ability to grow and even shrink your file systems! If you are using partitions it is very difficult, if not impossible, to resize a file system. With LVM all you need to do is extend the logical volume and resize the file system on top. This can be a life saver if you need to upgrade Oracle and OUI complains about a lack of space.

Boot loader configuration

Oracle Linux continues to use the GRand Unified Bootloader version 0.9x unlike many consumer distributions, which have switched to GRUB 2. The installation screen does not require too much attention if you are not planning on dual-booting the Oracle database server. As pointed out earlier, the main reason to dual-boot a production database server is when you want to perform an operating system upgrade.

If another Oracle Linux distribution is found and correctly detected it will be added to the boot menu.

Software installation

The Oracle Linux software selection screen is not the most user-friendly interface and occasionally can be difficult to use. The software selection screen is shown in Figure 5-5.

For now it is possibly the easiest to select “basic server”, which installs a slim variant of the operating system for use on a server, but without X11. Additional packages can always be added later. Resist the temptation to select “Database Server”-instead of the packages required by Oracle, this will install MySQL and PostgreSQL instead. What you should add however is the support for XFS, even if you are not planning on using it right now. To add XFS support, add an additional repository for the “Scalable Fileystem Support” which adds user land utilities to create and manage XFS file systems.

The package selection made in the screen shown in Figure 5-5 does not allow you to install the Oracle database immediately. Additional packages are necessary. In fact it is easier to install the missing packages for an Oracle installation later from the command line. Upon clicking the “Next” button one more time the installation process starts by copying the selected packages to the server. Notice that Oracle now boots off the Unbreakable Enterprise Kernel version 2 by default!

It is now time for a coffee, this process will take a few minutes to complete. After all packages have been transferred, the Anaconda session congratulates you on your new Oracle Linux installation. After clicking on “Reboot” the server will restart with the settings you created in the installation process. Please ensure the DVD is no longer in the drive or virtually mounted to the host.

Preparing for PXE booting

Booting into the Preboot Execution Environment and then using Kickstart to install the Linux system to match the Unix and Oracle standards is the ultimate form of database server build sophistication. The whole task can be a little bit tricky if a large number of different hardware platforms are used. This is why I suggested standardizing hardware offerings into three different classes matching processing needs in Chapter 1. This allows the engineering team to develop a stable set of installation images.

Before it is possible to spawn new servers or virtual machines using PXE-boot and network installations a little setup work is needed:

1. The Oracle Linux installation files must be made available over the network.
2. A TFTP (“trivial FTP”) server must be available to provide a boot environment.
3. A DHCP (“Dynamic Host Configuration Protocol”) server must be configured.

Normally you would combine the above points on one server. If provisioning of new Oracle Linux instances-physical or virtual-is a critical task, consider setting up multiple installation servers in your network segment. RSYNC or similar utilities could be used to keep the installation servers in sync.

Making the installation tree available

After the TFTP server is installed and ready, it’s time to make the installation sources available. The first task is to provide the installation source. First, you need to decide which way you would like to do this. You have a choice of providing the sources via the FTP, HTTP, NFS, and other protocols. In this example HTTP is used. To export the install source, you need to install a webserver such as apache2 using the yum utility. The example assumes that the yum repository has been configured according to your company’s standard. The installation of the webserver is shown here:

# yum install httpd
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.2.15-15.0.1.el6 will be installed
[...]
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=============================================================================
 Package              Arch          Version                  Repository Size
=============================================================================
Installing:
 httpd                x86_64        2.2.15-15.0.1.el6        cd        808 k
Installing for dependencies:
 apr                  x86_64        1.3.9-3.el6_1.2          cd        123 k
 apr-util             x86_64        1.3.9-3.el6_0.1          cd         87 k
 apr-util-ldap        x86_64        1.3.9-3.el6_0.1          cd         15 k
 httpd-tools          x86_64        2.2.15-15.0.1.el6        cd         69 k
 
Transaction Summary
=============================================================================
Install       5 Package(s)
 
Total download size: 1.1 M
Installed size: 3.5 M
Is this ok [y/N]: y
Downloading Packages:
-----------------------------------------------------------------------------
Total                                        8.1 MB/s | 1.1 MB         00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : apr-1.3.9-3.el6_1.2.x86_64                                1/5
  Installing : apr-util-1.3.9-3.el6_0.1.x86_64                           2/5
  Installing : httpd-tools-2.2.15-15.0.1.el6.x86_64                      3/5
  Installing : apr-util-ldap-1.3.9-3.el6_0.1.x86_64                      4/5
  Installing : httpd-2.2.15-15.0.1.el6.x86_64                            5/5
 
Installed:
  httpd.x86_64 0:2.2.15-15.0.1.el6
 
Dependency Installed:
  apr.x86_64 0:1.3.9-3.el6_1.2
  apr-util.x86_64 0:1.3.9-3.el6_0.1
  apr-util-ldap.x86_64 0:1.3.9-3.el6_0.1
  httpd-tools.x86_64 0:2.2.15-15.0.1.el6
 
Complete!

You should also enable apache to start at boot time using the familiar chkconfig command:

# chkconfig --level 345 httpd on

Do not start apache at this stage-a little more setup work is needed. Begin by mounting the Oracle Linux ISO image to your preferred location; this example assumes /media/ol64. The steps necessary to make the software available to apache are shown here (again assuming Oracle Linux 6.4):

# mkdir /media/ol64
# mount –o loop /m/downloads/linux/V37084-01.iso /media/ol64

Alternatively you could of course copy the contents of the DVD to a directory of your choice. Next apache must be told where to look for the files. Create a file ol64.conf in /etc/httpd/conf.d with the following contents:

Alias /ol64/ "/media/ol64/"
 
<Directory "/media/ol64">
    Options Indexes MultiViews FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from 192.168.100.0/255.255.255.0
</Directory>

This creates an alias for use with apache, and critically allows the installer to follow symbolic links which is important during the installation. The configuration item also restricts access to the installation tree to the build subnet 192.168.100.0/24.

With all this done, start apache using “service httpd start”. When referring to the Oracle Linux install source in a web browser ensure to end the URL with a slash, i.e., http://installServer/ol64/. Otherwise apache will complain that the directory does not exist. Firewalls are another potential source of problems. If you are using firewalls you need to permit access to port 80 on the webserver to access the previously copied files.

Setting up the TFTP server

The TFTP server will contain the initial installation image for Oracle Linux 6. To stay in line with the rest of the book I assumed that your installation server is an Oracle Linux 6 as well. Connect to your installation server and install the tftp-server.x86_64 package using yum as shown here:

# yum install tftp-server.x86_64
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package tftp-server.x86_64 0:0.49-7.el6 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=============================================================================
 Package            Arch          Version           Repository          Size
=============================================================================
Installing:
 tftp-server        x86_64        0.49-7.el6        ol6_u2_base         39 k
 
Transaction Summary
=============================================================================
Install       1 Package(s)
 
Total download size: 39 k
Installed size: 57 k
Is this ok [y/N]: y
Downloading Packages:
tftp-server-0.49-7.el6.x86_64.rpm                     |  39 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing : tftp-server-0.49-7.el6.x86_64                             1/1
 
Installed:
  tftp-server.x86_64 0:0.49-7.el6
 
Complete!

TFTP is not the most secure protocol, and therefore should be adequately protected. It is controlled via xinetd and deactivated by default. To enable and start it, execute the following commands:

# chkconfig --level 345 xinetd on
# chkconfig --level 345 tftp on
# service xinetd start
Starting xinetd:                                           [  OK  ]

The TFTP server supplied with Oracle Linux 6 uses the directory /var/lib/tftpboot to export files. To separate different install images it is a good idea to create a subdirectory per installation source. In the example I am using /var/lib/tftpboot/ol64/. The boot procedure laid out in the following example is based on PXELINUX, a network boot loader similar to SYSLINUX. You may know the latter from installing end-user Linux installations.

Install the syslinux package using YUM. Once the syslinux package has been installed, a few files need to be copied into /var/lib/tftpboot, as shown below:

# cp -iv /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/ol64/
`/usr/share/syslinux/pxelinux.0' -> `/var/lib/tftpboot/ol64/pxelinux.0'
# cp –iv /usr/share/syslinux/menu.c32 /var/lib/tftpboot/ol64/
`/usr/share/syslinux/menu.c32' -> `/var/lib/tftpboot/ol64/menu.c32'

Now the distribution specific files need to be copied from the installation source.

# cp –iv /media/ol64/images/pxeboot/* /var/lib/tftpboot/ol64/
`/media/ol64/images/pxeboot/initrd.img' -> `./initrd.img'
`/media/ol64/images/pxeboot/TRANS.TBL' -> `./TRANS.TBL'
`/media/ol64/images/pxeboot/vmlinuz' -> `./vmlinuz'
#

Nearly there! A boot menu is now required to allow the user to boot (automatically) into the operating system. The PXELINUX system requires a directory called “pxelinux.cfg” to be present, from where it can access boot configuration. The boot menu is created using the below configuration:

# mkdir -p /var/lib/tftpboot/ol64/pxelinux.cfg
 
# cat /var/lib/tftpboot/ol64/pxelinux.cfg/default
timeout 100
default menu.c32
 
menu title ------- Boot Menu -------
label 1
menu label ^ 1) ORACLE LINUX 6 KICKSTART
kernel vmlinuz
append initrd=initrd.img ks=http://imageServer/ks/ol64.cfgksdevice=link
 
label 2
menu label ^ 2) ORACLE LINUX 6 INTERACTIVE INSTALL
kernel vmlinuz
append initrd=initrd.img

The menu defines two items: the first automatically boots after 10 seconds (the timeout is defined in tenths of a second in the configuration file) of inactivity and begins the silent installation of Oracle Linux 6. It achieves this by pointing to the Kickstart file on the webserver just configured and specifying that the first device with an active link should be the Kickstart device. Using ksdevice=link nicely circumvents the problem that the first network card does not necessarily have eth0 assigned to it.

This concludes the TFTP server configuration. Unless you use firewalls again, in which case you need to allow access to port 69, protocol UDP! However it is not yet possible to start the installation-additional information must be passed on to the booting client via DHCP.

Configuring the Dynamic Host Configuration Protocol server

The final step in preparing your build server is to install and configure the DHCP server. A package is available to take care of this task for us, called dhcpd. Install it the usual way using yum:

# yum install dhcp
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package dhcp.x86_64 12:4.1.1-25.P1.el6 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=============================================================================
 Package       Arch            Version                     Repository   Size
=============================================================================
Installing:
 dhcp          x86_64          12:4.1.1-25.P1.el6          cd          815 k
 
Transaction Summary
=============================================================================
Install       1 Package(s)
 
Total download size: 815 k
Installed size: 1.9 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : 12:dhcp-4.1.1-25.P1.el6.x86_64                            1/1
 
Installed:
  dhcp.x86_64 12:4.1.1-25.P1.el6
 
Complete!

This version of DHCP is very recent, which is really good for PXE boot environments. DHCP is the magic that sends the configuration information to our new server over the network. Before it can do so however the configuration file /etc/dhcp/dhcpd.conf needs to be amended. The below is an example minimum configuration file needed for PXE clients:

allow booting;
allow bootp;
# other global options, such as domain-name, domain-name-servers, routers etc
 
# build network
subnet 192.168.100.0 netmask 255.255.255.0 {
  range 192.168.100.50    192.168.100.100;
  # further options...
 
  # specific hosts
  host pxeboot {
    hardware ethernet 08:00:27:62:66:EE;
    fixed-address 192.168.100.51;
  }
   
}
 
# grouping all PXE clients into a class
class "pxeclients" {
  match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
  next-server 192.168.100.2;
  filename "ol64/pxelinux.0";
}

The first two lines instruct DHCP to allow clients to PXE boot, which is the initial requirement followed by other generic configuration options. Next a subnet is defined, which would normally map to the “build” network. The build network is normally out of limits for normal users and specifically designed for software installations such as the one described.

All PXE clients are grouped into their own class “pxeclients.” The vendor-class-identifier is an ISC DHCP version 3 directive that lets us identify clients willing to boot via PXE. The “next-server” directive has to point to the tftp-server. The filename directive finally instructs the clients to look for the file ol64/pxelinux.0 which was copied during the TFTP configuration and set up earlier.

Before the first server can be started, a Kickstart file is required. Remember to add firewall rules for your DHCP server if needed.

Considerations for the Kickstart file

After all the preparations it is time to define how the system is to be installed. Any reputable Linux distribution comes with a lights-out installation mechanism, and Oracle Linux is no exception to that rule. Just like Red Hat Linux, Oracle Linux uses the Kickstart mechanism to perform automated installations. The Kickstart format can be quite daunting at first, and it occupies a lot of space in the installation manual.

It is not as difficult as one might think, mainly because of the following factors:

• Every time you manually install Oracle Linux the installer (called “Anaconda”) creates a Kickstart file with the options you chose in /root/anaconda-ks.cfg.
• A GUI tool, system-config-kickstart is available which guides you through the creation of a Kickstart file.

To automate the installation of Oracle Linux it is usually easiest to review the Anaconda-generated Kickstart file and make subtle changes as necessary. Consider the following Kickstart file which you could use as the basis for your own installation. It has been created as a result of the manual installation shown earlier in the chapter.

install
url --urlhttp://192.168.100.2/ol64/
lang en_US.UTF-8
keyboard uk
network --device eth0 --bootproto dhcp --noipv6
rootpw –iscrypted a...o
firewall --disabled
selinux --permissive
authconfig --enableshadow --passalgo=sha512
timezone --utc America/New_York
bootloader --location=mbr --driveorder=sda --append=" rhgb crashkernel=auto quiet"
# The following is the partition information you requested
# Note that any partitions you deleted are not expressed
# here so unless you clear all partitions first, this is
# not guaranteed to work
ignoredisk --only-use=sda
clearpart --initlabel --drives=sda
part /boot --fstype=ext2 --size=200 --ondisk=sda
part pv.008002 --grow --size=200 --ondisk=sda
volgroup rootvg --pesize=4096 pv.008002
logvol swap --name=swaplv --vgname=rootvg --size=2048
logvol / --fstype=ext4 --name=rootlv --vgname=rootvg --size=5940 --grow
 
%packages
@base
@client-mgmt-tools
@console-internet
@core
@debugging
@directory-client
@hardware-monitoring
@java-platform
@large-systems
@network-file-system-client
@performance
@perl-runtime
@server-platform
@server-policy
@system-admin-tools
pax
python-dmidecode
oddjob
sgpio
certmonger
screen
strace
pam_krb5
krb5-workstation
%end

In the above example the Anaconda installer is going to install (instead of upgrade) the software using the HTTP protocol pulling the files from our installation server. Language and keyboard are defined next before the network definition is set to use DHCP for the first Ethernet device. Additional networks can be defined in their own “network --device ethx” section.

The root password used is encrypted; the value is taken from the anaconda-ks.cfg file created during the manual installation. The clear text password has been entered during the initial manual installation performed earlier.

Specifying an encrypted password is much safer than a clear text password!

The following directives disable the firewall and set the security-enhanced Linux settings to permissive. Be sure to update this section after consulting your IT-security department! The authconfig command enables the use of shadow passwords which implies local user accounts on the machine. Furthermore the shadow passwords should use SHA512 as the password algorithm.

The timezone is then set to America/New_York-this should most definitely be adopted to match the closest location where the server is located. The next few lines are dealing with the boot loader location and the partitioning scheme. The chosen partitioning options can be translated into English as follows:

• clearpart: Clear all partitions and initialize the disk label
• part /boot: Create a boot partition of 200 MiB using ext4
• part pv008002: Create a physical volume and use all remaining space
• volgroup rootvg: Create a volume group “rootvg” using the physical volume just created
• logvol (twice): Create two logical volumes on volume group rootvg.

The kickstart file explicitly uses disk sda only. The remainder of the file specifies which YUM packages and groups should be installed. If you wonder which groups are available, you can use YUM on a system to execute the following query:

# yum grouplist

Although it looks intimidating at first-especially the partitioning part-the Kickstart process is logical and easy to master. The Kickstart format is also very well documented in the installation guide. Please refer to it for further information about available options. Once the Kickstart file is ready, move it to a directory on the webserver from where it is accessible to the build network. For security reasons it should not be exposed outside of the build network. To follow the example you need to place it into /var/www/html/ks/.

A final word of warning: please be careful with the file. It absolutely erases everything! As with anything in this book, please adapt the file to your needs and test it thoroughly before deploying it on real systems.

Testing the automated installation

With all the preparation work completed, it is time to put the procedure to test. Connect a machine to the build network and enable PXE booting on the hardware level. Most often you would connect to the lights-out management console before starting the server. After a little while, when the subsystems have initialized, the network card should advertise that it is trying to PXE boot. This triggers a DHCP request on your install server, which should be acknowledged. If you like (still on the install server) start tailing the /var/log/messages file to see what happens. If you are really curious you could start tcpdump on your install server to listen on port 69 to see if the tftp request can be satisfied.

If everything goes well then you will see a (very basic) boot menu which will automatically boot the default operating system. If you are curious you will see the whole installation happening on the lights-out management console. You should do this at least once to ensure that there are no problems with the process. Also bear in mind that you need to be careful where you install the operating system: there is no safety net: all the data on the machine is erased, and you get a fresh Oracle Linux server instead. If you used DDNS you should then have a server, which is correctly configured within DNS and ready for the next steps, the preparation of the OS to install Oracle Database 12.1.

Installing additional packages

The first step in the preparation for the Oracle installation involves completing the installation of the required set of packages. A local YUM repository such as the one created earlier can be used to download the packages and resolve any dependencies. To make the repository available to the system create a new file local.repo in /etc/yum.repos.d/ with these lines:

[local]
name = local installation tree
baseurl =http://imageServer/ol64/
enabled = 1
gpgcheck = 1

Note that the gpgcheck value is set to 1 in the repository configuration. Whenever you are using repositories you must ensure that the packages you download from the repository are signed and match the key! Using the above repository you should be able to install the packages required for the next steps.

# yum install compat-libcap1 compat-libstdc++-33 libstdc++-devel gcc-c++ ksh libaio-devel
# yum install xorg-x11-utils xorg-x11-server-utils twm tigervnc-server xterm

The preceding list of commands depends on the packages already present on your system. For a complete list of what is needed, please refer to the Oracle Database Quick Installation Guide 12c Release 1 for Linux x86-64. See specifically section 6.1, “Supported Oracle Linux 6 and Red Hat Enterprise Linux 6 Distributions for x86-64”, for a complete list.

If you are planning the installation of Oracle Restart, you also need to install the package “cvuqdisk,” which is part of the Grid Infrastructure installation package. Installing these packages satisfies the Oracle installer.

Creating the operating system users and groups

With all required packages in place it is time to consider the creation of the Oracle user and groups. Whereas this was a simple and straightforward task in releases up to Oracle 11.1, some more thought must now be put into the process if you plan on using ASM or Real Application Clusters. The new process in Oracle 12c even affects the database installation.

The main reason the user and group creation process has become more interesting is the “separation of duties”. In the very basic form, one operating system account-usually named “oracle”-owns the binaries for Grid Infrastructure and the database as well. This way whoever logs into the system as oracle has full control over every aspect of database management. While this is acceptable in most smaller companies, larger institutions use different teams for the management of the Oracle stack. In versions leading up to 12.1, the separation in broad terms was between the storage team and DBA team: if so desired, separate accounts were used to install and own Grid Infrastructure and the RDBMS binaries. In addition to the storage aspect, Oracle 12.1 introduced new responsibilities: backup, Data Guard, and encryption key management.

Similar to previous releases, the responsibilities are implemented using internal groups such as OSDBA to which operating system groups are mapped. Operating system accounts can then further be mapped to the groups, inheriting the privileges associated with the role. The mapping between Oracle groups and operating system groups can be found in Table 5-2.

Without a policy of separation of duties in place you could map the oracle user to all the above-mentioned groups. In a scenario where storage and database management are separated you could map the ASM-related groups to the grid user, and the rest to oracle. The oracle account also needs the OSDBA for ASM privilege to connect to the ASM instance; without it oracle can’t access its storage. Even if you are not planning on using multiple operating system accounts I still recommend creating the operating system groups. This is simply to give you greater flexibility later on, should you decide to allow accounts other than oracle and grid to perform administration tasks with the database.

Up until now, one other very important group has not been mentioned: oinstall. This group owns the Oracle inventory and is required for each account that needs to modify the binaries on the system. Oracle recommends that every Oracle-related operating system account should have oinstall as its primary group.

# groupadd –g 4200 oinstall
# groupadd –g 4201 dba

# groupadd –g 4202 backupdba
# groupadd –g 4203 dgdba
# groupadd –g 4204 kmdba
# groupadd –g 4205 asmdba
# groupadd –g 4206 asmadmin

# useradd –u 4200 –g oinstall –G dba,asmdba –m oracle
# passwd oracle
Changing password for user oracle
New password:
Retype new password:
passwd: all authentication tokens updated successfully

# usermod –G dba,backupdba,dgdba,kmdba,asmdba,asmadmin oracle

# id –a oracle
uid=4200(oracle) gid=4200(oinstall) groups=4200(oinstall),4201(dba)

# useradd –u 4200 –g oinstall –G asmdba,dba –m oracle
# passwd oracle
Changing password for user oracle
New password:
Retype new password:
passwd: all authentication tokens updated successfully

# useradd –u 4201 –g oinstall –G asmadmin,asmdba,dba –m grid
# passwd grid
Changing password for user grid
New password:
Retype new password:
passwd: all authentication tokens updated successfully

Checking kernel parameters

The Linux kernel has lots of tunables to affect the way it operates. The Oracle database makes intensive use of these, and you need to modify the standard parameters before you can install the database or Grid Infrastructure software. Many kernel parameters can be changed at runtime by echoing values to files in the /proc file system. To make these changes permanent, you need to modify the /etc/sysctl.conf file which is parsed at every system boot.

Oracle made it somewhat easier by adding an option to the OUI interface allowing you to run a fixup script to correct these values to their minimum required settings. If you are using Oracle Linux 5, you could alternatively install the oracle-validated RPM which helps setting some of the parameters before the installation. There is a similar RPM available for Oracle 12.1, named Oracle RDBMS Server 12cR1 Pre-Install RPM. You should also consult with your system administration team to adjust the values to fit your hardware optimally.

Tables 5-3 and 5-4 list the parameters and provide advice on setting them. Table 5-3 focuses upon semaphore parameters. Table 5-4 lists all the others. It makes sense to check the parameters even after having installed the preinstall RPM!

The values in Table 5-3 are all listed in /proc/sys/kernel/sem and are responsible for controlling the semaphores in a Linux system. Oracle uses shared memory and semaphores extensively for inter-process communication. In simple terms, semaphores are mainly required by processes to attach to the SGA and to control serialization. The values shown in Table 5-3 should be sufficient for most workloads.

Using the values from Tables 5-3 and 5-4, the resulting /etc/sysctl.conf file contains these lines:

 kernel.shmall = 1073741824
 kernel.shmmax = 4398046511104
 kernel.shmmni = 4096
 kernel.sem = 250 32000 100 128
 fs.file-max = 6815744
 net.core.rmem_default = 262144
 net.core.wmem_default = 262144
 net.core.rmem_max = 4194304
 net.core.wmem_max = 1048576
 fs.aio-max-nr = 1048576
 net.ipv4.ip_local_port_range = 9000 65500

You do not need to worry about these though. The Oracle Universal Installer provides a “fixup” option during the installation which can modify the kernel parameters to set the required minimum values. In addition, the Oracle server preinstall RPM ensures that you meet these minimum requirements. You can read more about the preinstall-RPM later in this chapter.

The Oracle mount points

The mount points and file systems suggested for use with the Oracle binaries have been discussed in the section “Considerations for the Oracle mount point”. Assuming that the storage has been formatted during the installation, the remaining work is simple: just update the file system table in /etc/fstab with a mount point for the Oracle software and you are done. Most often, the Oracle binaries are installed following the Optimal Flexible Architecture-OFA.

The starting point for mounting the file systems for use with Oracle usually is the /u01 directory and the hierarchy beneath. Alternatively you could use the top-level directory defined in your standards document. The next few paragraphs follow the OFA recommendation.

First, you need to consider the location of the Oracle inventory. The inventory is often owned by the oracle account but more importantly is owned by the operating system group oinstall. This ensures that in the case of separation of duties other operating system accounts have write permissions to the global inventory location. It is also the reason it is so important to define oinstall as the primary group for the oracle and grid user. In many deployments you will find the Oracle inventory in /u01/app/oraInventory, which is also the default.

Before discussing the installation location of Grid Infrastructure it is important to cover the ORACLE_BASE. For the Grid Infrastructure installation, the Oracle base signifies the location where certain diagnostic and other important log files are stored. From a database perspective, the most important subdirectory is the diagnostic destination which has been introduced with Oracle 11.

An important restriction exists for the installation of the Grid Infrastructure for a cluster: the GRID_HOME must not be in the path of any ORACLE_BASE on your system.

The default location for Grid Infrastructure in a cluster configuration is /u01/app/12.1.0/grid, but in the author’s opinion you should use one more digit to indicate the version number, i.e. /u01/app/12.1.0.1/grid instead. This will make it easier during patching to identify a software home. Remember that Oracle introduced full releases with the first patch set to 11g Release 2, which are out-of-place upgrades.

To sum it up the following directories are needed for an OFA-compliant installation. You can use these as mount points for the logical volumes defined earlier during the installation:

# mkdir -p /u01/app/oraInventory    # Path for the inventory
# mkdir –p /u01/app/oracle          # ORACLE_BASE for the database owner
# mkdir –p /u01/app/grid            # ORACLE_BASE for the grid owner

By defining directories with this structure it is ensured that the Oracle Universal Installer (OUI) will pick up the OFA-compliant setup. Assuming a separate oracle and grid user you would continue by setting the permissions as follows:

# chown grid:oinstall /u01/app/oraInventory/
# chmod 775 /u01/app/oraInventory/
 
# chown grid:oinstall /u01/app/
# chmod -R 775 /u01/app/
 
# chown -R oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/app/oracle

If you are planning on installing the database software only, there is no need for a grid user.

Setting session limits

The Pluggable Authentication Modules or PAM are a flexible way to authenticate users in a Unix environment. PAM itself is “only” a framework-its functionality is implemented via modules. Such modules exist for many authentication methods. When a user requests access to a Linux machine, the login process plays a crucial role in the process. Using the services provided by the PAM library, it assesses that the user requesting a service (bash/ksh or other shell) is actually who he claims to be. To that end, a multitude of authentication methods ranging from password to LDAP can be employed, which are beyond the scope of this chapter. From an Oracle point of view, one property of the login daemon is very important: it can assign limits to an individual user session.

The Oracle installation guide mandates that the following requirements are met:

• The maximum number of open file descriptors must have a soft limit of 1024 and a hard limit of 65536
• The number of processes a user can create must be at least 2047 with a hard limit of 16384
• A stack size (per process) of at least 10240KiB and at maximum of 32768 KiB

These settings are made in the /etc/security/limits.conf file, which unfortunately is still not documented properly in the installation guides. In this file, every line follows this format (see man 5 limits.conf):

domain    type    item    value.

The relevant domain for the Oracle database installation is a username. The type can be either hard or soft, and the item denotes which attribute is to be changed. To be more precise, we need at least amend the items “nofile,” “nproc,” and “stack.” On the x86-64 platform, the values set by the oracle-rdbms-server-12cR1-preinstall RPM were defined as follows:

[root@server1 ∼]# cat /etc/security/limits.d/oracle-rdbms-server-12cR1-preinstall.conf 
>| grep ^oracle
oracle   soft   nofile    1024
oracle   hard   nofile   65536
oracle   soft   nproc    16384
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768

You will notice a small variation to the requirements postulated by oracle: the value for the number of processes exceeds the recommendation by Oracle. The settings need to be repeated if you are planning on using the grid user to own the Grid Infrastructure installation:

grid   soft   nofile    1024
grid   hard   nofile   65536
grid   soft   nproc    16384
grid   hard   nproc    16384
grid   soft   stack    10240
grid   hard   stack    32768

The Oracle Database 11g Release 2 preinstall RPM installed its settings into limits.conf directly. The new version installs its settings into /etc/security/limits.d instead. The next time the oracle or grid user logs in the settings will take effect. The pam_limits.so module is automatically included in the login process via the system-auth module. This way, after traversing the requirements for an interactive shell login, the limits will be set for the oracle and grid user. You can check the limits by using ulimit command. More detailed information about these settings can be found in the man-page “pam_limits” (8).

Configuring large pages

Large pages, also known as “huge pages”, are a feature introduced to Linux with the advent of the 2.6 kernel. The use of large pages addresses problems Linux systems can experience managing processes on systems with large amounts of memory. Large amounts of memory at the time when large pages where introduced began with about 16GB of RAM which is not considered large any more. To explain, a little background information about memory management is required.

The Linux kernel on the Intel IA32 architecture (that is not Itanium!) uses a default memory page size of 4kb. All of the physical memory has to be managed by the kernel in tiny chunks of 4kb. Huge pages on the other hand use much larger page sizes of 2MB. It becomes immediately obvious that the kernel benefits from this as there are less memory pages to manage. But it is not only the reduced number of memory pages the kernel needs to keep track of; there is also a higher probability that the part of the CPU responsible for the translation of physical to virtual memory (the translation lookaside buffer, or TLB) will have a page address cached, resulting in faster access.

In earlier Oracle Releases you needed to manually calculate the number of huge pages before starting the Oracle instance. Oracle provides a script as part of a My Oracle Support note, called calc_hugePages.sh. Using it against a started Oracle instance it calculated the number of large pages. Thankfully, you get this information in the Oracle database instance’s alert.log now as shown here:

****************** Large Pages Information *****************
 
Total System Global Area in large pages = 0 KB (0%)
 
Large pages used by this instance: 0 (0 KB)
Large pages unused system wide = 0 (0 KB)
Large pages configured system wide = 0 (0 KB)
Large page size = 2048 KB
 
RECOMMENDATION:
  Total System Global Area size is 2514 MB. For optimal performance,
  prior to the next instance restart:
  1. Increase the number of unused large pages by
 at least 1257 (page size 2048 KB, total size 2514 MB) system wide to
  get 100% of the System Global Area allocated with large pages
***********************************************************

With the information that another 1257 large pages (of 2048KB) are required you can modify the /etc/systctl.conf file to ensure these are set aside when the system boots. The value to be entered is the sum of all the huge pages needed for all SGAs on the host plus a few extra ones for safety. Only one database will be used on the server in the following example:

vm.nr_hugepages = 1400

Memory permitting, you could try to reserve them without reboot: simply echo “1400” into /proc/sys/vm/nr_hugepages. If your memory is too fragmented this might not work and you have to reboot. The use of huge pages is shown in the /proc/meminfo file. To check if the requested number of huge pages is available, you could grep for HugePages:

[root@server1 ∼]# grep HugePages /proc/meminfo
HugePages_Total:     1400
HugePages_Free:      1400
HugePages_Rsvd:        0
HugePages_Surp:        0
[root@server1 ∼]#

The change to /etc/sysctl.conf alone will not allow the oracle (and grid) user to use large pages. Large pages require the memory to be “locked,” and large pages cannot be paged out. Since the 12c preinstall RPM does not set the necessary parameter in /etc/security/limit*, you need to do so yourself. Using your favorite text editor, modify /etc/security/limits.conf or its equivalent, and add configuration parameters similar to these:

oracle   soft    memlock   60397977
oracle   hard    memlock   60397977
 
grid     soft    memlock   60397977
grid     hard    memlock   60397977

The value to be set is in kilobytes. You could simply take the amount of RAM in your server minus 10 percent and set this in the file. Remember that this is not the allocation; it only defines how much memory the process may lock in theory.

A new session is required for these settings to take effect. If you want to ensure that your database enforces the use of large pages, set “use_large_pages” to “only”. Note this is incompatible with Automatic Memory Management! If you have memory_target set in your initialization file, you will get the following error:

******************************************************************
Large pages are not compatible with specified SGA memory parameters
use_large_pages = "ONLY" cannot be used with memory_target,
memory_max_target, _db_block_cache_protect or
use_indirect_data_buffers parameters
Large pages are compatible with sga_target and shared_pool_size
******************************************************************

If permissions are set correctly, namely the memlock item in /etc/security/limits.conf, and if enough large pages are available you will see a success message in the alert.log:

****************** Large Pages Information *****************
Parameter use_large_pages = only
Per process system memlock (soft) limit = 58 GB
 
Total System Global Area in large pages = 2514 MB (100%)
 
Large pages used by this instance: 1257 (2514 MB)
Large pages unused system wide = 143 (286 MB)
Large pages configured system wide = 1400 (2800 MB)
Large page size = 2048 KB
***********************************************************

It is possible that only a part of the SGA uses large pages, which is a scenario to be avoided. Setting the initialization parameter “use_large_pages” to “only” as shown in the preceding example ensures that the SGA in its entirety uses large pages. Please do not over-allocate large pages! Large pages affect only the SGA; private memory structures such as the PGA and UGA do not benefit from them. You need to leave enough memory available on the system for user sessions, or otherwise risk problems.

Introducing the oracle-rdbms-server preinstall package

Shortly after Oracle announced support for 11.2.0.3 on Oracle Linux 6 the company also released an updated RPM to simplify the installation of the software for RAC and single instance. Similar to the well-known “oracle-validated” RPM used with Oracle Linux 5, this new package performs many necessary pre-installation steps needed for Oracle database 12c (a separate RPM is available for 11g Release 2). Among those performed is the creation of the oracle account, modifying kernel and session related parameters. The package also modifies the kernel boot loader configuration file. It is a great help when getting started with Oracle, but it is not foolproof. As with any piece of software understanding the changes it makes to the operating system is crucial in configuring a robust system.

The preinstall RPMs are tied to a specific database version at the time of this writing, namely 11.2 and 12.1. During testing it made no difference to the installation of the 12c database and Grid Infrastructure packages whether the 11g Release 2 or 12c Release 1 RPM were installed. This is because the system requirements are very similar for both.

Before adding any of these packages to the default build, ensure that the package matches the build standard. This is especially important with regards to the user and group IDs. If needed, you can always get the source RPM (“SRPM”) file and modify the settings in it. In addition, none of the preinstall RPMs create a grid user for a separation of duties, nor do they install the new operating system groups introduced in the earler section “Creating the operating system users and groups”.

Partitioning LUNs

Before beginning the multipathing configuration which is covered in the next section it is beneficial to partition the devices at this stage. This prevents unloading and messing with the devices once the multipath configuration is in place. Two utilities exist for partitioning: parted and fdisk. This section focusses on the fdisk utility.

Once the LUN you are interested in is discovered on the operating system you can partition it. Most often you will get worldwide IDs from the storage team to use on the new database server. Using the /dev/disk/by-id directory it is possible to identify the SAN storage based on these WWIDs and to partition it. Pass the complete path of the disk as an argument to fdisk, as shown in this example:

# fdisk /dev/disk/by-id/scsi-1IET_00010001

Following this, you need to create a partition spanning the whole disk. The steps are shown in the below listing, where the following operations are carried out:

• The LUN is selected
• The display unit it changed to sectors
• A new primary partition is created with an offset. Different vendors require different offsets-check with your storage administrator on how to define the partition to match the requirements stated in the documentation. Also, many advanced formatting devices such as flash based storage are using a 4096 byte sector size instead of the previous 512 byte sectors. For optimal performance, partitions have to be aligned at 4k boundaries on these types of devices.

Following is an example. The text in bold are the responses that I typed in while generating the example.

# fdisk /dev/disk/by-id/scsi-1IET_00010001
 
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').
 
Command (m for help): u
Changing display/entry units to sectors
 
Command (m for help): c
DOS Compatibility flag is not set
 
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (2048-4196351, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4196351, default 4196351):
Using default value 4196351
 
Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.

This example created a primary partition on a SCSI device. With such a device partitioned, you can start configuring the multipath software.

Configuring dm-multipath

The principle behind any multipathing software is simple. Without the abstraction layer, the operating “sees” each block device once via each path. That means a LUN can be /dev/sdb and additionally /dev/sdk-but still be the same physical device, just using two different paths to the storage. Without a multipath driver the operating system could not easily assess that two devices are logically the same. Thanks to the driver however this becomes possible. Instead of using the native devices a new device is introduced to which the application (read: Oracle) sends I/O requests. The pseudo device is created in a number of different places:

• In the /dev directory such as /dev/dm-35
• Again in the /dev directory, but with /dev/mpathn or a WWID
• In the /dev/mapper directory with a user defined alias

In the case of the dm-multipath package, the mapping between block device and pseudo-device is performed in the main configuration file: /etc/multipath.conf. An important aspect is to only use the pseudo-device. Otherwise there would be no protection from path failures or no performance gains!

The installation of Oracle Linux comes with the multipath package installed as part of the standard installation. If the package is not yet available, you should install the package device-mapper-multipath.x86_64 including all its dependencies. When installed, you need to ensure that the multipath process is started at every system boot-this is done via the chkconfig application as shown here:

# chkconfig --level 35 multipathd on
# chkconfig --list | grep -i multipath
multipathd      0:off   1:off   2:off   3:on    4:off   5:on    6:off

Unlike Oracle Linux 5, there is no example content in /etc/multipath.conf-the file does not exist. A number of example configurations are available in the documentation directory. That directory is named as follows, but replacing version with your own version number:

 /usr/share/doc/device-mapper-multipath-version.

The quickest way to starting with a basic failover scenario is to use the recommendation from the online documentation by using the mpathconf utility as shown here:

# mpathconf --enable --with_multipathd y
# echo $?
0

The command creates a basic configuration file: /etc/multipath.conf and additionally loads the kernel modules necessary for the correct operation of the package. Querying the mpathconf command you are shown the successful execution:

# mpathconf
multipath is enabled
find_multipaths is disabled
user_friendly_names is enabled
dm_multipath module is loaded
multipathd is chkconfiged on

The next step is to review the newly created multipath.conf file, which is very similar to the format previously used in Oracle Linux 5. The file still is subdivided into section, using the familiar curly braces. The most important sections are these:

• Blacklist
• Defaults
• Devices
• Multipaths

The first section-blacklist {}-specifically excludes devices from being part of the multipathing configuration. This is necessary for local devices that should not be part of the configuration. A new directive, find_multipaths provides the administrator with some help in regards to blacklisted devices. Unlike the multipathing software in Oracle Linux 5, which tried to create a new pseudo-device for every path it encountered, this behavior can be kept in check without explicit blacklisting, using find_multipaths.

The next sections-defaults and device–are hugely vendor specific. Every storage vendor keeps information about the multipath.conf file and their storage products in their support portals. It is strongly recommended to either raise a call with your vendor or consult his documentation for the defaults and devices section. Interestingly the defaults section does not need to be supplied at all; the package uses built-in variables for any value not specified in the defaults-{} section. These defaults are documented in the /usr/share/doc/device-mapper-multipath-version/multipath.conf file. On an example system, the following values were used:

defaults {
        find_multipaths     yes
        user_friendly_names yes
}

All other values will be provided from the built-in defaults. This is not true for the devices-{} section, which overrides the defaults for a specific array. For example, the below has been copied from the Oracle Linux 6.4 multipath.conf.defaults file, and can be used for an EMC-Clariion array:

devices {
       device {
               vendor "DGC"
               product ".*"
               product_blacklist "LUNZ"
               path_grouping_policy group_by_prio
               getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
               path_selector "round-robin 0"
               path_checker emc_clariion
               features "1 queue_if_no_path"
               hardware_handler "1 emc"
               prio emc
               failback immediate
               rr_weight uniform
               no_path_retry 60
               rr_min_io 1000
               rr_min_io_rq 1
       }
}

Again, this is highly device-specific and should come from the storage vendor.

The last section, multipaths-{} contains specific mapping instructions for individual LUNs. This becomes clearer with an example:

multipaths {
        multipath {
                wwid "1IET     00010001"
                alias OCR001
        }
        multipath {
                ...
        }
        ...
}

This section is optional; multipath devices will be created even if there are no instructions for mappings between WWID and device name. In the above example, the device with WWID "1IET     00010001" will be configured as /dev/mapper/OCR001. The benefit of adding more human-friendly names is that troubleshooting becomes a lot easier. Instead of having to hunt down the planned purpose for device /dev/mpatha you immediately know why the device has been created. On the other hand there is added overhead involved in maintaining the mapping. Since the device naming can be a little bit confusing, here is a summary of how device names are created when using dm-multipath:

• If user_friendly_names is set to yes, the device will be created in /dev/mapper/mpath*.
• If user_friendly_names is set to no, the device will be created as /dev/mapper/WWID which is very unreadable for humans, making it difficult to find a specific device.
• If a direct mapping exists in the multipaths {} section, the alias name will be used.
• The internal devices (/dev/mpath* and /dev/dm-*) are always created, but should not be used in the context of the Oracle database.

For those who know the device mapper, you might wonder why there is no mention of the gid, uid, and mode settings. These were very useful in Oracle Linux 5.x to set ownership of LUNs when using ASM. Unfortunately this functionality has been deprecated, and once more UDEV rules have to be used instead. A recent alternative is the use of ASMLib. The use of udev and dm-multipath is a little complicated for an Oracle DBA, partly because LVM and the multipath driver share a very similar interface. Thankfully, the device mapper multipath module takes care of half the work. The rest needs to be done in a rules file.

All rules reside in /etc/udev/rules.d. Enter the directory, and create a new file. Name the file as 61-asm.rules for example. It is important that the file ends in “.rules,” otherwise it will not be parsed. The following setting allows the disks to be discovered by Oracle later, assuming a separation of duties:

KERNEL=="dm-*", PROGRAM="/sbin/scsi_id --page=0x83 --whitelisted --device=/dev/%k",RESULT=="1IET
00010001", OWNER="grid", GROUP="asmdba" MODE="0660"
 
KERNEL=="sd*", PROGRAM="/sbin/scsi_id --page=0x83 --whitelisted --device=/dev/%k",RESULT=="1IET
00010001", OWNER="grid", GROUP="asmdba" MODE="0660"

You have to restart udev to force the changes to take effect. This can be done without a restart, using the “udevadm trigger” command as root. The last example for this chapter demonstrates the correctness of the setting by launching a disk discovery from the command line. For example:

[grid@server1 OraInstall2013-08-25_05-22-30PM]$ ./ext/bin/kfod disks=all 
> asm_diskstring='/dev/mapper/*p1'
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group
================================================================================
   1:       2048 Mb /dev/mapper/OCR001p1                     grid     asmdba
...

As you can see, the setting is correct and the disk is discovered. Oracle executes the kfod command whenever it wants to configure ASM disks.