End of life announcements-Hardware

During the past few decades administrators and casual spectators have seen the demise of many platforms. Only very few, of which Linux is the most notable, have been able to step in the void and fill a gap. What almost looks like a zoo of platforms supported with Oracle 8i Release 3 from 1999 boiled down to a very modest few supported platforms with Oracle 12.1. Most platforms died silently without too much of an outcry in the user community.

However the biggest waves in recent times were caused by Oracle’s announcement to stop development for the HP-UX/Itanium platform. This announcement came at a time when HP was trying to (re-)define itself and the threat of the loss of a big revenue spinning platform was certainly not what the company expected. On their corporate website, Oracle states the following:

—http://www.oracle.com/us/corporate/press/346696

Ever since the announcement was made, the press had a real interest in the outcome of the story. What has been forgotten in this discussion was that Microsoft (and Red Hat somewhat more unnoticed) stopped their development for Itanium ports, which Oracle also acknowledges. This is a real shame since the processor design offered many interesting features, although in the end the Itanium range of processors proved to be exotic and not widely used. As any Oracle database administrator knows, the more widely used a platform, the less likely one runs into platform-specific problems. However, it should be made clear that an end-of-life announcement does not mean that Oracle abandons the platform completely, but rather that planning considerations on how to replace the platform should start sooner rather than later. In the above-mentioned, high-profile case with Itanium, Oracle issued a statement on September 2012 stating it was continuing to port the database to the platform.

From a planning point of view, although unpleasant, the de-support notice from the vendor for a certain platform might be suited to free up budget for a technology refresh of the database estate. Even though the effort might not encompass everything, the most critical systems at least need to be migrated before support for the current database release ceases.

High-profile Oracle customers might be lucky in prolonging support for an ageing hardware estate, but in general terms not acting when the time has come is going to introduce more problems later. Also, it should be noted that migration projects take a certain amount of time and planning, especially when the destination platform uses a different endian format. The author’s advice is not to delay the necessary until it is too late. Time pressure is known to introduce human mistakes!

In summary, end-of-life announcements provide the perfect occasion to get budget for a technology refresh, also affectionately referred to as “tech-refresh.” Even the tightest budget owner cannot ignore the fact that operations require supported platforms and hardware. Quite often, the platform to be replaced has outlived its useful life anyway, and (some) “big iron” can be replaced by industry standard hardware.

Support policy for Oracle Software

Hardware is not the only thing affected by end-of-life announcements. Oracle introduced their lifetime support policy with Oracle database 10g. The associated document, available from the Internet at: http://www.oracle.com/us/support/lifetime-support/index.html, details the Oracle product roadmap as well as the support matrix for not-so-current releases.

Think of the Oracle lifetime support policy as Oracle’s own product matrix, dividing the support Oracle sells into the following categories:

There is more to the lifetime support policy than can be covered in this short section, the reader should make sure to review the current document available from the Oracle website through the link provided in the introduction to this section. Additionally the reader should refer to the online resource as well to get a clear understanding of the situation. In summary, the lifetime policy offers peace of mind in situations where Premier Support ended. Nevertheless it is strongly recommended to upgrade wherever possible to avoid the monetary penalties.

At the time of this writing, the most widely used Oracle database releases have dates set for Premier, Extended, and Sustained support as shown in Table 1-1.

As stated above, and therefore not in this table, Sustaining Support does not expire until you stop paying licenses to Oracle. Although the options of staying on your current Oracle release may sound tempting, you are ultimately going to pay a higher price than necessary. I recommend implementing a policy in your company’s technology roadmap which moves ageing releases into the de-supported stage soon enough to guarantee a smooth transition. Chapter 12 will explain how to upgrade your current environment to Oracle 12.1.

Different kinds of consolidation

So far in the introduction you did not find specific information about the different ways to consolidate. Since that is important for the understanding of this book, I would like to introduce a systematic overview of the various aspects of consolidation.  Depending on your point of view there are different forms of classification, such as:

• Hardware consolidation
• Operating system consolidation
• Storage consolidation
• Platform consolidation

When speaking of hardware consolidation most people associate it with a move towards standard hardware. In many cases today this implies a move to blade servers or blades for short. Primarily based on the x86-64 platform, a blade server is a small somewhat naked looking device with the bare minimum required to function as a computer. Blades are not exclusively made for the x86 architecture. Apart from the mainstream there are enterprise blades for IBM Power and Oracle SPARC processors as well. If your IT department is absolutely sure it does not want to consider x86, then there is no need to move away from their favorite platform.

Multiple blades are combined in a so-called blade enclosure. The enclosure is mandatory and provides the power, cooling, and outward connectivity to the users of the whole system. Blade vendors have of course read the signs of the times already and add interesting management capabilities into their products. Chapter 3 describes blades in more detail.

Operating System consolidation is a term coined for those users who want to consolidate as many of their environments as possible under a common operating system. It is not to be confused with operating system virtualization, although the latter is often used in facilitating such a move! You have by now read a number of times that increased efficiency, a lowered license footprint, and a better support are the main drivers behind operating system consolidation.

Storage consolidation is another form of consolidation discussed in the media. Often Storage consolidation is initiated to counter the sprawl of storage systems in an enterprise. Due to cost constraints, absence of site-wide storage agreements, lack of oversight, or other reasons projects and product groups may have bought storage for their project on their own. These insular solutions are ineffective in the long run since a lot of time and energy has to be invested to keep the storage stack current. Additionally, storage technology is as complex as many other technologies, requiring in investment in talent to manage and operate it. The overhead of managing many different storage products becomes even more apparent during upgrades of components involved. Instead of repeating the certification and test process for controllers and firmware for each combination of storage array, fiber channel switch and host bus adapter synergies can be achieved by reducing the complexity of the stack.

Additionally, the move away from small or low-end storage may improve storage performance overall. One has to be careful though no to apply the rule without exception: storage is a complex topic, and the rule of thumb that one size fits all does not apply to all systems. High-end decision support systems and OLTP systems should probably not reside on the same storage array! The term storage consolidation is also applied when low-end systems move data from internal disk to a storage array.

As you may have already guessed, this book is about all of these aspects, something which is referred to as platform consolidation. Storage consolidation however will not be dealt with in more detail because many mid-sized and large enterprises are very likely to have an agreement with one of the major storage vendors to consolidate storage with them. This has the big advantage that storage sprawl is under (better) control, and also removes a lot of work from the storage engineers. Whoever has worked on a certified storage stack from storage array firmware to firmware version of the switches in the storage network to finally the host bus adapter will appreciate the reduction in moving parts.

Benefits of virtualization

The major vendors for virtualization products are keen to point out the advantages of platform virtualization. There are for example:

• Reducing cost (there it is again!) by better utilizing servers and all in all reduce the footprint in the data center
• Increase manageability
• Better ability to address a change in demand for resources
• Easier planning for disaster recovery
• Enhanced security

The cost aspect does not require a lot of explanation. It should immediately become apparent that by reducing the number of servers drawing power, there is less of a requirement for cooling, energy, and rack space. By better utilizing the existing servers less capacity is wasted idly.

Manageability is increased, and common “reference” templates make it very easy to get started with a new technology. “Standard” operating system images can be used as templates from which new virtual machines are rolled out very quickly. Oracle provides a wealth of pre-created templates for Oracle VM—the company’s premier enterprise virtualization product for the x86-architecture. Oracle’s competitors in the market, such as VMWare, use the same concept. In their language the getting-you-started-quickly Virtual Machines are called virtual appliances. Most commercial virtualization solutions feature a management interface for the administrator allowing him to see the state of his virtualized server farm. The better these management interfaces, the more likely the adoption of the product in the enterprise.

Another key element in virtualization is the potential to react to changed user demand. Many virtualization products allow the dynamic addition and removal of operating system resources at the virtual machine’s runtime. Imagine that the virtual machine requires more virtual CPU power for an important batch run. The administrator can grant a few more cores to the VM at runtime or otherwise increase computer resources. Live migration of a virtual machine has its main use in maintenance operations, but the flexibility to shuffle virtual machines around in the farm is very useful.

Finally, executing a workload on a virtualized platform offers more mobility in planned and unplanned downtime scenarios. For example, if the underlying physical hardware does not provide enough computing power then a live migration to another host can keep the users connected while the guest is moved to another server in the same farm. Similarly, disaster recovery solutions can be developed so that the impact on the service is minimized. If a virtualization vendor does not provide a product for the Disaster Recovery (DR) case, then it is usually possible to create one’s own.

More virtualization options

Apart from platform virtualization, there are other forms of virtualization to explore. From an Oracle DBA’s point of view the most important virtualization strategy is the use of operating system virtualization. The term is again somewhat loosely defined, but consensus seems to exist around the fact that an operating system can execute multiple isolated instances of itself. Good examples for operating system virtualization are Solaris Containers, and to a degree IBM Logical Partitions (LPARs).

Take a Solaris zone for example. Unlike hardware virtualization, a Solaris container cannot execute another operating system. This slight disadvantage is more than made up by the fact that the overhead with this virtualization method is very low. Solaris containers or zones are a very popular way to make better use of existing hardware, especially for development and user acceptance test environments. The whole storage presented to a zone can be block-level replicated to a remote data center, and with clever DNS changes make failover a simple operation. A simplified deployment, as opposed to unneeded complexity, is a target worth aiming for. The downside to the Solaris container approach is the relative high cost for the hardware. This is somewhat alleviated by the next generation SPARC chip, called T5. Initial benchmarking suggests that the single-thread performance has improved a lot from its predecessors.

A stepping stone on the way to the cloud

Readers with a bit of background of cloud computing will instantly recognize that many virtues of virtualization have made their way into the cloud paradigm. Especially the decoupling of location and user session is important, and virtualization vendors are integrating their own products into the public cloud.

Virtualization is of course used a lot in the cloud. For example, Enterprise Manager 12c “Cloud Control” has a built-in feature to manage Oracle VM server farms to provide cloud-like functionality inside the (private) data center, a deployment technique often referred to as “private” cloud. Cloud computing is yet another one of these overloaded terms requiring a little explanation, so let’s cover it next.

Infrastructure as a Service

My first contact with the cloud was by using Infrastructure as a Service (IaaS). Since then I have used IaaS in a number of projects. Opting for IaaS as opposed to other cloud deployment options gives the user the greatest flexibility, but also most responsibility for the environment. The option to install software on the system and to have full control over it gives the user said flexibility but it comes at a price. At the same time the system administrator is responsible for its security and for keeping it current in terms of patching.

The value proposition by the industry heavyweights (Amazon AWS maybe the best known) shows that the upfront investment in hardware can be costly. The initial deployment could also be undersized and unable to deal with unexpected peaks in the workload. Infrastructure as a service allows the user to add capacity on demand within very little time. In an IaaS environment the user gets a virtualized environment, often termed an instance. The user has full control, is able to install software, and modify the storage layout and any other parameter in the environment. Cost for operating such an instance is most often calculated depending on the type of the instance and its characteristics such as memory capacity, isolation from other users, and CPU power. Similar in a way to the popular root server which offers even more flexibility in regards of the software installation, a virtual instance needs to be maintained and hardened by its user to prevent security breaches.

Software as a Service

Software as a service (SaaS) is the most common touch-point to the cloud for non-infrastructure providers or developers. The model behind Software as a Service goes back a while, and in its basic form presents an application via the Internet. The application can be simpler (it wouldn’t really be simple) such as a web mail interface, or rather complex such as Sales Force.

SaaS as a concept probably does not need much introduction as it has been around for so long and many of us have already used it in some form. The first web clients appeared in the early half of the last decade, and since then there has been a consolidation of providers. Some of them, like Google, have successively enhanced their offering from the original core capability to web-based office suites. Established “fat client” vendors also offer an increasing number of web-based services to supplement their established revenue streams. A little while ago a hardware device was released by Google which is based entirely on the services provided over the Internet. Although such a class of devices is appealing in certain situations, the biggest drawback to them is their limited functionality when in offline mode. Who knows, we might see more of these in the future when true mobile broadband becomes a reality at a reasonable cost.

Platform as a Service

The final of the classic cloud computing layers to be discussed in this context is Platform as a Service (PaaS). This layer is a hybrid between full access as it is granted with an instance in an IaaS environment, and the ready-made web application available to you in the SaaS world. From the offerings available on the Internet, it appears as if PaaS is mainly targeted at developers, and it is aimed at making deploying applications easier. In addition to the developer focus, some PaaS providers have developed their own frameworks promising the user to deploy applications in very little time without being a hardcore coder.

Most providers of PaaS will allow access based on an API (Application Programming Interface) to the deployment environment. The benefit in such a scenario is again the limited cost of upfront development. A small startup company does not have to pay licenses to access Enterprise-grade software to get started with the implementation. The fee to be paid includes access to the hosted environment-specialist skill to maintain a few servers with databases and web server is often not required immediately. Previously, before the advent of a PaaS offering, the start-up fee presented substantial up-front cost before any revenue was generated.

The Public Cloud

When thinking about the Cloud as such, most users I spoke to immediately think about the public cloud. For example, your webmail client’s interface uses an external facing HTML 5 application hosted in a data center somewhere in the world. For most use cases this is not a big deal. Users of webmail clients and other services to synchronize files with multiple private computers do not worry where their date is stored. The security-conscious user thinks a bit differently and installs software to encrypt traffic to and from the cloud, and maybe the data that has been permanently stored in the cloud as well.

Commercial use of the public cloud requires a different approach. Many countries have very strict rules about data protection and privacy. While this is a great democratic achievement for the citizen, it imposes barriers to the use of the public cloud. If for example no personal data may be stored outside the country, then it is next to impossible to legally make use of external data centers.

Additionally the Internet infrastructure becomes very important: while most users enjoy a lot of bandwidth downstream, equal upstream capacity is either very expensive or simply not available. Anyone who uploaded data over the Internet will notice that immediately. Also a failure of the Internet connection will cut off the users from the service. Realistically speaking this should be a rare occurrence, but it can happen. Frequent travelers will know the problem of being disconnected during the time on an airplane, and roaming fees while travelling abroad naturally impose limits on bandwidth usage.

The Private Cloud

The private cloud takes the ideas already implemented outside the company’s data center in the public cloud inside the data center. There are many reasons for this: accessibility, connection quality, bandwidth, and security are just a few to mention. By “private cloud” I refer to a cloud-like deployment of applications and services which are not accessible to the public.

Private cloud is an oxymoron to the purist, and here is why. The Cloud Computing movement has advocated the shift of paradigm from capital expenditure (capex) to operational expenditure (opex). As you saw before, using for example Amazon’s EC2 offering you don’t need to invest upfront in dedicated hardware and storage (=capital expenditure), but pay for when you use it (=operational expenditure). So when using the in-house data center, clearly capex are not reduced.

When the benefits of automation, consolidation, and standards were discussed in this chapter, I implicitly assumed that they were implemented locally, in the company’s own, or alternatively outsourced, data center rather than with a third party.

In many ways application deployment in the private cloud is easier, mainly because the security, regulation, and other legal requirements are under tighter control. This of course can become a moot point if security is not properly implemented in the data center. Nevertheless tight security checks and measures should be implemented early during the product deployment stage to prevent frantic activity should audit points be discovered. This has become even more important after recent successful attacks on many high-profile services.

Security in the cloud

One aspect has not been discussed yet: security. Security is a big complex, and sometimes a psychological barrier more than a technical one. And when it comes to storing sensitive information in the cloud the mere thought of that rings the alarm bells with many. Who, for example, would want his mortgage data or other sensitive information to be stored in the public cloud? Even if such a storage option was one hundred percent secure from a technical point of view, it would be very hard to communicate that fact to customers.

For these reasons it would appear very unlikely for large enterprises with sensitive data to make use of the public cloud. And coincidentally this aspect is key in a design choice! The cloud hype has impacted the way infrastructure projects are managed in many ways and potentially changed the IT landscape profoundly. Management especially is keen to leverage the benefits the cloud computing paradigm offers, and this plays into the hands of the consolidation expert.

Use for the cloud hype

With many aspects of cloud computing hype, it is difficult to differentiate between plain wrong and correct information. I am hoping to have made the main aspects clearer for the reader, and especially why I think consolidation, automation, and the most suitable concepts taken from the cloud computing revolution fit into a bigger picture. Combining these properly we are well suited to fulfill the goals upper management sets to operations and engineering to help cut cost and streamline operations.

Processes with the potential for automation

There are a large number of database administration processes that can potentially be automated. When considering whether or not a process can be automated, audit requirements are to be taken into consideration as well. In recent times auditors have asked much more pertinent questions, and security problems have moved into focus. As an example, the following processes could be automated:

• Extension of a tablespace
• Taking a hot backup
• Unlocking a user
• Provisioning of a database
• Resetting a user password
• Creating a user
• Running an export of a schema
• And many more…

More fundamental tasks such as installing an operating system for the Oracle database server or deploying the Oracle database software are not listed here since they should already be automated. You can read more about these tasks in Chapters 5 and 6 covering the Linux installation and Oracle deployment.

It would appear logical not to allow any operation that could negatively affect the availability of the service to be automated. Not only could that cause much unwanted disruption, security problems with the code could also lead to malicious denial of service attacks. Shutting down or otherwise restarting a database or database service should probably not be in scope of a project to automate. Consider the simplified example in Figure 1-5.

Figure 1-5 describes one possible way to set up an automation gateway. A dedicated automation server, which has been specifically security hardened and equipped with an intrusion detection system (IDS) will accept communication from users via a self-service portal. The communication between the portal and automation server can be in many forms. In the past XML messages proved to be a popular means of communication, if sent securely. After verification of the validity of the message, to prevent illegal command injection, the request will be inserted into a workload queue. In addition to the syntax check there should also be a query against the enterprise trouble-ticket system to ensure that the raised request has been duly approved. The FIFO (first in first out) queuing mechanism can be configured either to execute the incoming requests immediately or delayed, depending on the request’s urgency. A one-time password should be generated for access to the database system, either from a local system or the enterprise standard authentication system. Access to the database server itself could be granted via SSH keys. And it goes without saying that all operations must be logged, ideally in a central location that cannot be tampered with.

The actual execution of the management task is most likely trivial in comparison. In the most basic form a shell script calls sqlplus with the verified and validated parameters and executes the task. Another possibility is the use of the Enterprise Manager command line interface to perform a task, but as with any development-related task there is more than one way to do it.

Auditing and securing automated processes

Many of the just-described operations require elevated privileges in the database. To facilitate a secure execution of automated change requests special precaution has to be taken. First, the scope of the change request has to be set. Many organizations may refrain from the idea of automating database administration tasks in a production environment. Although user acceptance tests (UAT) and development databases were traditionally attributed with a lower status than production, their importance for many applications ranks equally with production. It is very costly to incur downtime with a development database as developers cannot compile/test their code against it! Any automation should only go ahead after careful testing and an evaluation of the benefits against risks.

If appropriate coding, due diligence, and reviews of the automation process have taken place, logging and auditing have to be watertight to convince the auditors about the security of the implemented solution. In this context it is very important to ensure that the audit data is immutable. One way to secure the auditing information is to direct the information logged by the application and operating system to a trustworthy, central destination. If the implemented automation solution supports logging to the UNIX Syslog facility it is relatively straightforward to secure the audit trail. Modern Syslog daemons are capable of sending messages to a remote service.

Integrating acquisitions and mergers

Acquisitions form a special kind of challenge to the global support model. Although one can probably compare the IT estate of a company taken over with a previously independent business unit or a different region, acquired companies may use an entirely different approach to running Oracle. It is also possible that the newly integrated business uses a different set of hardware. In that case the systems are great candidates for consolidation, and we have come full circle to the first sections in this chapter. Whichever way the future direction of the common IT department goes, any change should be introduced gently so as not to alienate the “other” team!

However, an acquisition does not necessarily imply a negative challenge from a technical point of view. Sometimes companies are too successful, however paradoxical that might sound. Business units in such companies are very likely too busy adapting to the rapid requirements of customers to deliver new functionality in their software products that there is simply no time to develop a standards framework. And quite often there is not even time to consider how the estate should be managed when a “steady phase” is eventually reached. In such circumstances the acquisition of another company that has reached a healthy balance between the requirement to produce new features and the need to develop a corporate standard is most likely a great opportunity. Instead of having to reinvent the wheel, an existing standard model can be used as a baseline and refined to match the growing company’s needs.

How much standardization is needed?

Defining the scope of the standards set to be implemented is a difficult task. You read in the introduction to the standards section that very few business units use engineered builds right from the outset of their existence. As a direct result, very few Oracle installations look alike. And as you also read before, this makes it difficult for someone who is not familiar with an environment to support it efficiently. But often, as soon as efforts are set underfoot to align the configurations, difficulties are not far down the line.

The first logical choice is to set up a central body responsible for the definition of the standards. In large companies, this usually ends up in the hands of the engineering department. After this transition of “power,” it is possible that regional or product-aligned representatives, who in the past decided about deployments, versions, and other details feel sidelined.

Again, a gentle approach by the engineering department is needed, and as you can imagine regional representatives might need gentle nudging from time to time to move in unison. Where appropriate, regional “best practices” should be considered for inclusion in the standards document, and certainly not brushed aside. In the same way as concessions need to be considered, it also has to be made clear that the owner of the standards has the final say in how a specific feature is to be implemented. Therefore it is very important that the engineering department or whichever other team performs the task has full backing from management.

New environments brought into the organization, for example appliances which offer little (or less) possibility to fine-tune the setup pose a challenge. Should the appliance be “pressed” to fit the corporate standards, or should it remain outside? This question can only be answered individually. Very often, vendors claim their appliance is specifically built to make it easier to support. This may be true for the vendor, but not necessarily for the customer. If your vendor is flexible, it should hopefully be less of a problem to integrate standards into the product, but you might have to take initial teething problems into account. On the upside, however, management of the environment can be greatly simplified, and another application-specific siloed team of DBAs is avoided.

Another challenge is the evolution of standards. As business requirements change, and companies adopt new technologies, existing standard documents have to be reviewed periodically to ensure the current standard document is not holding the teams back. Such a review is more or less unavoidable when a new Oracle database release is adopted for production use. In this case though there should not be any problem to incorporate a new feature into the standard since it cannot collide with anything in the standards document. If the document structure permits it you could simply add a section pertaining to the new release. The standards-owner needs to discuss whether or not fundamental changes to the standard are retrofitted into existing environments or not. If it is decided to only build new systems to the current specification, environments could become fragmented again. On the other hand, changing environments to match the standards can be a very time-consuming process and most likely requires an outage, especially of the operating system or Oracle directory structure is touched upon. It is to hope that the standard document is suitably mature before modification to prevent said fragmentation and to preserve a common set of commands and directory structures. A newly developed consolidation platform is a great opportunity to test and implement the latest edition of the standards, and to ensure they are suitable for this kind of environment. Before going into detail as to which elements of the software stack could be standardized, a little discussion of the different aspects is necessary.

Standardization of the full stack

From a high level, multiple aspects to standardization can be made out. When thinking of standardization and certification you need to consider the full stack relevant to database deployments. To keep it in scope with this book, the discussion begins with storage, continues with hardware and the operating system and ends with Oracle software. The full stack encompasses even more: middle tier, the network layer and finally the clients. Since this book is about the database I will not discuss the higher tiers in the stack.

In large organizations, there are often separate engineering teams responsible for each of them:

• Storage engineering
• Operating system engineering
• Database Engineering

These teams can further be subdivided, adding complexity to the task of coordination of standards. It is not uncommon that these teams to work as if they were isolated from one another. Rarely are interfaces between the teams defined, which is unfortunate. These interfaces are very important for automation and consolidation. If, for example, the operating system team changes vital parameters of the Linux build, it may result in additional overhead for the database team to ensure all prerequisites for a database installation are in place.

In an ideal world the individual team would detail the current build on an internal website including external (to the team) interfaces and their change history. Everyone interested in a specific interface could subscribe to changes in the documentation and act accordingly well in advance. This is far more preferable than having to find out the hard way via failed builds and installations that an interface has changed. Unfortunately the benefit to the team is not immediately visible, which can cause such documentation tasks to be put on to the list of low-priority tasks.

Potential storage standards

The group responsible for storage backend system engineering is typically responsible for tasks such as these:

• Liaise with the storage vendors.
• Develop and support backup and recovery technologies.
• Certify storage technology for use with operating systems.
• Manage maintenance of non-Oracle file systems and volume managers if the UNIX engineers do not already take care of them.
• Develop and maintain block-level replication best practices and recommendations.

There are obviously many more tasks performed by the team, but from an Oracle DBA point of view the ones mentioned above are the most important. The third bullet point mentions certification of storage technology for use with operating systems. In this respect, storage engineering researches and evaluates new storage technology such as the use of Enterprise Flash Drives (EFDs) in traditional storage arrays, flash-based storage solutions inside and outside of the database server, new versions of backup and recovery software, and many more. The aim is the provision of a uniform solution internal customers can choose from.

In an environment where strict separation of duties is enforced the storage layer is often transparent to the database administrator. The introduction of Oracle Automatic Storage Management (ASM) with Oracle 10g Release 1 brought big changes for the storage team. Most of these changes are process related. The design and operation of ASM is fundamentally different from the way file systems have traditionally been used. As a consequence the storage engineer and administrator needs to develop an understanding of ASM, especially in the context of Real Application Cluster databases if they continue to be responsible for this part in the stack.

Standards developed by the storage team do not normally clash with the needs of the database administrator. Depending on the level of adoption of Oracle technologies in your organization, discussions will arise around the deployment of Oracle Automatic Storage Management and replication. Many storage engineers favor block-level replication on the storage level over application-level replication such as Data Guard. Unfortunately, the debate often becomes a religious one without a lot of reasoning to the arguments. To aid you to formulate an opinion, the pros and cons of block-level and application-level replication will be discussed in Chapter 9 which explains techniques for setting up disaster recovery solutions for Oracle databases.

Potential operating system standards

The Operating System engineers oversee the general direction of supported operating systems for use within the organization. Common tasks performed by the OS engineers include:

• Creating automated standard builds for the supported platforms.
• Maintaining a lifecycle matrix.
• Reacting to security vulnerability reports.
• Providing tested hot fixes.
• Evaluating (new) hardware for mass-rollout.
• Third-line support.

The most important aspect is certainly the building and maintenance of the standard operating system image for use by other consumers. One could envisage a tiered set of packages and configuration provided—a bare metal, an application server-specific set of packages and settings, or a database server package for example.

Changes to the operating system build for the Oracle database are critical to consistently deploying databases successfully. From a standards point of view it is very important to keep in touch with the operating system engineer. The standards defined in the operating system document touching the database installation are related to everything listed in the Oracle installation guide in the preinstallation requirements section. In other words the Oracle-specific package must define parameters in /etc/sysctl.conf as well as user shell limits and other relevant parameters. You can learn more about these requirements and how to automate them in Chapter 5. Ultimately the efforts of the engineers for creating operating system images need to be documented and then automated and regression tested for their respective use.

Potential database standards

Finally, we will address the standards the Oracle database administrator is most concerned about. After it has been ensured by means of open communication that the storage layer and operating system meet the requirements laid out by Oracle to install the database software, further detail can be cast in stone. Database standards can be roughly subdivided into two groups:

1. Operating system related
2. Database related

The first set of rules should be set around the operating system build, and ideally it is consistent across platforms. Items which certainly require standardization are the mount points for the database home and Grid Infrastructure (if applicable). Also, a generous use of space should be made for these mount points. Oracle homes grow and grow with every release, and many upgrades cannot be installed in place. Personal experience shows that the actual path to the Oracle binaries is not so important, as long as it is consistent. Paths such as /oracle, /u01/app/oracle etc. are all perfectly valid paths, if you decide to use one of them consistently.

The origin of the space needed for the mount points can either be on the storage area network or alternatively come from a set of local disk. If stateless computing as propagated for example by the Cisco UCS is used, then placement of Oracle binaries on the storage array is a big advantage.

Additionally the numeric group IDs and user IDs for the Oracle account and any supporting groups and users should be defined. This ensures that cluster installations succeed, which requires identically configured accounts. Even if you are not planning any cluster installation right now, you could be asked to do so in the future.

Difficulties with the current operational model

As with any theory, it is easier to talk about it than to implement it. Until Oracle 10g, the strict separation of duties between storage administrator, operating system administrator, and database administrator worked nicely. The storage administrator provided LUNs from a large machine in the data center. The storage administrator or the OS administrator then created physical volumes for use with the proprietary volume manager, and added file systems and other attributes the DBAs did not get involved with. Only then was the mount point information sent over to the database administrator. By and large, the infrastructure was out of scope for the 8i/9i DBA.

Then Oracle started to incorporate more and more infrastructure elements into its product. For example, beginning with 10.1, Automatic Storage Management has been introduced. This takes a little chunk out of the operating system administrator’s role. No longer does the DBA have to worry about mount options, concurrent I/O, direct I/O, and whether the file system can do asynchronous I/O or not. The storage model across all supported platforms can technically be unified. Similarly, for Real Application Clusters there is no more need for non-Oracle cluster software. Oracle Clusterware is perfectly capable of performing this task, especially now with IPMI support for proper fencing. Most organizations I know of delegate the administration of the cluster stack to the Oracle DBA, again widening the DBAs activities and narrowing the scope of the Unix administrator. And there are many more examples like this.

Coming back to the classical separation of duties that worked so well, taking care of this new set of responsibilities has quite drastic consequences. The DBA is asked to do more and more, but the tools and privileges he has at his disposal have not kept up with the times. For security and audit reasons the oracle account does not normally have the privilege to sudo to the root account. Elevated privileges however are very often required to do the job. For example, if you want to start the Oracle Restart software stack after troubleshooting a problem, root privileges are required. Similarly, certain diagnostic commands require root privileges and patching.

Forward thinking is required to overcome these difficulties. An appropriate communication structure, encompassing mitigating controls around procedures needs to be put in place to prevent DBAs from waiting unnecessarily for required privileges. When it comes to the standards documents, this needs to be clearly reflected. Initial problems with the process can be overcome if all parties involved work towards a resolution, instead of blocking it and insisting on their own position.

Or maybe it is time to reflect about the operations model? Maybe there is place for a new job title, a person who primarily deals with the infrastructure required to run an Oracle database—the title “Oracle Platform Operator” has been suggested for this. The OPA deals with storage, clustering, patching, and whatever else is appropriate in your organization. The Database Administrator can then finally go back and focus on application support. The time is near where the title Oracle Database Administrator does not reflect the actual daily work carried out.

A little bit of (UNIX) history

Speaking in greatly simplified terms, Linux is the result of a computer science student’s experiment, which has been continuously improved by a large number of mostly nonpaid volunteers. Most people consider a Linux distribution as “Linux,” but there is a subtle distinction to be made. Linux is only really the operating system’s kernel. The kernel includes the low-level drivers to address the hardware, and presents an operating environment to the rest of the ecosystem. Most of the utilities we like, such as bash, tar, and most other GNU tools were ported. Since it is impossible to explain the success of Linux without going even further back in time to its cousin, Unix, a little wider explanation needs to be made.

UNIX was written when operating systems were still in their infancy. Multi-user and multi-tasking support, which are taken for granted today, were certainly not the norm. UNIX meant to change this when Ken Thompson and many others developed it in 1969 as members of the Bell Laboratories. A child of its time, UNIX was entirely written in assembly code. This changed a few years later when it was ported to the C language. This was a great step ahead since it allowed the code to be ported to a larger number of platforms.

One particularly notable element of UNIX was that it (including the sources) was easily available to universities. This not only included the right to use it, but also the kernel and utilities. Like Linux today, that made for an excellent academic research object. For a nominal fee, tapes containing the operating system could be obtained and used. This unique feat is owed to the fact that AT&T, owning Bell Labs at the time, was not allowed to commercially engage businesses other than telecommunication. Many universities made use of the early UNIX and some even created their own derivative. A notable example is the Berkeley Software Distribution (BSD) of the University of California, Berkeley.

The legal requirement not to distribute UNIX commercially was lifted for AT&T when Bell was removed from the AT&T world in the early 1980s in what will turn out to be a very fortunate event for the UNIX community. This sounds paradox, but the commercialization as System V (SysV) of the UNIX code prompted researchers to think about freedom of information. One of the most prominent is Richard Stallman who started the GNU project in the same year: 1983. GNU is a recursive acronym and stands for GNU. It is not UNIX. The aim of the GNU project was to create a free version of the UNIX operating system, including a kernel as well as the necessary command line utilities. While the latter effort was successful, the endeavor to create a kernel was severely impacted by the release of Linux-but I am getting ahead of myself. Freedom in the GNU-context means that anyone should be able to use the software without paying a fee or license charge, and the source code had to be provided as well for modification or study. While this sounds radical to many vendors, it goes even further. The most interesting aspect of the free software ideal was the license under which the software was distributed. The GNU General Public License implements a “copyleft,” instead of “copyright.” Instead of keeping all the power in the software producer’s hand, the GPL as it is generally known reverses this concept. Software based on the GPL also has to be placed under the GPL, which prompted some observers to call it viral. It has however not prevented some of the most advanced and widely spread applications to be written and distributed under the GPL.

But before that could happen some other important events took place. In the early 1980s UNIX had split into many different ports, but all had in common that they were either BSD-based or based on the original sources from AT&T/Bell. Commercial versions of UNIX also began to appear throughout the 1980s: SunOS, Xenix OS, and others were based on BSD, while Solaris, HP-UX, AIX and IRIX were by and large descendants of SysV. The BSD Unix variant went through a difficult phase, when AT&T sued the University for copyright infringement. The claim was based on the fact that BSD was initially based on the free Unix sources from AT&T. The lawsuit prevented wider adoption of the Berkeley Software Distribution.

An attempt to merge the BSD-base provided by SunOS and the SysV branch of UNIX in System V Release 4 caused enough aggravation between vendors not part of the deal to form a counter-coalition. The to-ing and fro-ing would result in what is today referred to as the UNIX wars. Fractionalism between various vendor-coalitions lasted until the mid-1990s when the Open Group was formed. This consortium, which is very active still today, has published the Single Unix Specification and holds the UNIX trademark. Only systems meeting the requirements are allowed to be called UNIX systems. All others are UNIX-like operating systems.

Most UNIX systems discussed so far require special hardware to run which made them inaccessible for most outside of universities. With the proliferation of the personal computer and its increasing capabilities towards the mid-1990s users had the technical requirements to run Unix systems on their hardware, except that there were very few ports to the Intel IA32 architecture. This would change soon.

Enter Linus

With all of this background information, most notably about System V Unix, the Berkeley Software distribution and the GNU/free software movement it is possible to explain the Linux phenomenon. Linus Torvalds, at the time a student at the University of Helsinki, started developing an operating system to exploit his Intel 80386 hardware. Making use of the now almost forgotten Usenet, he posted his ideas to the Minix user group asking for opinions about his project and he got plenty. Interestingly the software was developed in a very distributed, de-centralized way by lots of interested individuals around the world. Linus was, and for most aspects still is, the maintainer of the kernel code. He incorporated a lot of these ideas to the kernel and over time the project got a lot of momentum. The code was eventually released on an FTP server, at a time where a full operating system code tree wasn’t available.

When early versions of Linux were developed, they greatly benefited from the work done by the GNU project initiated by Richard Stallman. Linux was the missing piece in the puzzle: when the GNU project set out, not only did they want to write the essential utilities, but also a free kernel, codenamed “Hurd.” The first few Linux versions saw a lot of the GNU utilities ported to them, which was essential in making Linux more user accessible. This is also the reason Linux really should be called GNU/Linux.

Initially Linux was developed under Torvalds’ own license, but was eventually put under the GPL. That move to the GPL was a great benefit to the free software movement. Very much like the Internet in its early days, there was a purist approach to Linux and commercial interest was not the driving factor behind development. In fact, it took a long time before companies could start to make money off of Linux. Most of them do this by bundling up software in what’s referred to as Linux distribution. To overcome skepticism in the industry, today’s commercially successful pioneers Red Hat and SUSE created their Enterprise Distributions. Whereas Red Hat is going from strength to strength, the same cannot be said about SuSE. Nevertheless their distributions are both widely spread, with Red Hat Enterprise Linux 6 and SuSE 11 being the current distributions. Another noteworthy distribution for Oracle, Asianux, is less widely spread in the English-speaking world but focuses on Asia, as the name implies.

Thanks to the copyleft clause in most free software, any distribution has to distribute the source code as well. This allows “clones” of popular distributions to appear. For an Oracle administrator, distributions based on the Red Hat Package Manager (RPM) are most relevant since only these are supported. Of all these, Red Hat is probably the most important one due to its market share. There are a number of clones for Red Hat: Scientific Linux and CentOS are the most popular. A more recent Red Hat derivative is Oracle Linux, which started off as a near identical clone without the copyrighted material such as logos. Additionally, Oracle has developed its own kernel which is shipped in parallel with the Red Hat kernel, while otherwise maintaining capability with Red Hat.

The non RPM-based distributions like Debian, Ubuntu, and their derivatives may have wider adoption, especially on the netbook and laptop class of hardware than Red Hat clones. However, they do not play a role for Oracle database deployments.

Despite the initial skepticism and aggressive marketing efforts trying to diminish its importance Linux is powering lots of business-critical applications and managed to get into even the most conservative industries. It has more than challenged traditional UNIX variants, for two main reasons: it compares to them in terms of scalability and stability, but it can also be deployed on industry standard hardware, significantly lowering the cost of entry for businesses.

Why is Linux so popular?

The answer to this question has two dimensions for the author: the first one is personal, the second expands on the first and shifts the focus to a business point of view. When I started to have a more serious interest in computing and the needs of data processing, it was next to impossible to get hands-on experience with the relevant commercial packages used in enterprises at the time. This includes databases such as DB2, Oracle, Informix, Adabas D, and others which required platforms not available to me . What this meant was that the interested student of computer science needed to go to a university to get access to the previously mentioned systems. The hardware platforms to run Solaris, Tru64, HP-UX, or AIX were simply unaffordable to a student.

Linux changed all that inaccessibility. In the late 1990s many relevant applications had been ported or porting efforts to Linux were underway. Thanks to its flexibility Linux ran on almost any student’s hardware, and allowed instant access to the Oracle database for example. Whenever you had an idea and wanted to try something out (maybe LDAP integration with the database), you didn’t have to dial in to the university campus network (those of us were the lucky ones) or physically get to the campus. Instead you started your PC, selected Linux from the boot menu, and began experimenting. This was taken further in the last few years where memory and disk space became cheaper commodities, allowing users to use virtualization to build lab environments to be used with Oracle clustering and RAC databases. Linux was the catalytic element by significantly lowering the entry hurdle to high tech. We have come a long way since the initial ports of Oracle to Linux. No longer is it necessary to buy extensions to an operating system to build a cluster. No proprietary interconnect technology had to be purchased, no expensive clustering software was needed, no additional cluster logical volume manager, and so on.

Of course this trend is a double-edged sword, and vendors of specific clustering technology are under pressure. And so are the traditional Unix vendors: some have disappeared altogether; others were bought. But the common problem all of them face is the diminishing market share. The Intel/AMD platform certainly has played a major role. Where Unix lost ground, Windows and Linux gained. Due to the nature of the Open Source model it is difficult to get real numbers of Linux deployments. Oracle Linux is free to download and use, and I assume a great number of deployments will never be counted. Any figure comparing Linux market penetration with other operating systems therefore is to be viewed with this in mind.

The only serious alternatives to Linux on i386 at the time I started a serious interest in computing were SCO Unix and UnixWare and Solaris. Sun, still independent at the time, had a few attempts at porting Solaris to the Intel platforms but could not enjoy the same level of success as Linux. This does not do the Solaris Intel port any justice: Solaris 11/Intel is a mature operating system despite its slow adoption.

If one considers that many students of computer science had exposure to Linux at university, and assuming that experience did not put them off, this helps explain the success of Linux in the commercial world. A large number of graduates are likely to prefer Linux over other operating systems leading to a generational change in corporations. This again shows why Linux is gaining ground, and the lower initial cost of deploying Linux helps the proliferation.

Another reason Linux is a desirable platform comes from Oracle Corporation. When the development/reference platform for the database and some other products changed to Linux the Linux support from Oracle became a lot better. Rather than having to wait for a port from the reference platform, Oracle Support can coordinate the release of corrections a lot more efficiently.