Chapter 2. The Database Cloud Administrator—Duties and Roles

In Chapter 1, “Database as a Service Concepts—360 Degrees,” we introduced the core concepts that define and drive cloud computing in general. We went into some detail about how the concept of cloud computing applies to the database world in general. We talked about the roles and interactions that exist in a cloud environment at a very high level.

In this chapter, we explore what it means to be an Oracle database administrator (DBA) in the cloud computing world. The basic role and functions of the DBA working in a cloud environment are not much different than in a traditional, on-premises environment. What does change with the cloud paradigm is the thought process and viewpoint.

DBA Responsibilities in a Traditional Environment

In the traditional database paradigm, the DBA role falls loosely into two categories: the application DBA and the operational DBA (see Figure 2.1). They work on the same database but look at it from two different perspectives. DBAs in both roles must understand the core concepts and internals of the database to perform their job. They also need to understand the various database features, how they work, and how to best use them.

Figure 2.1. DBA responsibility in a traditional model

Depending upon the role or interests, DBAs may gravitate to functionality at an object level (partitioning, indexing, compression, virtual columns), to functionality useful to programmers and database users (results caches, advanced queueing, or Java within the dataset, XML processing, etc.), or to functionality applicable at the database level and focused on infrastructure needs (Data Guard, Real Application Clusters [RAC], recovery management, encryption, etc.).

Application DBAs tend to work with databases from a logical implementation viewpoint. They look at the database from an application- and data-centric focus. They are more in tune with relational data models and star schema warehouse models, data relationships and usage, and database features directly consumable by the application team, such as partitioning, queueing, replication, Procedural Language/Structured Query Language (PL/SQL) features, and enhancements.

Operational DBAs, by contrast, tend to look at the database from a physical implementation perspective. They administer the physical aspects of the database, such as backups, database storage allocation and capacity, database server health, disaster recovery, and consolidation. Operational DBAs are focused on database features that are directly consumed by the infrastructure side of the equation. The benchmark of success is meeting key performance metrics around I/O and network latencies.

The roles of the application DBA and the operational DBA are equally important, and in many areas, they overlap. Both must understand the core concepts of the database and how it functions. Both are concerned with performance tuning, availability, and scalability.

Take performance tuning as an example. An application DBA tunes the database with the purpose of optimizing the application and its use of the database. An operational DBA tunes the database with the purpose of optimizing the physical infrastructure—reducing I/O latencies or pressure, relieving memory pressure, improving operating system performance, and fine-tuning the network.

More specifically, both DBAs tune SQL. The application DBA tunes SQL by understanding the application and how it uses the data and by working with the development team to optimize that process. The operational DBA tunes SQL by looking at Explain Plans, I/O characteristics, and latencies.

The most successful DBAs are those who can look at the same problem from both viewpoints. The best solution is the one that balances infrastructure and application concerns.

DBAs are often seen as the gatekeepers of the databases. They see the database as “their” property and themselves as the guardians of their database. Server administrators, storage administrators, backup administrators, and network administrators share this outlook as it applies to their areas. Consequently, IT infrastructure management is often seen as a collection of siloed groups, much like an island nation of small, individual islands.

In actuality, there is a smaller and relatively unknown group of DBAs that we call database engineers. This group of DBAs straddles the fence between the various groups responsible for the IT infrastructure, and their role is similar to that of application DBAs. Database engineers work closely with the infrastructure teams—with the storage team to build adequate storage, with the server team to choose servers and operating systems to best support database operations, and with the network teams to configure the networking, all to ensure that the database performance can be maximized and optimized.

What’s Changed with DBaaS

With the evolution of cloud computing, the roles and viewpoints of the DBAs also have evolved. Cloud computing introduces a fundamental change in database ownership and responsibility. This change has a ripple effect felt throughout the entire workflow between applications and the database. As a result, each individual role is now distinct, visible, and critical in its own way.

As defined in Chapter 1, the database cloud, or database as a service (DBaaS), is all about providing a platform that frees the application development teams from having to worry about servers and hardware, allowing them to focus all of their attention on application ownership and development.

In other words, DBaaS splits the ownership and responsibility into two tracks, one at an infrastructure layer and the other at an application layer. DBaaS divides performance responsibilities by differentiating between application-owner performance and infrastructure-owner performance:

• Infrastructure teams own and are responsible for performance at a hardware and infrastructure layer.

• Application teams own performance responsibility from an application perspective, which includes how the application interacts with the database.

DBaaS divides scalability responsibilities, once again, by differentiating between application scalability and infrastructure scalability:

• Infrastructure teams own the responsibility of creating a platform that is elastic and scalable on demand. In other words, infrastructure teams own managing capacity in anticipation of growth.

• Application teams own the responsibility of creating an application that is scalable on demand and, more important, understanding and obtaining what the application needs for scaling.

Availability responsibilities also are divided by ownership. It is crucial to understand the distinction between performance and scalability, on the one hand, and availability, on the other. Performance and scalability have a direct impact on availability.

The database and infrastructure team is responsible for availability of the service itself, and the application team is responsible for availability at the application layer. This statement is broad and rather useless unless we better define and distinguish service availability and application availability.

To start with, let us agree that if the application is unresponsive to the end user, it can be considered as being unavailable. The real question is, Why was the application unresponsive to the end user? The difference between the availability of the DBaaS and the availability of the application is rooted in the problem that is making the application unavailable in the first place.

From the DBaaS team’s perspective, if the database is up and running and is able to respond to a login request, it is considered as being available. But that seems too generic and simplistic; therefore, we add quality of service (QoS) and the corresponding service level agreements (SLAs) to this definition of availability. The SLA might state availability requirements in terms of I/O response time as compared to I/O throughput expectations, such as input/output operations per second (IOPS), megabytes per second (MBps), and CPU capacity. Therefore, from a DBaaS provider perspective, the database is considered as being available if it is up and running, accepting database login requests, and producing QoS metrics that show all SLAs are being met.

Some examples where the DBaaS provider will be responsible for the outage and for identifying and resolving the issues are:

• The archive destination space was not managed properly and fills up as a result. The database is unavailable because the archiver has insufficient space to archive and release the online redo logs.

• There was an issue at the storage area network (SAN) layer, which caused the I/O performance to drop dramatically.

• A database backup job kicked off at the wrong time or has been running for too long and is impacting the data file I/O performance.

• The database or the server crashes or aborts due to either a hardware or a software bug.

The key point is that, from a business unit perspective, availability is based on the ability to perform the business function according to the SLA. In other words, this is the availability of the entire application the business uses to deliver the function. For an application to be available, therefore, the infrastructure must be available and the application must be able to use and capitalize on the infrastructure strength. Let us take a deeper look at availability from a database perspective.

Database availability is defined in terms of the database being open and able to accept connections and respond to a beacon set of requests within a defined response-time window. Events that would constitute the database itself being down include, but are not limited to, the following:

• The database being hung in an archiver stuck mode.

• The database failing requests due to lack of space.

• The database is up and running, but the listener is down, and therefore the database is not available.

Once the database is deemed available, the application owners and developers are responsible for identifying and resolving the issue that is interfering with availability. The reason this rather hard line is drawn here is that the application development team has the ownership of and authority over the application design and architecture. In such cases, an availability hit is recorded against the application, and it is the application team’s responsibility to identify and remediate the situation. Following are a few examples where availability is impacted by application design and architecture:

• Excessive row-level locking and blocking causes an unresponsive application.

• The application does not manage its connection pool properly and consequently exceeds the configured maximum number of connections. As a result, the application receives an error when establishing connections and is rendered unresponsive and unavailable.

• The database load and throughput exceed the volume specified by the SLA for the solution design.

• Availability requirements of the database have increased since the original solution design.

In terms of cloud service, these examples emphasize the need for proper and clear definition of what is called the service catalog. The service catalog describes all the responsibilities and expectations of the cloud provider as well as of the cloud subscriber. From an availability perspective, for example, the service catalog would describe

• The definition and expectations on overall SLAs related to the database infrastructure.

• The QoS and beacons used to measure and report on SLAs.

• The definition of planned outages versus unplanned outages and their durations.

This discussion leads directly to the topic of feature set and capabilities of a good DBaaS solution. Not only must the solution be proactive in terms of ability to handle pure infrastructure issues, it must also be able to assist application owners in identifying, addressing, and managing application availability. In other words, a good DBaaS solution is one that provides the features and toolkits that enable the consumer and provider to be proactive. For example, the solution should have in place

• A monitoring and alerting system that is comprehensive and customizable in terms of metrics monitored and who receives the notifications.

• A dashboard system that clearly shows how the service is performing against the QoS goals and SLAs.

• A system that allows the consumers of DBaaS to scale and grow on demand.

What this really means is that DBaaS has changed the database paradigm by redefining and redrawing the lines of ownership and responsibility and, at the same time, combining the individual infrastructure fiefdoms into a single role, at least from the end user viewpoint. End users expect a service delivered at an agreed-upon quality level, with the toolkit and functionality that gives them visibility, control, and flexibility. The consumer is not—and does not want to be—aware of the mechanics of how that service is delivered.

The New Role of the Cloud DBA

In the earlier sections, we talked about application DBAs, operational DBAs, and database engineers. With the introduction of DBaaS, these roles are now spread out and sometimes even decentralized. What we mean is that some aspects of the DBA role must logically move under the umbrella of the application development teams, while others are centralized to a much greater extent:

• The application DBAs work with the application development team very closely. They are the point of integration between the developers and infrastructure team. In other words, they can speak the developers’ language as well as the infrastructure team’s language and bridge that gap.

• The operational DBAs are now part of the centralized group, taking care of the day-to-day care maintenance of the underlying infrastructure for the database.

• The database engineers are the integration point between the application and operational DBAs. They are the ones who work with the application DBAs to define the QoS metrics and SLAs.

In order to understand these roles and how they map into the overall DBaaS platform and concept, we need to define the roles and responsibilities in greater detail.

The Application DBA Role in DBaaS

As we said earlier, the application DBA is a part of the application development team working closely with the developers. To be able to influence the application design and architecture, the application DBA role is now decentralized and embedded within the application development team.

The application DBA’s primary responsibility is to work with the application developers and architects early on and throughout the application development lifecycle:

• Ensure the application development team understands how databases work and uses them appropriately.

• Ensure the application development team understands how the database events can impact an application.

• Work with the application development team to understand how database events and outages are manifested to the application. This understanding enables the application development team to build in mechanisms to gracefully handle and manage these events.

• Work with the application development team to design and build the logical database using the appropriate features to maximize performance.

• Facilitate communications between the application development team and the core DBaaS teams.

• Translate the application development team’s performance, availability, and infrastructure requirements into terms and metrics the DBaaS team can use to properly size, design, and deploy a solution.

• Translate the DBaaS team’s assessments, statements, and findings such that the application development team can understand and use the information appropriately. This information includes projections for storage space, CPU requirements, I/O metrics, uptime, and availability.

In other words, application DBAs are key during the application design and development phase. Their role becomes one of ensuring that both the database and the application in terms of how it uses the database are architected and designed properly. The best way to achieve this goal is for the DBA to be embedded early on in the process and to work with the application teams proactively on potential issues. As a result, the application DBAs work closely with both the application teams and the database engineers on the DBaaS team.

The Operational DBA Role in DBaaS

The operational DBA role still remains pretty much intact and crucial in the new paradigm created by DBaaS. The operational DBA’s primary responsibility in a DBaaS environment is still one of ensuring that the ancillary processes around managing the database are managed and controlled effectively. These responsibilities include things such as the following:

• As the first line of defense for a database, triage and handle database-related alerts.

• Work with the DBaaS team to ensure that the database is being managed to meet and exceed agreed-upon QoS and SLA requirements. For example:

• Ensure database backups are being performed per the defined SLAs.

• Ensure I/O performance is compliant with the stated SLAs.

• Ensure patching and maintenance work is being performed in compliance with the end user availability SLA as well as being communicated to the end user teams.

• Database performance issues occurring due to infrastructure problems are being identified and resolved as per the specified SLAs.

• Communicate with the application development team, especially the application DBA, on application-owned performance bottlenecks and issues. These issues could include, for example, excessive application-level row locking, excessive redo generation, or inefficient use of partitioning leading to query plans that perform an excessive amount of I/O.

• Communicate with the application development team when the database is coming close to utilizing all the assigned resources and needs more. Resource usage could be in terms of CPU utilization, I/O limits, and storage space.

In other words, operational DBAs perform key roles once application databases are promoted into production. They are the primary recipients of the monitoring system’s alerts and thus work very closely with the application DBAs and database engineers to ensure a database is performing as expected or better than expected.

The Database Engineer Role in DBaaS

The database engineer role is not necessarily a new role in terms of function, but it is a new role in terms of its singular focus and attention. Database engineers focus on designing, building, and deploying the overall DBaaS. The primary responsibilities of the database engineer therefore includes the following:

• Define, design, and build the DBaaS tiers based on QoS and SLA requirements.

• Define the solution architecture to deliver databases based on the defined metrics.

• Define and develop the operational tools and procedures required to support the ongoing database maintenance and management.

• Define and develop the tools and dashboards that enable the service consumer to measure and validate service delivery and to manage capacity needs on demand.

• Plan and manage capacity growth and demands at the DBaaS offering level rather than at a single database level. Capacity planning efforts account for storage volume, storage performance, compute capacity, as well as network bandwidth and capacity.

Let us look at a few examples of these tasks, high availability being the first. There are many technologies and methods available to provide for database-level high availability. For example, the most often used technologies include Oracle RAC, Oracle Data Guard, Oracle Golden Gate Replication, storage replication, database sharing, virtualization techniques and features, and so on. One solution stack might be as follows:

• Use Oracle RAC along with Oracle Data Guard to deliver the highest level of availability.

• At the next level, deploy single-instance with Data Guard.

• At the third and lowest level, deploy single-instance databases with nightly backups.

From an I/O performance perspective, the DBaaS could provide tiered storage, starting with a low-end Internet small computer system interface (iSCSI), Network File System (NFS) storage going all the way up to high-end SAN using tiered storage and solid-state drives (SSDs) or flash memory drives.

For capacity and scaling, the solution could either be RAC, wherein compute capacity is added by growing the system horizontally, or it could be based on a virtualization tool, allowing for vertical scaling by adding virtual CPUs, and so on.

Another crucial aspect of the overall DBaaS is database backup. Once again, multiple options and technologies are available for performing database backups. The most common examples are Oracle Recovery Manager (RMAN)-based backups to either active or standby databases and RMAN in conjunction with storage technology such as snapshots and mirror splits. These backups can be written to different locations or devices to meet the backup speeds or RTO objectives. Options for database backup destinations include dedicated backup devices such as the Oracle Zero Data Loss appliance or the Data Domain appliance or cheaper SANs being managed as virtual tape libraries.

Preparing to Be a Cloud DBA

Depending on the organizational structure and whether it is a private cloud or a public cloud, the following roles can be fulfilled by DBAs in different parts of the organization:

• In the case of the public cloud database provider, the database engineer and operational DBA roles are with the cloud provider, but the application DBA is embedded into the application development team.

• In the case of large institutions building a private cloud database that is shared across the entire organization, the same structure is followed.

• In the case of private cloud, these roles may be combined under the same organizational entity as well.

It is important to understand that the DBA organizational structure may morph from one structure to another based on organizational needs and demands. Figure 2.2 represents the DBA roles that exist in the DBaaS-based solution. As stated earlier, the same roles existed before but are now much more visible and distinctly defined.

Figure 2.2. DBA responsibilities in a database cloud model

As a DBA working in the current traditional model who is preparing to move into the DBaaS model, the transition is not about skill sets only. You would obviously need to possess or acquire the specific skills needed to execute your responsibilities within the role. The transition is more about mindset. The transition into DBaaS or database clouds is also about delivering a service and being accountable for it.

The approach to DBaaS is more consultative and democratic than dictatorial in nature. To take it a step further, the very concept of DBaaS is based on free market and demand and supply. The DBaaS provider is providing a service. End users will consume the service if it meets their needs in terms of functionality, cost, ease of use, and more. On the flip side, the provider cannot accept any requests or conditions that it cannot deliver, which leads us to the second mindset change: adaptability and flexibility (within reason).

From a technical perspective, DBAs in the DBaaS paradigm, particularly operational DBAs, still need a lot of the same skills as those in a traditional database environment. Database engineers and application DBAs will need to pick up some ancillary supporting technologies.

Application DBAs are going to work with developers more closely, so they should learn and understand some of the new technologies in the application space, development languages (Java, Ruby, etc.), technologies supporting applications such as application servers, web servers, and even new concepts such as big data and NoSQL.

Database engineers and cloud architects also need to become familiar with technologies and concepts in the infrastructure space, including server, virtualization, storage, and networking technology improvements and changes. These technologies are the underpinning of DBaaS and therefore essential to a successful database cloud service.

Summary

The purpose of this chapter was to help database technologists understand the implications of database cloud services to their environments and jobs. DBaaS is an offshoot of infrastructure as a service and platform as a service, which are well-entrenched and successful technologies.

Cloud DBAs can expect to interact with a larger set of technologies than they have in the past and therefore will need to expand their existing database-related skills.