Factors in a Migration

When you want to move a database that you are running on-premises to a managed service in the cloud, you need to deal with physical, software, and organizational issues. It requires attention to several levels of change:

Physical data movement

You need to get the data itself into the cloud. Data transfer for large databases can be both time-consuming and costly, but there are migration services to help.

Infrastructure compatibility

Check every aspect of the cloud service against your legacy on-premises systems. A case study on the AWS blog offers an interesting example of snags that might occur during a test migration by unexpected software incompatibilities and minor errors. This particular migration was held up for some time just because of differences in SSL implementations by the on-premises systems and the cloud provider.

Database compatibility

The cloud database is likely to differ from your on-premises database in some features that affect migration. Subtleties such as database character sets and permissions on stored procedures can trip you up. A homogeneous migration (such as from one Oracle database to another Oracle database) will probably go more smoothly than a heterogeneous migration (such as from an Oracle database to a PostgreSQL database, or one of the proprietary cloud vendors’ databases). “Checking for Incompatibilities” explains what you can do to reduce friction between databases. For certain kinds of migrations, cloud vendors might provide tools that take some of the work off your shoulders.

If you’re currently running a very old version of a database on-premises, it might be time to upgrade. Instead of merely choosing a newer version of the same relational database management system (RDBMS), which will involve a certain amount of pain, you might use this opportunity to try a heterogeneous migration to a new RDBMS that shows promise for the future.

Organizational changes

People from several teams, especially DBAs and application developers, will need to devote time to learning the cloud tools and managing the new instances while keeping the legacy systems going on-premises until you are ready to move everything to the cloud. You might choose to spend months or even years doing a full migration. You might also keep some of the data (or a copy of the data) permanently on-premises.

Major Migration Tasks

We suggest the following steps for a successful migration.

Create a cloud migration plan

The cloud migration plan should list all the databases and applications in the order in which you want to move them to the cloud. The final assessment plan should delineate the migration plan for all databases. If you need new resources—in software, finance, or personnel—in the new database environment, that should also be in the plan.

Determine who performs the migration

DBAs and developers should work together on the migration effort because each team has something to offer. While choosing staff, you can also decide whether to employ a migration service provided by the cloud vendor or a third-party company.

Educational tasks

These include broad brush tasks such as understanding the use of the cloud vendor’s tools, the specific features of the target database engine, the scope of the database migration, and the architecture of the cloud databases.

Create the cloud database architecture

Select the type of databases that you want to use, as well as whether to use a managed or self-managed database. When making this choice, weigh all relevant factors such as cost, performance, reliability, and scalability.

Choose a migration process

“Migrating the Database” lays out the criteria for choosing among replication, backup/restore, or a dedicated migration service.

For a self-managed database, create your computing infrastructure

If you choose to manage your own databases in the cloud, create virtual instances and deploy them through the vendor’s compute service, such as Amazon Elastic Compute Cloud (Amazon EC2).

Determine opportunities for architectural changes

You might find that choices you made on-premises are no longer right for the new environment in the cloud. For instance, you might be able to consolidate shards.

Set up the cloud database accounts

Set up general accounts for users in the cloud through IAM before you set up the users, roles, and groups in the database.

Upgrade and test applications

You might need to rewrite code to work with the new database. Some of the migration of applications may be automatable, just as with the database itself.

Readiness Assessment

A readiness assessment helps you to estimate the costs, the architecture of the cloud database, migration plans, and the impact of the move to the cloud on compliance regulations. The result of a cloud readiness assessment is a detailed report of your company’s readiness to move its databases to the cloud.

Following are brief descriptions of the key steps during a cloud readiness assessment.

Stakeholder interviews

Talking to application developers, business users, and others concerned with the use of data within your organization team will help you to determine requirements pertaining to performance, high availability, and the features and capabilities of the cloud-based databases.

Analysis of current on-premises databases

Go back over your current databases to determine growth patterns of data, backup and recovery strategies, ongoing data exports and imports, and so on. Understanding the current database usage patterns will help you when it’s time to decide what databases to use in the cloud, and whether you’ll need a particular database offering in order to get the capabilities you need.

Prioritization of the databases to move

Pick the databases that you would like to move to the cloud first. One important criterion is the extent of changes to applications that will be required, which in turn calls for coordination with developers.

Migration cost analysis

Perform a thorough analysis of the costs of migrating to the cloud versus staying in the on-premises datacenters so that you fully understand the TCO and ROI of the cloud migration.

Lowering costs by using a cloud database service might be one of your key goals. But a poor choice of cloud services will defeat this purpose.

Security and compliance

Sometimes, you need special regions or AZs to comply with standards such as Service Organization Control 2 (SOC 2), PCI DSS, or the US HIPAA. Fortunately, cloud providers often match particular regions and AZs to these legal requirements. In addition, there are specialized regions, such as AWS’s GovCloud (US), which is an isolated AWS region subject to FedRAMP High and Moderate baselines. Finally, you need to check whether you can meet your organization’s service-level agreements (SLAs) in your chosen cloud setting. These SLAs typically include metrics for planned maintenance, backups, recovery point objectives (RPOs), and recovery time objectives (RTOs).

If you’ve done your cloud assessment correctly, you should also have a good idea of the scope of your cloud migration. Some of your applications might be so old that they would need to be completely redesigned to move to the cloud. If you don’t currently have the resources to rewrite the applications, you can keep them in on-premises databases or move the entire servers into the cloud.

Migration steps

You should plan exactly when and how you will do the data migration, along with preparations for rolling back the change if later testing shows that the database is not working properly. Establish tests and the success criteria for each step.

Application changes

You need to do this in coordination with the development team. This includes scheduling of the application changes into the development process and testing.

Automating the migration

This is valuable because it lets you repeat the migration as often as you need to, fixing errors as you encounter them or changing parameters. You also can adapt and reuse the automation framework for further migrations.

Checking for Incompatibilities

If you are using the same RDBMS (such as MySQL, for instance) for both databases, check the versions to determine whether you are using features that are unsupported on the cloud version. These checks are obviously even more likely to turn up incompatibilities if you are moving to a new database engine.

The conversion issues that you encounter when performing a heterogeneous migration to a cloud database are the same as when you’re performing such a migration on-premises; however, you have access to a wider range of solutions.

The database vendors themselves generally offer tools that help move data to a cloud database of the same type. For example, Oracle comes with data migration tools such as Datapump, RMAN, and SQL Developer that help you to move your Oracle databases to the cloud. In addition, cloud providers and third parties offer services such as AWS’s Schema Conversion Tool to move schemas. These tell you whether the schemas can be translated, and provide suggestions for workarounds if necessary.

Still, most heterogeneous database migrations require both automated tools and manual action by the DBA. If you’re moving to an AWS database, you can draw on specific recommendations on the AWS website.

Some migration tools employ a translator to convert one database’s objects, such as Oracle stored procedures, to a non-Oracle database such as PostgreSQL. These tools, however, might do an incomplete job, because each database engine employs unique coding practices.

This is the reason for performing stringent tests on the target database for accuracy and performance after migrating to the cloud. Some programming techniques employed by a specific RDBMS might not have an exact equivalent in the target database. The migration tool might flag these types of conversion issues during the migration process. In many cases, it can also show you the code that you might need to run to overcome these issues and successfully complete the database migration.

Migrating the Database

Many cloud providers and third parties offer migration services, such as Microsoft Azure Database Migration Service and AWS Database Migration Service, that are well worth consideration. These are advertised as being fast, smooth, and easy. However, you might prefer to use existing processes with which you are familiar, such as restoring from a backup to a new database environment, and using replication.

If you use a database migration service, you can keep your source databases fully operational during the migration to the cloud and minimize the downtime for applications that rely on the database. This facilitates the key goal of minimal downtime, cited by many organizations migrating to the cloud. Figure 3-1 illustrates how AWS Database Migration Service creates the tables, loads data, and keeps the tables synchronized with the source database tables.

Figure 3-1. Keeping applications up during the database migration

The migration process usually revolves around a typical backup/restore process. The tasks depend on whether you can afford to take the database down during the transfer. If you must leave the database running, here’s the basic sequence to follow:

1. Do a hot backup of the original database while it is running.

2. Copy and restore the backup in the cloud.

3. Enable replication to incorporate into the cloud version any changes made to the original database after you created the backup.

Of course, backup/restore is simpler if you can take the database out of operation. Migration then follows this pattern:

1. Shut down the original database.

2. Do a backup.

3. Copy and restore the backup in the cloud.

4. Start the new version in the cloud.

But you are likely to spend time testing the new cloud version before you are ready to put it into production to make sure that applications work, that security is in place, and that you are getting the performance you need. So you still might want to restart the original version before you’re ready to trust the new version, and then you’ll use replication to synchronize the new one with the original one.

Migrating Applications

There are two basic strategies for moving your databases and related applications:

Lift and shift

Move an entire database as is, with all of the applications it supports, to the new database in the cloud. This is the “old wine in a new bottle” application migration strategy given that you make little or no use of cloud native features.

Redesign the databases

Rethink how your on-premises databases deal with administrative tasks such as scaling and high availability, and investigate the cloud provider’s offerings in those areas. They are getting more and more sophisticated. More work and effort is required for this strategy, of course. You might also need to recode the applications to adjust to the changes in database strategy.

Often, customers start with “lift and shift” and follow up with redesigning and other modernizations.

Post-Migration Checks

After the data is completely copied over to the cloud database, validate the target database to ensure that all of the database objects are present on the cloud database. Several tests determine whether the migration was successful:

Validate data

This can be as simple as checking the number of rows, or running a checksum to indicate that there was no loss or corruption. More complex validation processes factor in the schema changes inherent in heterogenous migrations. Unfortunately, this basic validation is often forgotten.

Basic functionality

Perform end-to-end testing to ensure that you performed the migration successfully and that the system is operating as it is supposed to. Test the use of features that tend to differ between database engines and versions, such as triggers and stored procedures.

Performance testing

Stress-test the new systems and benchmark their performance to get an idea of where you can enhance response time and throughput.

Security assessment

Ensure that the cloud systems are secure by performing vulnerability assessments and penetration tests.

These steps also form a transition to the next phase, optimization. This phase follows the successful migration of your on-premises databases and applications to the cloud. The crucial elements in this phase are performance management and cost optimization. Careful, continuous monitoring of the new systems is critical so that you can quickly revert to the old systems in case you run into unexpected glitches.

Availability

This, of course, is the fundamental requirement that you must meet in any setting. Moving to the cloud, fortunately, removes some causes of failure (if you properly configure the restarting of failed services). Furthermore, the cloud vendor provides numerous tools for predicting failures and alerting you to failures. To take advantage of these, carry out the planning in this section (many of these activities are also useful for performance optimization):

Disaster recovery (DR) plan

With a large and robust cloud vendor, you can fail over to a new AZ if the current one fails. But you need both a plan and an automated process for failover. The plan should address the RTO, RPO, and geographic redundancy. As much as possible, exploit the failover capabilities provided by the vendor instead of handcrafting your own.

Logs and system monitoring

Determine the events that can indicate imminent problems as well as problems that have already occurred. These can be incorporated into automated tracking. The tracking should convey enough information to tell you the source of failure: for instance, whether they stem from a user action such as a reboot of a service, from an attack, or from other changes in the environment. Some failures can be considered normal and can be addressed by your automated tools; these should be recorded but do not need to issue alerts to the administrator.

Change monitoring

Administrators should always know what changes to database configuration, instance sizing, or cluster topology can affect availability. Modern development environments use robust processes for change tracking and version control so that every change goes through a vetting process and can be reversed.

System testing

Try to determine the weak points in your system and anticipate failure. Some teams go through “pre-mortem” exercises to identify and remove potential sources of failure. Large sites can afford to bring down systems deliberately and watch whether recovery is adequate; this kind of testing, called chaos engineering, was popularized through Netflix’s Chaos Monkey. Just as you do regular restores to make sure backups are working, you should test your recovery procedures.

Performance Optimization

Performance can benefit from the practices in the preceding section, notably monitoring. Performance monitoring should allow you to determine the relationships between events and changes in the database metrics, as well as to see discrepancies between the predicted and actual performance trends. Performance can additionally be maintained and improved through additional processes.

Workload testing

Growth in the size and complexity of data, as well as application behavior changes, affect performance. Test performance regularly so that you learn of degradation before your customers tell you about it; you can then scale or make other changes to adapt. It can take a while for the database cache and table statistics to update after a major change, which means that query plans and overall performance might take a while to return to desired levels.

Distributed optimization

In fast-changing application environments, the developers understand the database queries better than the DBA. If they take on query testing and optimization, they can achieve good performance faster than the DBA can, freeing the DBA to do long-term optimizations and planning.

Knowing the importance of performance, cloud vendors have inserted traces and exposed a large number of statistics to users. AWS’s RDS Performance Insights, for instance, lets you pull up historical data on a dashboard and view charts showing the load placed by particular operations, users, or SQL statements on the database. You can consult the dashboards to answer simple questions such as “When are my hosts overloaded?” or more complex ones such as “What is making this query produce suboptimal performance?” Azure offers both performance monitoring and automatic tuning. Third-party vendors also produce tools for monitoring and performance in the cloud.

Adapting to Differences in the Cloud

Finally, the vendor themselves will offer new services, new VM options, new hardware (such as SSDs), and other changes that can offer you big benefits. Always be aware of what the vendor can do for you, so that you can offload work onto its well-tested and standardized processes. As you do so, update your documented operational procedures. Finally, you can report abnormal events to the vendor’s customer support, indicating the urgency of the event.