© Moritz Lenz 2019
Moritz LenzPython Continuous Integration and Deliveryhttps://doi.org/10.1007/978-1-4842-4281-0_13

13. State Management

Moritz Lenz1 
(1)
Fürth, Bayern, Germany
 

Continuous delivery (CD) is nice and easy for a stateless application, that is, for an application that does not have data persistently stored. Installing a new application version is a simple task, which just requires the installation of the new binaries (or sources, in case of a language that’s not compiled), stopping the old instance, and starting a new instance.

As soon as there is persistent state to consider, things become more complicated. Here, I will consider traditional relational databases with schemas. You can avoid some problems by using a schema-less “noSQL” database, but you don’t always have that luxury. If you do go schema-less, you have to deal with older data structures inside the application code, not through the deployment process.

Along with the schema changes, you might have to consider data migrations, which might involve such things as filling out missing values with a default or importing data from a different data source. In general, such data migrations fit the same pattern as schema migrations, which is to execute either a piece of SQL and data definition language (DDL) 1 or run an external command that directly talks to the database.

13.1 Synchronization Between Code and Database Versions

State management is difficult, because code is usually tied to a version of the database schema. There are several cases in which this can cause problems.

  • Database changes are often slower than application updates. If version 1 of your application can only deal with version 1 of the schema, and version 2 of the application can only deal with version 2 of the schema, you have to stop version 1 of the application, do the database upgrade, and start up version 2 of the application only after the database migration has finished.

  • Rollbacks to a previous version of an application, and thus its database schema version, become painful. Typically, either a database change or its rollback can lose data, so you cannot easily do an automated release and rollback over these boundaries.

To elaborate on the last point, consider the case in which a column is added to a table in the database. In this case, the rollback of the change (deleting the column again) loses data. Conversely, if the original change is to delete a column, that step usually cannot be reversed. You can re-create a column of the same type, but the data is lost. Even if you archive the deleted column data, new rows might have been added to the table, and there is no archived data for these new rows.

13.2 Decoupling Application and Database Versions

There is tooling that can help you get your database schema into a defined state reproducibly, but it does not solve the problem of potential data loss through rollbacks for you. The only practical approach is to establish collaboration between the application developers and the database administrators and break up problematic changes into multiple steps.

Suppose your desired change is to drop a column that has a NOT NULL constraint . Simply dropping the column in one step comes with the problems outlined in the previous section. Instead, you might be able to do the following steps:
  1. 1.

    Deploy an application version that can deal with reading NULL values from the column, even though NULL values are not yet allowed.

     
  2. 2.

    Wait until you’re sure you don’t want to roll back to an application value that cannot deal with NULL values.

     
  3. 3.

    Deploy a database change that makes the column nullable (or give it a default value).

     
  4. 4.

    Wait until you’re sure you don’t want to roll back to a schema version where this column is NOT NULL.

     
  5. 5.

    Deploy a new version of the application that doesn’t use the column anymore.

     
  6. 6.

    Wait until you’re sure you don’t want to roll back to a version of your application that uses this column.

     
  7. 7.

    Deploy a database change that drops the column entirely.

     
Some scenarios allow you to skip some of these steps or fold multiple steps into one. Adding a column to a table is a similar process, as follows:
  1. 1.

    Deploy a database change that adds the new column with a default value (or allows NULL values).

     
  2. 2.

    Deploy a version of the application that writes to the new column.

     
  3. 3.

    Optionally run some migrations that fill the column for old rows.

     
  4. 4.

    Optionally deploy a database change that adds constraints (like NOT NULL) that weren’t possible at the start.

    … with the appropriate waits between the steps.

     

Example of a Schema Change

Suppose you have a web application backed by a PostgreSQL database and, currently, the application logs login attempts into the database. So, the schema looks like this:
CREATE TABLE users (
    id          SERIAL,
    email       VARCHAR NOT NULL,
    PRIMARY KEY(id)
);
CREATE TABLE login_attempts (
    id          SERIAL,
    user_id     INTEGER NOT NULL REFERENCES users (id),
    success     BOOLEAN NOT NULL,
    timestamp   TIMESTAMP NOT NULL DEFAULT NOW(),
    source_ip   VARCHAR NOT NULL,
    PRIMARY KEY(id)
);

As the load on the web application increases, you realize that you are creating unnecessary write load for the database and start logging to an external log service. The only thing you really require in the database is the date and time of the last successful login (which your CEO insists you show on each login, because an auditor was convinced it would improve security).

So, the schema you want to end up with is this:
CREATE TABLE users (
    id          SERIAL,
    email       VARCHAR NOT NULL,
    last_login  TIMESTAMP NOT NULL,
    PRIMARY KEY(id)
);
A direct database change script to get there would be
DROP TABLE login_attempts;
ALTER TABLE users
    ADD COLUMN last_login TIMESTAMP NOT NULL;

but that suffers from the problem previously outlined that it ties the schema version to the application version, but also that you cannot introduce a NOT NULL column without a default and without supplying values for it.

Let’s break it down into separate steps that don’t suffer from these problems.

Creating the New Column, NULLable

The first step is to add the new column, users.last_login , as optional (by allowing NULL values). If the starting point was version 1 of the schema, this is version 2:
CREATE TABLE users (
    id          SERIAL,
    email       VARCHAR NOT NULL,
    last_login  TIMESTAMP,
    PRIMARY KEY(id)
);
-- table login_attempts omitted, because it's unchanged.
Running apgdiff, Another PostgreSQL Diff Tool,2 against the two scheme files gives us:
$ apgdiff schma-1.sql schema-2.sql
ALTER TABLE users
    ADD COLUMN last_login TIMESTAMP;

which is the forward migration script from schema 1 to schema 2. Note that we don’t necessarily need a rollback script, because every application version that can deal with version 1 of the schema can also deal with schema version 2 (unless the application does something stupid like SELECT * FROM users and expects a certain number or order of results. I’ll assume the application isn’t that stupid).

This migration script can be applied to the database while the web application is running, without any downtime.

../images/456760_1_En_13_Chapter/456760_1_En_13_Figa_HTML.jpg MySQL has the unfortunate property that schema changes are not transactional and they lock the whole table during the schema changes, which negates some advantages you gain from incremental database updates.

To mitigate this, there are some external tools that work around this by creating a modified copy of the table, gradually copying the data from the old to the new table, then finally doing a rename to replace the old table. One such tool is gh-ost 3 by GitHub.

These tools typically come with only limited support for foreign key constraints, so evaluate them carefully before using them.

When the schema change has finished, you can deploy a new version of the web application that writes to users.last_login whenever a successful login occurs. Note that this application version must be able to deal with reading NULL values from this column, for example, by falling back to table login_attempts, to determine the last login attempt.

This application version can also stop inserting new entries into table login_attempts . A more conservative approach is to defer that step for a while, so that you can safely roll back to an older application version.

Data Migration

In the end, users.last_login is meant to be NOT NULL, so you have to generate values for where it’s NULL. Here, table last_login is a source for such data.
UPDATE users
  SET last_login = (
         SELECT login_attempts.timestamp
           FROM login_attempts
          WHERE login_attempts.user_id = users.id
            AND login_attempts.success
       ORDER BY login_attempts.timestamp DESC
          LIMIT 1
       )
 WHERE users.last_login IS NULL;
If NULL values remain, say, because a user never logged in successfully, or because table last_login doesn’t go back far enough, you must have some fallback, which could be a fixed value. Here, I’m taking the easy road and simply using NOW() as the fallback.
UPDATE users SET last_login = NOW() WHERE last_login IS NULL;

These two updates can again run in the background, while the application is running. After this update, no further NULL values should show up in users.last_login. After waiting a few days, and verifying that this is indeed the case, it’s time to apply the necessary constraint.

Applying Constraints, Cleaning Up

Once you are confident that there are no rows that miss values in the column last_login, and that you aren’t going to roll back to an application version that introduces missing values, you can deploy an application version that stops using table login_attempts, dispose of the table login_attempts , and then apply the NOT NULL constraint (see also Figure 13-1).
DROP TABLE login_attempts;
ALTER TABLE users
    ALTER COLUMN last_login SET NOT NULL;
../images/456760_1_En_13_Chapter/456760_1_En_13_Fig1_HTML.png
Figure 13-1

Sequence of application and database update steps. Each database version is compatible with the application versions before and after it, and vice versa.

In summary, a single logical database change has been spread over three database updates (two schema updates and one data migration) and two application updates.

This makes application development a bit more of an effort, but you gain operational advantages. One of these advantages is keeping the application releasable to production at all times.

Prerequisites

If you deploy a single logical database change in several steps, you must do several deployments, instead of one big deployment that introduces both code and schema changes at once. That’s only practical if the deployments are (at least mostly) automated, and if the organization offers enough continuity that you can actually finish the change process.

If the developers are constantly putting out fires, chances are they never get around to adding that final desired NOT NULL constraint , and some undiscovered bug will lead to missing information later down the road.

You should also set up some kind of issue tracker with which you can trace the path of schema migrations, to make sure that none remains unfinished, for example, in the case of a developer leaving the company.

Tooling

Unfortunately, I know of no tooling that fully supports the intertwined database and application release cycle that I outlined. There are tools that manage schema changes in general. For example, Sqitch4 and Flyway5 are rather general frameworks for managing database changes and rollbacks.

On the lower level, there are tools such as apgdiff that compare the old and new schemas and use that comparison to generate DDL statements that bring you from one version to the next. Such automatically generated DDLs can form the basis of the upgrade scripts that Sqitch or Flyway then manage.

Some ORMs also come with frameworks that promise to manage schema migrations for you. Carefully evaluate whether they allow rollbacks without losing data.

Structure

If you decouple application deployments from schema deployments, it follows that you must have at least two separately deployable packages: one for the application and one for the database schema and schema migration scripts. If you want or have to support rollbacks of database schemas, you must remember that you need the metadata associated with the new schema to be able to roll back to the old version.

The database description for version 5 of the schema doesn’t know how to roll back from version 6 to version 5, because it knows nothing about version 6. So, you should always keep the newest version of the schema file package installed and separate the installed version from the currently active database version. The tooling that controls the schema migrations can be independent of the application and its schema, and so should live in a third software package.

No Silver Bullet

There is no single solution that manages all your data migrations automatically for you during your deployments. You have to carefully engineer the application and database changes to decouple and deploy them separately. This is typically more work on the application development side, but it buys you the ability to deploy and roll back without being blocked by database changes.

Tooling is available for some pieces but typically not for the big picture. Someone has to keep track of the application and schema versions—or automate them.

13.3 Summary

State held in a database can complicate application upgrades.

Incompatible data structure and schema changes can be broken up into several smaller steps, each of which is compatible with the previous one.

This allows application upgrades without downtime, at the cost of having to do several application and schema deployments.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset