Virtualization technologies have been around since the days of the IBM mainframes. Most people have not had a chance to work on a mainframe, but we are sure some readers of this book remember the days when they had to set up or use a bare-metal server from a manufacturer such as HP or Dell. These manufacturers are still around today, and you can still use bare-metal servers hosted in a colocation facility, like in the good old days of the dot-com era.
When most people think of virtualization, however, they do not automatically have a mainframe in mind. Instead, they most likely imagine a virtual machine (VM) running a guest operating system (OS) such as Fedora or Ubuntu on top of a hypervisor such as VMware ESX or Citrix/Xen. The big advantage of VMs over regular bare-metal servers is that by using VMs, you can optimize the server’s resources (CPU, memory, disk) by splitting them across several virtual machines. You can also run several operating systems, each in its own VM, on top of one shared bare-metal server, instead of buying a dedicated server per targeted OS. Cloud computing services such as Amazon EC2 would not have been possible without hypervisors and virtual machines. This type of virtualization can be called kernel-level because each virtual machine runs its own OS kernel.
In the never-ending quest for more bang for their buck, people realized that virtual machines were still wasteful in terms of resources. The next logical step was to isolate an individual application into its own virtual environment. This was achieved by running containers within the same OS kernel. In this case, they were isolated at the file-system level. Linux containers (LXC) and Sun Solaris zones were early examples of such technologies. Their disadvantage was that they were hard to use and were tightly coupled to the OS they were running on. The big breakthrough in container usage came when Docker started to offer an easy way to manage and run filesystem-level containers.
A Docker container encapsulates an application together with other software packages and libraries it requires to run. People sometimes use the terms Docker container and Docker image interchangeably, but there is a difference. The filesystem-level object that encapsulates the application is called a Docker image. When you run the image, it becomes a Docker container.
You can run many Docker containers, all using the same OS kernel. The only requirement is that you must install a server-side component called the Docker engine or the Docker daemon on the host where you want to run the containers. In this way, the host resources can be split and utilized in a more granular way across the containers, giving you more bang for your buck.
Docker containers provide more isolation and resource control than regular Linux processes, but provide less than full-fledged virtual machines would. To achieve these properties of isolation and resource control, the Docker engine makes use of Linux kernel features such as namespaces, control groups (or cgroups), and Union File Systems (UnionFS).
The main advantage of Docker containers is portability. Once you create a Docker image, you can run it as a Docker container on any host OS where the Docker server-side daemon is available. These days, all the major operating systems run the Docker daemon: Linux, Windows, and macOS.
All this can sound too theoretical, so it is time for some concrete examples.
Since this is a book on Python and DevOps, we will take the canonical Flask “Hello World” as the first example of an application that runs in a Docker container. The examples shown in this section use the Docker for Mac package. Subsequent sections will show how to install Docker on Linux.
Here is the main file of the Flask application:
$
cat
app
.
py
from
flask
import
Flask
app
=
Flask
(
__name__
)
@app.route
(
'/'
)
def
hello_world
():
return
'Hello, World! (from a Docker container)'
if
__name__
==
'__main__'
:
app
.
run
(
debug
=
True
,
host
=
'0.0.0.0'
)
We also need a requirements file that specifies the version of the Flask package to be installed with pip
:
$ cat requirements.txt Flask==1.0.2
Trying to run the app.py file directly with Python on a macOS laptop without first installing the requirements results in an error:
$ python app.py Traceback (most recent call last): File "app.py", line 1, in <module> from flask import Flask ImportError: No module named flask
One obvious way to get past this issue is to install the requirements with pip
on your local machine. This would make everything specific to the operating system you are running locally. What if the application needs to be deployed on a server running a different OS? The well-known issue of “works on my machine” could arise, where everything works beautifully on a macOS laptop, but for some mysterious reason, usually related to OS-specific versions of Python libraries, everything breaks on the staging or production servers running other operating systems, such as Ubuntu or Red Hat Linux.
Docker offers an elegant solution to this conundrum. We can still do our development locally, using our beloved editors and toolchains, but we package our application’s dependencies inside a portable Docker container.
Here is the Dockerfile describing the Docker image that is going to be built:
$
cat DockerfileFROM
python:3.7.3-alpine
ENV
APP_HOME /app
WORKDIR
$APP_HOME
COPY requirements.txt .RUN
pip install -r requirements.txtENTRYPOINT
[ "python" ]
CMD
[ "app.py" ]
A few notes about this Dockerfile:
Use a prebuilt Docker image for Python 3.7.3 based on the Alpine distribution that produces slimmer Docker images; this Docker image already contains executables such as python
and pip
.
Install the required packages with pip
.
Specify an ENTRYPOINT and a CMD. The difference between the two is that when the Docker container runs the image built from this Dockerfile, the program it runs is the ENTRYPOINT, followed by any arguments specified in CMD; in this case, it will run python app.py
.
If you do not specify an ENTRYPOINT in your Dockerfile, the following default will be used: /bin/sh -c
.
To create the Docker image for this application, run docker build
:
$ docker build -t hello-world-docker .
To verify that the Docker image was saved locally, run docker images
followed by the name of the image:
$ docker images hello-world-docker REPOSITORY TAG IMAGE ID CREATED SIZE hello-world-docker latest dbd84c229002 2 minutes ago 97.7MB
To run the Docker image as a Docker container, use the docker run
command:
$ docker run --rm -d -v `pwd`:/app -p 5000:5000 hello-world-docker c879295baa26d9dff1473460bab810cbf6071c53183890232971d1b473910602
A few notes about the docker run
command arguments:
The --rm
argument tells the Docker server to remove this container once it stops running. This is useful to prevent old containers from clogging the local filesystem.
The -d
argument tells the Docker server to run this container in the background.
The -v
argument specifies that the current directory (pwd) is mapped to the /app directory inside the Docker container. This is essential for the local development workflow we want to achieve because it enables us to edit the application files locally and have them be auto-reloaded by the Flask development server running inside the container.
The -p 5000:5000
argument maps the first port (5000) locally to the second port (5000) inside the container.
To list running containers, run docker ps
and note the container ID because it will be used in other docker
commands:
$ docker ps CONTAINER ID IMAGE COMMAND CREATED c879295baa26 hello-world-docker:latest "python app.py" 4 seconds ago STATUS PORTS NAMES Up 2 seconds 0.0.0.0:5000->5000/tcp flamboyant_germain
To inspect the logs for a given container, run docker logs
and specify the container name or ID:
$ docker logs c879295baa26 * Serving Flask app "app" (lazy loading) * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) * Restarting with stat * Debugger is active! * Debugger PIN: 647-161-014
Hit the endpoint URL with curl
to verify that the application works. Because port 5000 of the application running inside the Docker container was mapped to port 5000 on the local machine with the -p
command-line flag, you can use the local IP address 127.0.0.1 with port 5000 as the endpoint for the application.
$ curl http://127.0.0.1:5000 Hello, World! (from a Docker container)%
Now modify the code in app.py with your favorite editor. Change the greeting text to Hello, World! (from a Docker container with modified code). Save app.py and notice lines similar to these in the Docker container logs:
* Detected change in '/app/app.py', reloading * Restarting with stat * Debugger is active! * Debugger PIN: 647-161-014
This shows that the Flask development server running inside the container has detected the change in app.py and has reloaded the application.
Hitting the application endpoint with curl
will show the modified greeting:
$ curl http://127.0.0.1:5000 Hello, World! (from a Docker container with modified code)%
To stop a running container, run docker stop
or docker kill
and specify the container ID as the argument:
$ docker stop c879295baa26 c879295baa26
To delete a Docker image from local disk, run docker rmi
:
$ docker rmi hello-world-docker Untagged: hello-world-docker:latest Deleted:sha256:dbd84c229002950550334224b4b42aba948ce450320a4d8388fa253348126402 Deleted:sha256:6a8f3db7658520a1654cc6abee8eafb463a72ddc3aa25f35ac0c5b1eccdf75cd Deleted:sha256:aee7c3304ef6ff620956850e0b6e6b1a5a5828b58334c1b82b1a1c21afa8651f Deleted:sha256:dca8a433d31fa06ab72af63ae23952ff27b702186de8cbea51cdea579f9221e8 Deleted:sha256:cb9d58c66b63059f39d2e70f05916fe466e5c99af919b425aa602091c943d424 Deleted:sha256:f0534bdca48bfded3c772c67489f139d1cab72d44a19c5972ed2cd09151564c1
This output shows the different filesystem layers comprising a Docker image. When the image is removed, the layers are deleted as well. Consult the Docker storage drivers documentation for more details on how Docker uses filesystem layers to build its images.
Once you have a Docker image built locally, you can publish it to what is called a Docker registry. There are several public registries to choose from, and for this example we will use Docker Hub. The purpose of these registries is to allow people and organizations to share pre-built Docker images that can be reused across different machines and operating systems.
First, create a free account on Docker Hub and then create a repository, either public or private. We created a private repository called flask-hello-world
under our griggheo
Docker Hub account.
Then, at the command line, run docker login
and specify the email and password for your account. At this point, you can interact with Docker Hub via the docker
client.
Before showing you how to publish your locally built Docker image to Docker Hub, we want to point out that best practice is to tag your image with a unique tag. If you don’t tag it specifically, the image will be tagged as latest
by default. Pushing a new image version with no tag will move the latest
tag to the newest image version. When using a Docker image, if you do not specify the exact tag you need, you will get the latest
version of the image, which might contain modifications and updates that might break your dependencies. As always, the principle of least surprise should apply: you should use tags both when pushing images to a registry, and when referring to images in a Dockerfile. That being said, you can also tag your desired version of the image as latest
so that people who are interested in the latest and greatest can use it without specifying a tag.
When building the Docker image in the previous section, it was automatically tagged as latest
, and the repository was set to the name of the image, signifying that the image is local:
$ docker images hello-world-docker REPOSITORY TAG IMAGE ID CREATED SIZE hello-world-docker latest dbd84c229002 2 minutes ago 97.7MB
To tag a Docker image, run docker tag
:
$ docker tag hello-world-docker hello-world-docker:v1
Now you can see both tags for the hello-world-docker
image:
$ docker images hello-world-docker REPOSITORY TAG IMAGE ID CREATED SIZE hello-world-docker latest dbd84c229002 2 minutes ago 97.7MB hello-world-docker v1 89bd38cb198f 42 seconds ago 97.7MB
Before you can publish the hello-world-docker
image to Docker Hub, you also need to tag it with the Docker Hub repository name, which contains your username or your organization name. In our case, this repository is griggheo/hello-world-docker
:
$ docker tag hello-world-docker:latest griggheo/hello-world-docker:latest $ docker tag hello-world-docker:v1 griggheo/hello-world-docker:v1
Publish both image tags to Docker Hub with docker push
:
$ docker push griggheo/hello-world-docker:latest $ docker push griggheo/hello-world-docker:v1
If you followed along, you should now be able to see your Docker image published with both tags to the Docker Hub repository you created under your account.
Now that the Docker image is published to Docker Hub, we are ready to show off the portability of Docker by running a container based on the published image on a different host. The scenario considered here is that of collaborating with a colleague who doesn’t have macOS but likes to develop on a laptop running Fedora. The scenario includes checking out the application code and modifying it.
Launch an EC2 instance in AWS based on the Linux 2 AMI, which is based on RedHat/CentOS/Fedora, and then install the Docker engine. Add the default user on the EC2 Linux AMI, called ec2-user
, to the docker
group so it can run docker
client commands:
$ sudo yum update -y $ sudo amazon-linux-extras install docker $ sudo service docker start $ sudo usermod -a -G docker ec2-user
Make sure to check out the application code on the remote EC2 instance. In this case, the code consists only of app.py file.
Next, run the Docker container based on the image published to Docker Hub.
The only difference is that the image used as an argument to the docker run
command
was griggheo/hello-world-docker:v1
instead of simply hello-world-docker
.
Run docker login
, then:
$ docker run --rm -d -v `pwd`:/app -p 5000:5000 griggheo/hello-world-docker:v1 Unable to find image 'griggheo/hello-world-docker:v1' locally v1: Pulling from griggheo/hello-world-docker 921b31ab772b: Already exists 1a0c422ed526: Already exists ec0818a7bbe4: Already exists b53197ee35ff: Already exists 8b25717b4dbf: Already exists d997915c3f9c: Pull complete f1fd8d3cc5a4: Pull complete 10b64b1c3b21: Pull complete Digest: sha256:af8b74f27a0506a0c4a30255f7ff563c9bf858735baa610fda2a2f638ccfe36d Status: Downloaded newer image for griggheo/hello-world-docker:v1 9d67dc321ffb49e5e73a455bd80c55c5f09febc4f2d57112303d2b27c4c6da6a
Note that the Docker engine on the EC2 instance recognizes that it does not have the Docker image locally, so it downloads it from Docker Hub, then runs a container based on the newly downloaded image.
At this point, access to port 5000 was granted by adding a rule to the security group associated with the EC2 instance. Visit http://54.187.189.51:50001 (with 54.187.189.51 being the external IP of the EC2 instance) and see the greeting Hello, World! (from a Docker container with modified code).
When modifying the application code on the remote EC2 instance, the Flask server running inside the Docker container will auto-reload the modified code. Change the greeting to Hello, World! (from a Docker container on an EC2 Linux 2 AMI instance) and notice that the Flask server reloaded the application by inspecting the logs of the Docker container:
[ec2-user@ip-10-0-0-111 hello-world-docker]$ docker ps CONTAINER ID IMAGE COMMAND CREATED 9d67dc321ffb griggheo/hello-world-docker:v1 "python app.py" 3 minutes ago STATUS PORTS NAMES Up 3 minutes 0.0.0.0:5000->5000/tcp heuristic_roentgen [ec2-user@ip-10-0-0-111 hello-world-docker]$ docker logs 9d67dc321ffb * Serving Flask app "app" (lazy loading) * Debug mode: on * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) * Restarting with stat * Debugger is active! * Debugger PIN: 306-476-204 72.203.107.13 - - [19/Aug/2019 04:43:34] "GET / HTTP/1.1" 200 - 72.203.107.13 - - [19/Aug/2019 04:43:35] "GET /favicon.ico HTTP/1.1" 404 - * Detected change in '/app/app.py', reloading * Restarting with stat * Debugger is active! * Debugger PIN: 306-476-204
Hitting http://54.187.189.51:50002 now shows the new greeting Hello, World! (from a Docker container on an EC2 Linux 2 AMI instance).
It is worth noting that we did not have to install anything related to Python or Flask to get our application to run. By simply running our application inside a container, we were able to take advantage of the portability of Docker. It is not for nothing that Docker chose the name “container” to popularize its technology—one inspiration was how the shipping container revolutionized the global transportation industry.
Read “Production-ready Docker images” by Itamar Turner-Trauring for an extensive collection of articles on Docker container packaging for Python applications.
In this section we will use the “Flask By Example” tutorial that describes how to build a Flask application that calculates word-frequency pairs based on the text from a given URL.
Start by cloning the Flask By Example GitHub repository:
$ git clone https://github.com/realpython/flask-by-example.git
We will use compose
to run multiple Docker containers representing the different parts of the example application. With Compose, you use a YAML file to define and configure the services comprising an application, then you use the docker-compose
command-line utility to create, start, and stop these services that will run as Docker containers.
The first dependency to consider for the example application is PostgreSQL, as described in Part 2 of the tutorial.
Here is how to run PostgreSQL in a Docker container inside a docker-compose.yaml file:
$ cat docker-compose.yaml
version
:
"3"
services
:
db
:
image
:
"postgres:11"
container_name
:
"postgres"
ports
:
-
"5432:5432"
volumes
:
-
dbdata:/var/lib/postgresql/data
volumes
:
dbdata
:
A few things to note about this file:
Define a service called db
based on the postgres:11
image published on Docker Hub.
Specify a port mapping from local port 5432 to the container port 5432.
Specify a Docker volume for the directory where PostgreSQL stores its data, which is /var/lib/postgresql/data. This is so that the data stored in PostgreSQL will persist across restarts of the container.
The docker-compose
utility is not part of the Docker engine, so it needs to be installed separately. See the official documentation for instructions on installing it on various operating systems.
To bring up the db
service defined in docker-compose.yaml, run the docker-compose up -d db
command, which will launch the Docker container for the db
service in the background (the -d
flag):
$ docker-compose up -d db Creating postgres ... done
Inspect the logs for the db
service with the docker-compose logs db
command:
$ docker-compose logs db Creating volume "flask-by-example_dbdata" with default driver Pulling db (postgres:11)... 11: Pulling from library/postgres Creating postgres ... done Attaching to postgres postgres | PostgreSQL init process complete; ready for start up. postgres | postgres | 2019-07-11 21:50:20.987 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 postgres | 2019-07-11 21:50:20.987 UTC [1] LOG: listening on IPv6 address "::", port 5432 postgres | 2019-07-11 21:50:20.993 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres | 2019-07-11 21:50:21.009 UTC [51] LOG: database system was shut down at 2019-07-11 21:50:20 UTC postgres | 2019-07-11 21:50:21.014 UTC [1] LOG: database system is ready to accept connections
Running docker ps
shows the container running the PostgreSQL database:
$ docker ps dCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 83b54ab10099 postgres:11 "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 0.0.0.0:5432->5432/tcp postgres
Running docker volume ls
shows the dbdata
Docker volume mounted for the PostgreSQL /var/lib/postgresql/data directory:
$ docker volume ls | grep dbdata local flask-by-example_dbdata
To connect to the PostgreSQL database running in the Docker container associated with the db
service, run the command docker-compose exec db
and pass it the command line psql -U postgres
:
$ docker-compose exec db psql -U postgres psql (11.4 (Debian 11.4-1.pgdg90+1)) Type "help" for help. postgres=#
Following “Flask by Example, Part 2”, create a database called wordcount
:
$ docker-compose exec db psql -U postgres psql (11.4 (Debian 11.4-1.pgdg90+1)) Type "help" for help. postgres=# create database wordcount; CREATE DATABASE postgres=# l
List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+--------+----------+----------+----------+-------------------- postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres + | | | | |postgres=CTc/postgres template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres + | | | | |postgres=CTc/postgres wordcount| postgres | UTF8| en_US.utf8 | en_US.utf8 | (4 rows) postgres=# q
Connect to the wordcount
database and create a role called wordcount_dbadmin
that will be used by the Flask application:
$ docker-compose exec db psql -U postgres wordcount wordcount=# CREATE ROLE wordcount_dbadmin; CREATE ROLE wordcount=# ALTER ROLE wordcount_dbadmin LOGIN; ALTER ROLE wordcount=# ALTER USER wordcount_dbadmin PASSWORD 'MYPASS'; ALTER ROLE postgres=# q
The next step is to create a Dockerfile for installing all the prerequisites for the Flask application.
Make the following modifications to the requirements.txt file:
Modify the version of the psycopg2
package from 2.6.1
to 2.7
so that it supports PostgreSQL 11
Modify the version of the redis
package from 2.10.5
to 3.2.1
for better Python 3.7 support
Modify the version of the rq
package from 0.5.6
to 1.0
for better Python 3.7 support
Here is the Dockerfile:
$
cat DockerfileFROM
python:3.7.3-alpine
ENV
APP_HOME /app
WORKDIR
$APP_HOME
COPY requirements.txt .RUN
apk add --no-cache postgresql-libs
&&
apk add --no-cache --virtual .build-deps gcc musl-dev postgresql-dev
&&
python3 -m pip install -r requirements.txt --no-cache-dir
&&
apk --purge del .build-deps COPY . .
ENTRYPOINT
[ "python" ]
CMD
["app.py"]
There is an important difference between this Dockerfile and the version used in the first hello-world-docker example. Here the contents of the current directory, which includes the application files, are copied into the Docker image. This is done to illustrate a scenario different from the development workflow shown earlier. In this case, we are more interested in running the application in the most portable way, for example, in a staging or production environment, where we do not want to modify application files via mounted volumes as was done in the development scenario. It is possible and even common to use docker-compose
with locally mounted volumes for development purposes, but the focus in this section is on the portability of Docker containers across environments, such as development, staging, and production.
Run docker build -t flask-by-example:v1 .
to build a local Docker image. The output of this command is not shown because it is quite lengthy.
The next step in the “Flask By Example” tutorial is to run the Flask migrations.
In the docker-compose.yaml file, define a new service called migrations
and specify its image
, its command
, its environment
variables, and the fact that it depends on the db
service being up and running:
$ cat docker-compose.yaml
version
:
"3"
services
:
migrations
:
image
:
"flask-by-example:v1"
command
:
"manage.py
db
upgrade"
environment
:
APP_SETTINGS
:
config.ProductionConfig
DATABASE_URL
:
postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
depends_on
:
-
db
db
:
image
:
"postgres:11"
container_name
:
"postgres"
ports
:
-
"5432:5432"
volumes
:
-
dbdata:/var/lib/postgresql/data
volumes
:
dbdata
:
The DATABASE_URL
variable uses the name db
for the PostgreSQL database host. This is because the name db
is defined as a service name in the docker-compose.yaml file, and docker-compose
knows how to link one service to another by creating an overlay network where all services defined in the docker-compose.yaml file can interact with each other by their names. See the docker-compose networking reference for more details.
The DATABASE_URL
variable definition refers to another variable called DBPASS
, instead of hardcoding the password for the wordcount_dbadmin
user. The docker-compose.yaml file is usually checked into source control, and best practices are not to commit secrets such as database credentials to GitHub. Instead, use an encryption tool such as sops
to manage a secrets file.
Here is an example of how to create an encrypted file using sops
with PGP encryption.
First, install gpg
on macOS via brew install gpg
, then generate a new PGP key with an empty passphrase:
$ gpg --generate-key pub rsa2048 2019-07-12 [SC] [expires: 2021-07-11] E14104A0890994B9AC9C9F6782C1FF5E679EFF32 uid pydevops <[email protected]> sub rsa2048 2019-07-12 [E] [expires: 2021-07-11]
Next, download sops
from its release page.
To create a new encrypted file called, for example, environment.secrets, run sops
with the -pgp
flag and give it the fingerprint of the key generated above:
$ sops --pgp BBDE7E57E00B98B3F4FBEAF21A1EEF4263996BD0 environment.secrets
This will open the default editor and allow for the input of the plain-text secrets. In this example, the contents of the environment.secrets file are:
export DBPASS=MYPASS
After saving the environment.secrets file, inspect the file to see that it is encrypted, which makes it safe to add to source control:
$ cat environment.secrets { "data": "ENC[AES256_GCM,data:qlQ5zc7e8KgGmu5goC9WmE7PP8gueBoSsmM=, iv:xG8BHcRfdfLpH9nUlTijBsYrh4TuSdvDqp5F+2Hqw4I=, tag:0OIVAm9O/UYGljGCzZerTQ==,type:str]", "sops": { "kms": null, "gcp_kms": null, "lastmodified": "2019-07-12T05:03:45Z", "mac": "ENC[AES256_GCM,data:wo+zPVbPbAJt9Nl23nYuWs55f68/DZJWj3pc0 l8T2d/SbuRF6YCuOXHSHIKs1ZBpSlsjmIrPyYTqI+M4Wf7it7fnNS8b7FnclwmxJjptBWgL T/A1GzIKT1Vrgw9QgJ+prq+Qcrk5dPzhsOTxOoOhGRPsyN8KjkS4sGuXM=,iv:0VvSMgjF6 ypcK+1J54fonRoI7c5whmcu3iNV8xLH02k=, tag:YaI7DXvvllvpJ3Talzl8lg==, type:str]", "pgp": [ { "created_at": "2019-07-12T05:02:24Z", "enc": "-----BEGIN PGP MESSAGE----- hQEMA+3cyc g5b/Hu0OvU5ONr/F0htZM2MZQSXpxoCiO WGB5Czc8FTSlRSwu8/cOx0Ch1FwH+IdLwwL+jd oXVe55myuu/3OKUy7H1w/W2R PI99Biw1m5u3ir3+9tLXmRpLWkz7+nX7FThl9QnOS25 NRUSSxS7hNaZMcYjpXW+w M3XeaGStgbJ9OgIp4A8YGigZQVZZFl3fAG3bm2c+TNJcAbl zDpc40fxlR+7LroJI juidzyOEe49k0pq3tzqCnph5wPr3HZ1JeQmsIquf//9D509S5xH Sa9lkz3Y7V4KC efzBiS8pivm55T0s+zPBPB/GWUVlqGaxRhv1TAU= =WA4+ -----END PGP MESSAGE----- ", "fp": "E14104A0890994B9AC9C9F6782C1FF5E679EFF32" } ], "unencrypted_suffix": "_unencrypted", "version": "3.0.5" } }%
To decrypt the file, run:
$ sops -d environment.secrets export DBPASS=MYPASS
There is an issue with sops
interacting with gpg
on a Macintosh. You will need to run the following commands before being able to decrypt the file with sops
:
$ GPG_TTY=$(tty) $ export GPG_TTY
The goal here is to run the migrations
service defined previously in the docker-compose.yaml_ file.
To tie the +sops
secret management method into docker-compose
, decrypt the environments.secrets file with sops -d
, source its contents into the current shell, then invoke docker-compose up -d migrations
using one command line that will not expose the secret to the shell history:
$ source <(sops -d environment.secrets); docker-compose up -d migrations postgres is up-to-date Recreating flask-by-example_migrations_1 ... done
Verify that the migrations were successfully run by inspecting the database and verifying that two tables were created: alembic_version
and results
:
$ docker-compose exec db psql -U postgres wordcount psql (11.4 (Debian 11.4-1.pgdg90+1)) Type "help" for help. wordcount=# dt
List of relations Schema | Name | Type | Owner --------+-----------------+-------+------------------- public | alembic_version | table | wordcount_dbadmin public | results | table | wordcount_dbadmin (2 rows) wordcount=# q
Part 4 in the “Flask By Example” tutorial is to deploy a Python worker process based on Python RQ that talks to an instance of Redis.
First, Redis needs to run. Add it as a service called redis
into the docker_compose.yaml file, and make sure that its internal port 6379 is mapped to port 6379 on the local OS:
redis: image: "redis:alpine" ports: - "6379:6379"
Start the redis
service on its own by specifying it as an argument to docker-compose up -d
:
$ docker-compose up -d redis Starting flask-by-example_redis_1 ... done
Run docker ps
to see a new Docker container running based on the redis:alpine
image:
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a1555cc372d6 redis:alpine "docker-entrypoint.s…" 3 seconds ago Up 1 second 0.0.0.0:6379->6379/tcp flask-by-example_redis_1 83b54ab10099 postgres:11 "docker-entrypoint.s…" 22 hours ago Up 16 hours 0.0.0.0:5432->5432/tcp postgres
Use the docker-compose logs
command to inspect the logs of the redis
service:
$ docker-compose logs redis Attaching to flask-by-example_redis_1 1:C 12 Jul 2019 20:17:12.966 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1:C 12 Jul 2019 20:17:12.966 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=1, just started 1:C 12 Jul 2019 20:17:12.966 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 1:M 12 Jul 2019 20:17:12.967 * Running mode=standalone, port=6379. 1:M 12 Jul 2019 20:17:12.967 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 1:M 12 Jul 2019 20:17:12.967 # Server initialized 1:M 12 Jul 2019 20:17:12.967 * Ready to accept connections
The next step is to create a service called worker
for the Python RQ worker process in docker-compose.yaml:
worker
:
image
:
"flask-by-example:v1"
command
:
"worker.py"
environment
:
APP_SETTINGS
:
config.ProductionConfig
DATABASE_URL
:
postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
REDISTOGO_URL
:
redis://redis:6379
depends_on
:
-
db
-
redis
Run the worker service just like the redis
service, with docker-compose up -d
:
$ docker-compose up -d worker flask-by-example_redis_1 is up-to-date Starting flask-by-example_worker_1 ... done
Running docker ps
will show the worker container:
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 72327ab33073 flask-by-example "python worker.py" 8 minutes ago Up 14 seconds flask-by-example_worker_1 b11b03a5bcc3 redis:alpine "docker-entrypoint.s…" 15 minutes ago Up About a minute 0.0.0.0:6379->6379/tc flask-by-example_redis_1 83b54ab10099 postgres:11 "docker-entrypoint.s…" 23 hours ago Up 17 hours 0.0.0.0:5432->5432/tcp postgres
Look at the worker container logs with docker-compose logs
:
$ docker-compose logs worker Attaching to flask-by-example_worker_1 20:46:34 RQ worker 'rq:worker:a66ca38275a14cac86c9b353e946a72e' started, version 1.0 20:46:34 *** Listening on default... 20:46:34 Cleaning registries for queue: default
Now launch the main Flask application in its own container. Create a new service called app
in docker-compose.yaml:
app
:
image
:
"flask-by-example:v1"
command
:
"manage.py
runserver
--host=0.0.0.0"
ports
:
-
"5000:5000"
environment
:
APP_SETTINGS
:
config.ProductionConfig
DATABASE_URL
:
postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
REDISTOGO_URL
:
redis://redis:6379
depends_on
:
-
db
-
redis
Map port 5000 from the application container (the default port for a Flask application) to port 5000 on the local machine. Pass the command manage.py runserver --host=0.0.0.0
to the application container to ensure that port 5000 is exposed correctly by the Flask application inside the container.
Start up the app
service with docker compose up -d
, while also running sops -d
on the encrypted file containing DBPASS
, then sourcing the decrypted file before calling docker-compose
:
source <(sops -d environment.secrets); docker-compose up -d app postgres is up-to-date Recreating flask-by-example_app_1 ... done
Notice the new Docker container running the application in the list returned by docker ps
:
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d99168a152f1 flask-by-example "python app.py" 3 seconds ago Up 2 seconds 0.0.0.0:5000->5000/tcp flask-by-example_app_1 72327ab33073 flask-by-example "python worker.py" 16 minutes ago Up 7 minutes flask-by-example_worker_1 b11b03a5bcc3 redis:alpine "docker-entrypoint.s…" 23 minutes ago Up 9 minutes 0.0.0.0:6379->6379/tcp flask-by-example_redis_1 83b54ab10099 postgres:11 "docker-entrypoint.s…" 23 hours ago Up 17 hours 0.0.0.0:5432->5432/tcp postgres
Inspect the logs of the application container with docker-compose logs
:
$ docker-compose logs app Attaching to flask-by-example_app_1 app_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Running docker-compose logs
with no other arguments allows us to inspect the logs of all the services defined in the docker-compose.yaml file:
$ docker-compose logs Attaching to flask-by-example_app_1, flask-by-example_worker_1, flask-by-example_migrations_1, flask-by-example_redis_1, postgres 1:C 12 Jul 2019 20:17:12.966 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1:C 12 Jul 2019 20:17:12.966 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=1, just started 1:C 12 Jul 2019 20:17:12.966 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 1:M 12 Jul 2019 20:17:12.967 * Running mode=standalone, port=6379. 1:M 12 Jul 2019 20:17:12.967 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 1:M 12 Jul 2019 20:17:12.967 # Server initialized 1:M 12 Jul 2019 20:17:12.967 * Ready to accept connections app_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) postgres | 2019-07-12 22:15:19.193 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 postgres | 2019-07-12 22:15:19.194 UTC [1] LOG: listening on IPv6 address "::", port 5432 postgres | 2019-07-12 22:15:19.199 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres | 2019-07-12 22:15:19.214 UTC [22] LOG: database system was shut down at 2019-07-12 22:15:09 UTC postgres | 2019-07-12 22:15:19.225 UTC [1] LOG: database system is ready to accept connections migrations_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl. migrations_1 | INFO [alembic.runtime.migration] Will assume transactional DDL. worker_1 | 22:15:20 RQ worker 'rq:worker:2edb6a54f30a4aae8a8ca2f4a9850303' started, version 1.0 worker_1 | 22:15:20 *** Listening on default... worker_1 | 22:15:20 Cleaning registries for queue: default
The final step is to test the application. Visit http://127.0.0.1:5000 and enter python.org
in the URL field.
At that point, the application sends a job to the worker process, asking it to execute the function count_and_save_words
against the home page of python.org
.
The application periodically polls the job for the results, and upon completion, it displays the word frequencies
on the home page.
To make the docker-compose.yaml file more portable, push the flask-by-example
Docker image to Docker Hub, and reference the Docker Hub image in the container section for the app
and worker
services.
Tag the existing local Docker image flask-by-example:v1
with a name prefixed by a Docker Hub username, then push the newly tagged image to Docker Hub:
$ docker tag flask-by-example:v1 griggheo/flask-by-example:v1 $ docker push griggheo/flask-by-example:v1
Change docker-compose.yaml to reference the new Docker Hub image. Here is the final version of docker-compose.yaml:
$ cat docker-compose.yaml
version
:
"3"
services
:
app
:
image
:
"griggheo/flask-by-example:v1"
command
:
"manage.py
runserver
--host=0.0.0.0"
ports
:
-
"5000:5000"
environment
:
APP_SETTINGS
:
config.ProductionConfig
DATABASE_URL
:
postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
REDISTOGO_URL
:
redis://redis:6379
depends_on
:
-
db
-
redis
worker
:
image
:
"griggheo/flask-by-example:v1"
command
:
"worker.py"
environment
:
APP_SETTINGS
:
config.ProductionConfig
DATABASE_URL
:
postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
REDISTOGO_URL
:
redis://redis:6379
depends_on
:
-
db
-
redis
migrations
:
image
:
"griggheo/flask-by-example:v1"
command
:
"manage.py
db
upgrade"
environment
:
APP_SETTINGS
:
config.ProductionConfig
DATABASE_URL
:
postgresql://wordcount_dbadmin:$DBPASS@db/wordcount
depends_on
:
-
db
db
:
image
:
"postgres:11"
container_name
:
"postgres"
ports
:
-
"5432:5432"
volumes
:
-
dbdata:/var/lib/postgresql/data
redis
:
image
:
"redis:alpine"
ports
:
-
"6379:6379"
volumes
:
dbdata
:
To restart the local Docker containers, run docker-compose down
followed by docker-compose up -d
:
$ docker-compose down Stopping flask-by-example_worker_1 ... done Stopping flask-by-example_app_1 ... done Stopping flask-by-example_redis_1 ... done Stopping postgres ... done Removing flask-by-example_worker_1 ... done Removing flask-by-example_app_1 ... done Removing flask-by-example_migrations_1 ... done Removing flask-by-example_redis_1 ... done Removing postgres ... done Removing network flask-by-example_default $ source <(sops -d environment.secrets); docker-compose up -d Creating network "flask-by-example_default" with the default driver Creating flask-by-example_redis_1 ... done Creating postgres ... done Creating flask-by-example_migrations_1 ... done Creating flask-by-example_worker_1 ... done Creating flask-by-example_app_1 ... done
Note how easy it is to bring up and down a set of Docker containers with docker-compose
.
Even if you want to run a single Docker container, it is still a good idea to include it in a docker-compose.yaml file and launch it with the docker-compose up -d
command. It will make your life easier when you want to add a second container into the mix, and it will also serve as a mini Infrastructure as Code example, with the docker-compose.yaml file reflecting the state of your local Docker setup for your application.
We will now show how to take the docker-compose
setup from the preceding section and port it to a server running Ubuntu 18.04.
Launch an Amazon EC2 instance running Ubuntu 18.04 and install docker-engine
and docker-compose
:
$ sudo apt-get update $ sudo apt-get remove docker docker-engine docker.io containerd runc $ sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - $ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" $ sudo apt-get update $ sudo apt-get install docker-ce docker-ce-cli containerd.io $ sudo usermod -a -G docker ubuntu # download docker-compose $ sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose- $(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose $ sudo chmod +x /usr/local/bin/docker-compose
Copy the docker-compose.yaml file to the remote EC2 instance and start the db
service first, so that the database used by the application can be created:
$ docker-compose up -d db Starting postgres ... Starting postgres ... done $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 49fe88efdb45 postgres:11 "docker-entrypoint.s…" 29 seconds ago Up 3 seconds 0.0.0.0:5432->5432/tcp postgres
Use docker exec
to run the psql -U postgres
command inside the running Docker container for the PostgreSQL database. At the PostgreSQL prompt, create the wordcount
database and wordcount_dbadmin
role:
$ docker-compose exec db psql -U postgres psql (11.4 (Debian 11.4-1.pgdg90+1)) Type "help" for help. postgres=# create database wordcount; CREATE DATABASE postgres=# q $ docker exec -it 49fe88efdb45 psql -U postgres wordcount psql (11.4 (Debian 11.4-1.pgdg90+1)) Type "help" for help. wordcount=# CREATE ROLE wordcount_dbadmin; CREATE ROLE wordcount=# ALTER ROLE wordcount_dbadmin LOGIN; ALTER ROLE wordcount=# ALTER USER wordcount_dbadmin PASSWORD 'MYPASS'; ALTER ROLE wordcount=# q
Before launching the containers for the services defined in docker-compose.yaml, two things are necessary:
Run docker login
to be able to pull the Docker image pushed previously to Docker Hub:
$ docker login
Set the DBPASS
environment variable to the correct value in the current shell. The sops
method described in the local macOS setup can be used, but for this example, set it directly in the shell:
$ export DOCKER_PASS=MYPASS
Now launch all the services necessary for the application by running docker-compose up -d
:
$ docker-compose up -d Pulling worker (griggheo/flask-by-example:v1)... v1: Pulling from griggheo/flask-by-example 921b31ab772b: Already exists 1a0c422ed526: Already exists ec0818a7bbe4: Already exists b53197ee35ff: Already exists 8b25717b4dbf: Already exists 9be5e85cacbb: Pull complete bd62f980b08d: Pull complete 9a89f908ad0a: Pull complete d787e00a01aa: Pull complete Digest: sha256:4fc554da6157b394b4a012943b649ec66c999b2acccb839562e89e34b7180e3e Status: Downloaded newer image for griggheo/flask-by-example:v1 Creating fbe_redis_1 ... done Creating postgres ... done Creating fbe_migrations_1 ... done Creating fbe_app_1 ... done Creating fbe_worker_1 ... done $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f65fe9631d44 griggheo/flask-by-example:v1 "python3 manage.py r…" 5 seconds ago Up 2 seconds 0.0.0.0:5000->5000/tcp fbe_app_1 71fc0b24bce3 griggheo/flask-by-example:v1 "python3 worker.py" 5 seconds ago Up 2 seconds fbe_worker_1 a66d75a20a2d redis:alpine "docker-entrypoint.s…" 7 seconds ago Up 5 seconds 0.0.0.0:6379->6379/tcp fbe_redis_1 56ff97067637 postgres:11 "docker-entrypoint.s…" 7 seconds ago Up 5 seconds 0.0.0.0:5432->5432/tcp postgres
At this point, after allowing access to port 5000 in the AWS security group associated with our Ubuntu EC2 instance, you can hit the external IP of the instance on port 5000 and use the application.
It’s worth emphasizing one more time how much Docker simplifies the deployment of applications. The portability of Docker containers and images means that you can run your application on any operating system where the Docker engine runs. In the example shown here, none of the prerequisites needed to be installed on the Ubuntu server: not Flask, not PostgreSQL, and not Redis. It was also not necessary to copy the application code over from the local development machine to the Ubuntu server. The only file needed on the Ubuntu server was docker-compose.yaml. Then, the whole set of services comprising the application was launched with just one command:
$ docker-compose up -d
Beware of downloading and using Docker images from public Docker repositories, because many of them include serious security vulnerabilities, the most serious of which can allow an attacker to break through the isolation of a Docker container and take over the host operating system. A good practice here is to start with a trusted, pre-built image, or build your own image from scratch. Stay abreast of the latest security patches and software updates, and rebuild your image whenever any of these patches or updates are available. Another good practice is to scan all of your Docker images with one of the many Docker scanning tools available, among them Clair, Anchore, and Falco. Such scanning can be performed as part of a continuous integration/continuous deployment pipeline, when the Docker images usually get built.
Although docker-compose
makes it easy to run several containerized services as part of the same application, it is only meant to be run on a single machine, which limits its usefulness in production scenarios. You can really only consider an application deployed with docker-compose
to be “production ready” if you are not worried about downtime and you are willing to run everything on a single machine (this being said, Grig has seen hosting providers running Dockerized applications in production with docker-compose
). For true “production ready” scenarios, you need a container orchestration engine such as Kubernetes, which will be discussed in the next chapter.
Familiarize yourself with the Dockerfile reference.
Familiarize yourself with the Docker Compose configuration reference.
Create an AWS KMS key and use it with sops
instead of a local PGP
key. This allows you to apply AWS IAM permissions to the key, and restrict access to the key to only the developers who need it.
Write a shell script that uses docker exec
or docker-compose exec
to run the PostgreSQL commands necessary for creating a database and a role.
Experiment with other container technologies, such as Podman.