Chapter 5: Implementing Storage for the Container's Data
In the previous chapters, we explored how to run and manage our containers using Podman, but we will soon come to realize in this chapter that these operations aren't useful in certain scenarios where the applications included in our containers need to store data in a persistent mode. Containers are ephemeral by default, and this is one of their main features, as we described in the first chapter of this book, and for this reason, we need a way to attach persistent storage to a running container to preserve the container's important data.
In this chapter, we're going to cover the following main topics:
- Why does storage matter for containers?
- Containers' storage features
- Copying files into and out of a container
- Attaching host storage to a container
Technical requirements
Before proceeding with the chapter's lecture and examples, a machine with a working Podman installation is required. As stated in Chapter 3, Running the First Container, all the examples in the book are executed on a Fedora 34 system or later but can be reproduced on the reader's OS of choice.
Finally, a good understanding of the topics covered in Chapter 4, Managing Running Containers, is useful in terms of being able to easily grasp concepts regarding OCI images and container execution.
Why does storage matter for containers?
Before moving forward in the chapter and answering this interesting question, we need to distinguish between two kinds of storage for containers:
- External storage attached to running containers to store data, making it persistent on a container's restart
- Underlying storage for root filesystems of our containers and container images
Talking about external storage, as we described in Chapter 1, Introduction to Container Technology, containers are stateless, ephemeral, and often with a read-only filesystem. This is because the theory behind the technology states that containers should be used for spawning scalable and distributed applications that have to scale horizontally instead of vertically.
Scaling an application horizontally means that in case we require additional resources for our running services, we will not increase CPU or RAM for a single running container, but we will instead launch a brand new container that will handle the incoming requests along with the existing container. This is the same well-known paradigm adopted in the public cloud. The container in principle should be ephemeral because any additional copy of the existing container image should be run at any time for empowering the existing running service.
Of course, exceptions exist, and it could happen that a running container cannot be scaled horizontally or that it simply needs to share configurations, cache, or any other data relevant to other copies of the same container images at startup time or during runtime.
Let's understand this with the help of a real-life example. Using a car-sharing service to get a new car for every destination inside a city can be a useful and smart way to move around without worrying about parking fees, fuel, and other things. However, at the same time, this service cannot allow you to store or leave your stuff inside of a parked car. Therefore, when using a car-sharing service, we can unpack our stuff once we get into a car, but we must pack it back before we leave that car. The same applies similarly to containers, where we must attach to them some storage for letting our container write data down but then, once our container stops, we should detach that storage so that a brand-new container can use it when needed.
Here's another more technical example: let's consider a standard three-tier application with a web, a backend, and a database service. Every layer of this application may need storage, which it will use in a variety of ways. The web service may need a place to save a cache, store rendered web pages, some customized images at runtime, and so on. The backend service will need a place to store configuration and synchronization data between the other running backend services, if any, and so on. The database service will surely need a place to store the DB data.
Storage is often associated with low-level infrastructure, but in a container, the storage becomes important even for developers, who should plan where to attach the storage, and the features needed for their application.
If we extend the topic to container orchestration, then the storage inherits a strategic role because it should be as elastic and feasible as the Kubernetes orchestrator that we might use it with. The container storage in this case should become more like software-defined storage – able to provide storage resources in a self-service way to developers, and to containers in general.
Although this book will talk about local storage, it's important to note that this is not enough for the Kubernetes orchestrator because containers should be portable from one host to another depending on the availability and scaling rules defined. This is where software-defined storage could be the solution!
As we can deduct from the previous examples, external storage matters in containers. The usage may vary depending on the running application inside our container, but it is required. At the same time, another key role is driven by the underlying container storage that is responsible for handling the correct storage of containers and the container images' root filesystem. Choosing the right, stable, and performing underlying local storage will ensure better and correct management of our containers.
So, let's first explore a bit of the theory of container storage and then discuss how to work with it.
Containers' storage features
Before going into a real example and use cases, we should first dig into the main differences between container storage and a container storage interface (CSI).
Container storage, previously referred to as underlying container storage, is responsible for handling container images on Copy-on-Write (COW) filesystems. Container images need to be transferred and move around until a container engine is instructed to run them, so we need a way to store that image until it is run. That's the role of container storage.
Once we start using an orchestrator such as Kubernetes, CSI instead is responsible for providing container block or file storage that containers need to write data to.
In the next section of this chapter, we will concentrate on container storage and its configuration. Later, we will talk about external storage for containers and the options we have in Podman to expose the host local storage to the running containers.
A great innovation introduced with Podman is the containers/storage project (https://github.com/containers/storage), a great way to share an underlying common method for accessing container storage on a host. With the arrival of Docker, we were forced to pass through the Docker daemon to interact with container storage. With no other way to directly interact with the underlying storage, the Docker daemon just hid it from the user as well as the system administrator.
With the containers/storage project, we now have an easy way to use multiple tools for analyzing, managing, or working with container storage at the same time.
The configuration of this low-level piece of software is so important for Podman as well as for other companion tools of Podman and can be inspected or edited through its configuration file available at /etc/containers/storage.conf.
Looking at the configuration file, we can easily discover that we can change a lot of options in terms of how our containers interact with the underlying storage. Let's inspect the most important option – the storage driver.
Storage driver
The configuration file, as one of its first options, gives the opportunity to choose the default Copy On Write (COW) container storage driver. The configuration file in the current version, at the time of writing this book, supports the following COW drivers:
- overlay
- vfs
- devmapper
- aufs
- btrfs
- zfs
These are also often referred to as graph drivers because most of them organize the layers they handle in a graph structure.
Using Podman on Fedora 34 or later, the container's storage configuration file is shipped with overlay as the default driver.
Another important thing to mention is that, at the time of writing this book, there are two versions of the overlay filesystem – version 1 and version 2.
The original overlay filesystem version 1 was initially used by the Docker container engine, but was later abandoned in favor of version 2. That's why Podman and the container's storage configuration file refers generically to the name overlay, but it instead uses the new version 2.
Before going into detail regarding the other options and, finally, the practical examples contained in this chapter, let's further explore how one of these COW filesystem drivers works.
The overlay union filesystem has been present in a Linux kernel since version 3.18. It is usually enabled by default and activated dynamically once a mount is initiated with this filesystem.
The mechanism behind this filesystem is really simple but powerful – it allows a directory tree to be overlaid on another, storing only the differences, but showing the latest updated, squashed tree of directories.
Usually, in the world of containers, we start using a read-only filesystem, adding one or more layers, read-only again, until a running container will use this bunch of squashed layers as its root filesystem. This is where the last read-write layer will be created as an overlay of the others.
Let's see what happens under the hood once we pull down a brand-new container image with Podman:
Important Note
If you wish to proceed with testing the following example on your test machine, ensure that you remove any running container and container images to easily match the image with the layers that Podman will download for us.
# podman pull quay.io/centos7/httpd-24-centos7:latest
Trying to pull quay.io/centos7/httpd-24-centos7:latest...
Getting image source signatures
Copying blob 5f2e13673ac2 done
Copying blob 8dd5a5013b51 done
Copying blob b2cc5146c9c7 done
Copying blob e17e89f32035 done
Copying blob 1b6c93aa6be5 done
Copying blob 6855d3fe68bc done
Copying blob f974a2323b6c done
Copying blob d620f14a5a76 done
Copying config 3b964f33a2 done
Writing manifest to image destination
Storing signatures
3b964f33a2bf66108d5333a541d376f63e0506aba8ddd4813f9d4e104 271d9f0
We can see from the previous command output that multiple layers have been downloaded. That's because the container image we pulled down is composed of many layers.
Now we can start inspecting just the downloaded layers. First of all, we have to locate the right directory, which we can search for inside the configuration file. Alternatively, we can use an easier technique. Podman has a command dedicated to displaying its running configuration and other useful information – podman info. Let's see how it works:
# podman info | grep -A19 "store:"
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mountopt: nodev,metacopy=on
graphRoot: /var/lib/containers/storage
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "true"
imageStore:
number: 1
runRoot: /run/containers/storage
volumePath: /var/lib/containers/storage/volumes
To reduce the output of the podman info command, we used the grep command to only match the store section that contains the current configuration in place for container storage.
As we can see, the driver used is overlay, and the root directory to search our layers is reported as the graphRoot directory: /var/lib/containers/storage; for rootless containers, the equivalent is $HOME/.local/share/containers/storage. We also have other paths reported, but we will talk about these later in this section. The keyword graph is a term derived from the category of drivers we just introduced earlier.
Let's take a look into that directory to see what the actual content is:
# cd /var/lib/containers/storage
# ls
libpod mounts overlay overlay-containers overlay-images overlay-layers storage.lock tmp userns.lock
We have several directories available for which the names are pretty self-explanatory. The ones we are looking for are as follows:
- overlay-images: This contains the metadata of the container images downloaded.
- overlay-layers: This contains the archives for all the layers of every container image.
- overlay: This is the directory containing the unpacked layers of every container image.
Let's check the content of the first directory, overlay-images:
# ls -l overlay-images/
total 8
drwx------. 1 root root 630 15 oct 18.36 3b964f33a2bf66108d5333a541d376f63e0506aba8ddd4813f9d4e10427 1d9f0
-rw-------. 1 root root 1613 15 oct 18.36 images.json
-rw-r--r--. 1 root root 64 15 oct 18.36 images.lock
As we can imagine, in this directory, we can find the metadata of the only container image we pulled down and, in the directory with a very long ID, we will find the manifest file describing the layers that make up our container image.
Let's now check the content of the second directory, overlay-layers:
# ls -l overlay-layers/
total 1168
-rw-------. 1 root root 2109 15 oct 18.35 0099baae6cd3ca0ced38d658d7871548b32bd0e42118b788d818b76131ec 8e75.tar-split.gz
-rw-------. 1 root root 795206 15 oct 18.35 53498d66ad83a29fcd7c7bcf4abbcc0def4fc912772aa8a4483b51e232309 aee.tar-split.gz
-rw-------. 1 root root 52706 15 oct 18.35 6c26feaaa75c7bac1f1247acc06e73b46e8aaf2e741ad1b8bacd6774bffdf6 ba.tar-split.gz
-rw-------. 1 root root 1185 15 oct 18.35 74fa1495774e94d5cdb579f9bae4a16bd90616024a6f4b1ffd13344c367df1 f6.tar-split.gz
-rw-------. 1 root root 308144 15 oct 18.36 ae314017e4c2de17a7fb007294521bbe8ac1eeb004ac9fb57d1f1f03090f78 c9.tar-split.gz
-rw-------. 1 root root 1778 15 oct 18.36 beba3570ce7dd1ea38e8a1b919a377b6dc888b24833409eead446bff401d8f 6e.tar-split.gz
-rw-------. 1 root root 697 15 oct 18.36 e59e7d1e1874cc643bfe6f854a72a39f73f22743ab38eff78f91dc019cca91 f5.tar-split.gz
-rw-------. 1 root root 5555 15 oct 18.36 e5a13564f9c6e233da30a7fd86489234716cf80c317e52ff8261bf0cb34dc 7b4.tar-split.gz
-rw-------. 1 root root 3716 15 oct 18.36 layers.json
-rw-r--r--. 1 root root 64 15 oct 19.06 layers.lock
As we can see, we just found all the layers' archives downloaded for our container image, but where they have been unpacked? The answer is easy – in the third folder, overlay:
# ls -l overlay
total 0
drwx------. 1 root root 46 15 oct 18.35 0099baae6cd3ca0ced38d658d7871548b32bd0e42118b788d818b76131ec 8e75
drwx------. 1 root root 46 15 oct 18.35 53498d66ad83a29fcd7c7bcf4abbcc0def4fc912772aa8a4483b51e23230 9aee
drwx------. 1 root root 46 15 oct 18.35 6c26feaaa75c7bac1f1247acc06e73b46e8aaf2e741ad1b8bacd6774bffd f6ba
drwx------. 1 root root 46 15 oct 18.35 74fa1495774e94d5cdb579f9bae4a16bd90616024a6f4b1ffd13344c367d f1f6
drwx------. 1 root root 46 15 oct 18.35 ae314017e4c2de17a7fb007294521bbe8ac1eeb004ac9fb57d1f1f03090f 78c9
drwx------. 1 root root 46 15 oct 18.36 beba3570ce7dd1ea38e8a1b919a377b6dc888b24833409eead446bff401d 8f6e
drwx------. 1 root root 46 15 oct 18.36 e59e7d1e1874cc643bfe6f854a72a39f73f22743ab38eff78f91dc019cca 91f5
drwx------. 1 root root 46 15 oct 18.36 e5a13564f9c6e233da30a7fd86489234716cf80c317e52ff8261bf0cb34d c7b4
drwx------. 1 root root 416 15 oct 18.36 l
The first question that could arise when looking at the latest directory content is, what's the purpose of the l (L in lowercase) directory?
To answer this question, we have to inspect the content of a layer directory. We can start with the first one on the list:
# ls -la overlay/0099baae6cd3ca0ced38d658d7871548b32bd0e42118b788d818b 76131ec8e75/
total 8
drwx------. 1 root root 46 15 oct 18.35 .
drwx------. 1 root root 1026 15 oct 18.36 ..
dr-xr-xr-x. 1 root root 24 15 oct 18.35 diff
-rw-r--r--. 1 root root 26 15 oct 18.35 link
-rw-r--r--. 1 root root 86 15 oct 18.35 lower
drwx------. 1 root root 0 15 oct 18.35 merged
drwx------. 1 root root 0 15 oct 18.35 work
Let's understand the purpose of these files and directories:
- diff: This directory represents the upper layer of the overlay, and is used to store any changes to the layer.
- lower: This file reports all the lower layer mounts, ordered from uppermost to lowermost.
- merged: This directory is the one that the overlay is mounted on.
- work: This directory is used for internal operations.
- link: This file contains a unique string for the layer.
Now, coming back to our question, what's the purpose of the l (L in lowercase) directory?
Under the l directory, there are symbolic links with unique strings pointing to the diff directory for every layer. The symbolic links reference lower layers in the lower file. Let's check it:
# ls -la overlay/l/
total 32
drwx------. 1 root root 416 15 oct 18.36 .
drwx------. 1 root root 1026 15 oct 18.36 ..
lrwxrwxrwx. 1 root root 72 15 oct 18.35 A4ZYMM4AK5NM6JYJA7 EK2DLTGA -> ../74fa1495774e94d5cdb579f9bae4a16bd90616024a6f4 b1ffd13344c367df1f6/diff
lrwxrwxrwx. 1 root root 72 15 oct 18.35 D2WVDYIWL6I77ZOIXR VQKCXNG2 -> ../ae314017e4c2de17a7fb007294521bbe8ac1eeb004ac9fb57d1f1f03090 f78c9/diff
lrwxrwxrwx. 1 root root 72 15 oct 18.36 G4KXMAOCE56TIB252 ZMWEFRFHU -> ../beba3570ce7dd1ea38e8a1b919a377b6dc888b24833409eead446bff401 d8f6e/diff
lrwxrwxrwx. 1 root root 72 15 oct 18.35 JHHF5QA7YSKDSKRSC HNADBVKDS -> ../53498d66ad83a29fcd7c7bcf4abbcc0def4fc912772aa8a4483b51e2 32309aee/diff
lrwxrwxrwx. 1 root root 72 15 oct 18.36 KNCK5EDUAQJDAIDWQ6 TWDFQF5B -> ../e59e7d1e1874cc643bfe6f854a72a39f73f22743ab38eff78f91dc019cca 91f5/diff
lrwxrwxrwx. 1 root root 72 15 oct 18.35 LQUM7XDVWHIJRLIWAL CFKSMJTT -> ../0099baae6cd3ca0ced38d658d7871548b32bd0e42118b788d818b76131 ec8e75/diff
lrwxrwxrwx. 1 root root 72 15 oct 18.35 V6OV3TLBBLTATIJDCTU 6N72XQ5 -> ../6c26feaaa75c7bac1f1247acc06e73b46e8aaf2e741ad1b8bacd6774bf fdf6ba/diff
lrwxrwxrwx. 1 root root 72 15 oct 18.36 ZMKJYKM2VJEAYQHCI7SU Q2R3QW -> ../e5a13564f9c6e233da30a7fd86489234716cf80c317e52ff8261bf0cb34dc7 b4/diff
To double-check what we just learned, let's find the first layer of our container image and check whether there is a lower file for it.
Let's inspect the manifest file for our container image:
# cat overlay-images/3b964f33a2bf66108d5333a541d376f63e0506ab a8ddd4813f9d4e104271d9f0/manifest | head -15
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1 +json",
"size": 16212,
"digest": "sha256:3b964f33a2bf66108d5333a541d376f63e0506aba8ddd4813f9d4 e104271d9f0"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 75867345,
"digest": "sha256:b2cc5146c9c7855cb298ca8b77ecb153d37e3e5c69916ef42361 3a46a70c0503"
},
Then, we must compare the checksum of the compressed archive with the list of all the layers we downloaded:
Good to Know
SHA-256 is an algorithm used to produce a unique cryptographic hash that can be used to verify the integrity of a file (checksum).
# cat overlay-layers/layers.json | jq | grep -B3 -A10 "sha256:b2cc5"
{
"id": "53498d66ad83a29fcd7c7bcf4abbcc0def4fc912772aa8a4483b51e23 2309aee",
"created": "2021-10-15T16:35:49.782784856Z",
"compressed-diff-digest": "sha256:b2cc5146c9c7855cb298ca8b77ecb153d37e3e5c69916ef423 613a46a70c0503",
"compressed-size": 75867345,
"diff-digest": "sha256:53498d66ad83a29fcd7c7bcf4abbcc0def4fc912772aa8a448 3b51e232309aee",
"diff-size": 211829760,
"compression": 2,
"uidset": [
0,
192
],
"gidset": [
0,
The file we just analyzed, overlay-layers/layers.json, was not indented. For this reason, we used the jq utility to format it and make it human-readable.
Good to Know
If you cannot find the jq utility on your system, you can install it through the operating system default package manager. On Fedora, for example, you can run dnf install jq.
As you can see, we just found the ID of our root layer. Now, let's look at its content:
# ls -l overlay/53498d66ad83a29fcd7c7bcf4abbcc0def4fc912772aa8a448 3b51e232309aee/
total 4
dr-xr-xr-x. 1 root root 158 15 oct 18.35 diff
drwx------. 1 root root 0 15 oct 18.35 empty
-rw-r--r--. 1 root root 26 15 oct 18.35 link
drwx------. 1 root root 0 15 oct 18.35 merged
drwx------. 1 root root 0 15 oct 18.35 work
As we can verify, there is not a lower file inside the layer's directory because this is the first layer of our container image!
The difference we might notice is the presence of a directory named empty. This is because if a layer has no parent, then the overlay system will create a dummy lower directory named empty and it will skip writing a lower file.
Finally, as the last stage of our practical example, let's run our container and verify that a new diff layer will be created. We expect that this layer will contain only the difference between the lower ones.
First, we run our container image we just analyzed:
# podman run -d quay.io/centos7/httpd-24-centos7
bd0eef7cd50760dd52c24550be51535bc11559e52eea7d782a1fa69 76524fa76
As you can see, we started it in the background through the -d option to continue working on the system host. After this, we will execute a new shell on the pod to actually check the container's root folder and create a new file on it:
# podman exec -ti bd0eef7cd50760dd52c24550be51535bc11559e52eea7d782a1fa69 76524fa76 /bin/bash
bash-4.2$ pwd
/opt/app-root/src
bash-4.2$ echo "this is my NOT persistent data" > tempfile.txt
bash-4.2$ ls
tempfile.txt
This new file we just created will be temporary and will only last for the lifetime of the container. It is now time to find the diff layer that was just created by the overlay driver on our host system. The easiest way is to analyze the mount points used in the running container:
bash-4.2$ mount | head
overlay on / type overlay (rw,relatime,context="system_u:object_r:container_file_t:s0:c300,c861",lowerdir=/var/lib/containers/storage/overlay/l/ZMKJYKM2VJEAYQHCI7SUQ2R3QW:/var/lib/containers/storage/overlay/l/G4KXMAOCE56TIB252ZMWEFRFHU:/var/lib/containers/storage/overlay/l/KNCK5EDUAQJDAIDWQ6TWDF QF5B:/var/lib/containers/storage/overlay/l/D2WVDYIWL6I77ZOIX RVQKCXNG2:/var/lib/containers/storage/overlay/l/LQUM7XDVWHI JRLIWALCFKSMJTT:/var/lib/containers/storage/overlay/l/A4ZYMM4AK5NM6JYJA7EK2DLTGA:/var/lib/containers/storage/overlay/l/V6OV3TLBBLTATIJDCTU6N72XQ5:/var/lib/containers/storage/overlay/l/JHHF5QA7YSKDSKRSCHNADBVKDS,upperdir=/var/lib/containers/storage/overlay/b71e4bea5380ca233bf6b0c7a1c276179b841e263ee293e987c6cc54 af516f23/diff,workdir=/var/lib/containers/storage/overlay/b71e4bea5380ca233bf6b0c7a1c276179b841e263ee293e987c6cc54af 516f23/work,metacopy=on)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,context="system_u:object _r:container_file_t:s0:c300,c861",size=65536k,mode=755,inode64)
As you can see, the first mount point of the list shows a very long line full of layer paths divided by colons. In this long line, we can find the upperdir directory we are searching for:
upperdir=/var/lib/containers/storage/overlay/b71e4bea5380ca233bf6b0c7a1c276179b841e263ee293e987c6cc54af5 16f23/diff
Now, we can inspect the content of this directory and navigate to the various paths available to find the container root directory where we wrote that file in the previous commands:
# ls -la /var/lib/containers/storage/overlay/b71e4bea5380ca233bf6b0c7a1c276179b841e263ee293e987c6cc54af5 16f23/diff/opt/app-root/src/
total 12
drwxr-xr-x. 1 1001 root 58 16 oct 00.40 .
drwxr-xr-x. 1 1001 root 12 22 set 10.39 ..
-rw-------. 1 1001 root 81 16 oct 00.46 .bash_history
-rw-------. 1 1001 root 1024 16 oct 00.38 .rnd
-rw-r--r--. 1 1001 root 31 16 oct 00.39 tempfile.txt
# cat /var/lib/containers/storage/overlay/b71e4bea5380ca233bf6b0c7a1c276179b841e263ee293e987c6cc54af5 16f23/diff/opt/app-root/src/tempfile.txt
this is my NOT persistent data
As we verified, the data is stored on the host operating system, but it is stored in a temporary layer that will sooner or later be removed once the container is removed!
Now, coming back to the original topic that sent us on this small trip under the hood of the overlay storage driver, we were talking about /etc/containers/storage.conf. This file holds all the configurations for the containers/storage project that is responsible for sharing an underlying common method to access container storage on a host.
The other options available in this file are related to the customization of the storage driver as well as changing the default path for the internal storage directories.
The last point we should briefly talk about is the runroot directory. In this folder, the container storage program will store all temporary writable content produced by the container.
If we inspect the folder on our running host where we started the container for the previous example, we will find that there is a folder named with its ID with various files that have been mounted on the container to replace the original files:
# ls -l /run/containers/storage/overlay-containers/bd0eef7cd50760dd52c24550be51535bc11559e52eea7d782a1fa69765 24fa76/userdata
total 20
-rw-r--r--. 1 root root 6 16 oct 00.38 conmon.pid
-rw-r--r--. 1 root root 12 16 oct 00.38 hostname
-rw-r--r--. 1 root root 230 16 oct 00.38 hosts
-rw-r--r--. 1 root root 0 16 oct 00.38 oci-log
-rwx------. 1 root root 6 16 oct 00.38 pidfile
-rw-r--r--. 1 root root 34 16 oct 00.38 resolv.conf
drwxr-xr-x. 3 root root 60 16 oct 00.38 run
As you can see from the preceding output, the container's folder under the runroot path contains various files that have been mounted directly onto the container to customize it.
To wrap up, in the previous examples, we analyzed the anatomy of a container image and what happens once we run a new container from that image. The technology behind the scenes is amazing and we saw that a lot of features are related to the isolation capabilities offered by the operating system. Here, storage offers other important functionalities that have made containers the greatest technology that we all now know about.
Copying files in and out of a container
Podman enables users to move files into and out of a running container. This result is achieved using the podman cp command, which can move files and folders to and from a container. Its usage is quite simple and will be illustrated in the next example.
First, let's start a new Alpine container:
$ podman run -d --name alpine_cp_test alpine sleep 1000
Now, let's grab a file from the container – we have chosen the /etc/os-release file, which provides some information about the distribution and its version ID:
$ podman cp alpine_cp_test:/etc/os-release /tmp
The file has been copied to the host /tmp folder and can be inspected:
$ cat /tmp/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.14.2
PRETTY_NAME="Alpine Linux v3.14"
HOME_URL=https://alpinelinux.org/
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
In the opposite direction, we can copy files or folders from the host to the running container:
$ podman cp /tmp/build_folder alpine_cp_test:/
This example copies the /tmp/build_folder folder, and all its content, under the root filesystem of the Alpine container. We can then inspect the result of the copy command by using podman exec with the ls utility command.
Interacting with overlayfs
There is another way to copy files from a container to the host, which is by using the podman mount command and interacting directly with the merged overlays.
To mount a running rootless container's filesystem, we first need to run the podman unshare command, which permits users to run commands inside a modified user namespace:
$ podman unshare
This command drops a root shell in a new user namespace configured with UID 0 and GID 0. It is now possible to run the podman mount command and obtain the absolute path of the mount point:
# cd $(podman mount alpine_cp_test)
The preceding command uses shell expansion to change to the path of the MergedDir, which, as the name says, merges the LowerDir and UpperDir contents to provide a unified view of the different layers. From now on, it is possible to copy files to and from the container root filesystem.
The previous examples were based on rootless containers, but the same logic applies to rootful containers. Let's start a rootful Nginx container:
$ sudo podman run -d
--name rootful_nginx docker.io/library/nginx
To copy files in and out, we need to prepend the sudo command:
$ sudo podman cp
rootful_nginx:/usr/share/nginx/html/index.html /tmp
The preceding command copies the default index.html page to the host /tmp directory. Keep in mind that sudo elevates the user privileges to root, and therefore copied files will have UID 0 and GID 0 ownership.
The practice of copying files and folders from a container is especially useful for troubleshooting purposes. The opposite action of copying them inside a running container can be useful for updating and testing secrets or configuration files. In that case, we have the option of persisting those changes, as described in the next subsection.
Persisting changes with podman commit
The previous examples are not a method for permanently customizing running containers, since the immutable nature of containers implies that persistent modifications should go through an image rebuild.
However, if we need to preserve the changes and produce a new image without starting a new build, the podman commit command provides a way to persist the changes to a container into a new image.
The commit concept is of primary importance in Docker and OCI image builds. In fact, we can see the different steps of a Dockerfile as a series of commits applied during the build process.
The following example shows how to persist a file copied into a running container and produce a new image. Let's say we want to update the default index.html page of our Nginx container:
$ echo "Hello World!" > /tmp/index.html
$ podman run --name custom_nginx -d -p
8080:80 docker.io/library/nginx
$ podman cp /tmp/index.html
custom_nginx:/usr/share/nginx/html/
Let's test the changes applied:
$ curl localhost:8080
Hello World!
Now we want to persist the changed index.html file into a new image, starting from the running container with podman commit:
$ podman commit -p custom_nginx hello-world-nginx
The preceding command persists the changes by effectively creating a new image layer containing the updated files and folders.
The previous container can now be safely stopped and removed before testing the new custom image:
$ podman stop custom_nginx && podman rm custom_nginx
Let's test the new custom image and inspect the changed index.html file:
$ podman run -d -p 8080:80 --name hello_world
localhost/hello-world-nginx
$ curl localhost:8080
Hello World!
In this section, we have learned how to copy files to and from a running container and how to commit the changes on the fly by producing a new image.
In the next section, we are going to learn how host storage is attached to a container by introducing the concept of volumes and bind mounts.
Attaching host storage to a container
We have already talked about the immutable nature of containers. Starting from pre-built images, when we run a container, we instance a read/write layer on top of a stack of read-only layers using a copy-on-write approach.
Containers are ephemeral objects based on a stateful image. This implies that containers are not meant to store data inside them – if a container crashes or is removed, all the data would be lost. We need a way to store data in a separate location that is mounted inside the running container, preserved when the container is removed, and ready to be reused by a new container.
There is another important caveat that should not be forgotten – secrets and config files. When we build an image, we can pass all the files and folders we need inside it. However, sealing secrets like certificates or keys inside a build is not a good practice. If we need, for example, to rotate a certificate, we must rebuild the whole image from scratch. In the same way, changing a config file that resides inside an image implies a new rebuild every time we change a setting.
For these reasons, OCI specifications support volumes and bind mounts to manage storage attached to a container. In the next sections, we will learn how volumes and bind mounts work and how to attach them to a container.
Managing and attaching bind mounts to a container
Let's start with bind mounts since they leverage a native Linux feature. According to the official Linux man pages, a bind mount is a way to remount a part of the filesystem hierarchy somewhere else. This means that using bind mounts, we can replicate the view of a directory under another mount point in the host.
Before learning how containers use bind mounts, let's see a basic example where we simply bind mount the /etc directory under the /mnt directory:
$ sudo mount --bind /etc /mnt
After issuing this command, we will see the exact contents of /etc under /mnt. To unmount, simply run the following command:
$ sudo umount /mnt
The same concept can be applied to containers – Podman can bind mount host directories inside a container and offers dedicated CLI options to simplify the mount process.
Podman offers two options that can be used to bind mount: -v|--volume and –mount. Let's cover these in more detail.
-v|--volume option
This option uses a compact, single field argument to define the source host directory and the container mount point with the pattern /HOST_DIR:/CONTAINER_DIR. The following example mounts the /host_files directory on the /mnt mount point inside the container:
$ podman run -v /host_files:/mnt docker.io/library/nginx
It is possible to pass extra arguments to define mount behavior; for example, to mount the host directory as read-only:
$ podman run –v /host_files:/mnt:ro
docker.io/library/nginx
Other viable options for bind mounts using the -v|--volume option can be found in the run command man page (man podman-run).
--mount option
This option is more verbose since it uses a key=value syntax to define source and destinations as well as the mount type and extra arguments. This option accepts different mount types (bind mounts, volumes, tmpfs, images, and devpts) in the type=TYPE,source=HOST_DIR,destination=CONTAINER_DIR pattern. The source and destination keys can be replaced with the shorter src and dst, respectively. The previous example can be rewritten as follows:
$ podman run
--mount type=bind,src=/host_files,dst=/mnt
docker.io/library/nginx
We can also pass an extra option by adding an extra comma; for example, to mount the host directory as read-only:
$ podman run
--mount type=bind,src=/host_files,dst=/mnt,ro=true
docker.io/library/nginx
Despite being very simple to use and understand, bind mounts have some limitations that could impact the life cycle of the container in some cases. Host files and directories must exist before running the containers and permissions must be set accordingly to make them readable or writable. Another important caveat to keep in mind is that a bind mount always obfuscates the underlying mount point in the container if populated by files or directories. A useful alternative to bind mounts is volumes, described in the next subsection.
Managing and attaching volumes to a container
A volume is a directory created and managed directly by the container engine and mounted to a mount point inside the container. They offer a great solution for persisting data generated by a container.
Volumes can be managed using the podman volume command, which can be used to list, inspect, create, and remove volumes in the system. Let's start with a basic example, with a volume automatically created by Podman on top of the Nginx document root:
$ podman run -d -p 8080:80 --name nginx_volume1 -v /usr/share/nginx/html docker.io/library/nginx
This time, the –v option has an argument with only one item – the document root directory. In this case, Podman automatically creates a volume and bind mounts it to the target mount point.
To prove that a new volume has been created, we can inspect the container:
$ podman inspect nginx_volume1
[...omitted output...]
"Mounts": [
{
"Type": "volume",
"Name": "2ed93716b7ad73706df5c6f56bda262920accec59e7b6642d36f938e936 d36d9",
"Source": "/home/packt/.local/share/containers /storage/volumes/2ed93716b7ad73706df5c6f56bda262920accec59e7b6 642d36f93 8e936d36d9/_data",
"Destination": "/usr/share/nginx/html",
"Driver": "local",
"Mode": "",
"Options": [
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rprivate"
}
],
[…omitted output]
In the Mounts section, we have a list of objects mounted in the container. The only item is an object of the volume type, with a generated UID as its Name and a Source field that represents its path in the host, while the Destination field is the mount point inside the container.
We can double-check the existence of the volume with the podman volume ls command:
$ podman volume ls
DRIVER VOLUME NAME
local 2ed93716b7ad73706df5c6f56bda262920accec59e7b6642d36f93 8e936d36d9
Looking inside the source path, we will find the default files in the container document root:
$ ls -al
/home/packt/.local/share/containers/storage/volumes/2ed93716b7ad73706df5c6f56bda262920accec59e7b6642d36f93 8e936d36d9/_data
total 16
drwxr-xr-x. 2 gbsalinetti gbsalinetti 4096 Sep 9 20:26 .
drwx------. 3 gbsalinetti gbsalinetti 4096 Oct 16 22:41 ..
-rw-r--r--. 1 gbsalinetti gbsalinetti 497 Sep 7 17:21 50x.html
-rw-r--r--. 1 gbsalinetti gbsalinetti 615 Sep 7 17:21 index.html
This demonstrated that when an empty volume is created, it is populated with the content of the target mount point. When a container stops, the volume is preserved along with all the data and can be reused when the container is restarted by another container.
The preceding example shows a volume with a generated UID, but it is possible to choose the name of the attached volume, as in the following example:
$ podman run -d -p 8080:80 --name nginx_volume2 -v nginx_vol:/usr/share/nginx/html docker.io/library/nginx
In the preceding example, Podman creates a new volume named nginx_vol and stores it under the default volumes directory. When a named volume is created, Podman does not need to generate a UID.
The default volumes directory has different paths for rootless and rootful containers:
- For rootless containers, the default volume storage path is <USER_HOME>/.local/share/containers/storage/volumes.
- For rootful containers, the default volume storage path is /var/lib/containers/storage/volumes.
Volumes created in those paths are persisted after the container is destroyed and can be reused by other containers.
To manually remove a volume, use the podman volume rm command:
$ podman volume rm nginx_vol
When dealing with multiple volumes, the podman volume prune command removes all the unused volumes. The following example prunes all the volumes in the user default volume storage (the one used by rootless containers):
$ podman volume prune
The next example shows how to remove volumes used by rootful containers by using the sudo prefix:
$ sudo podman volume prune
Important Note
Do not forget to monitor volumes accumulating in the host since they consume disk space that could be reclaimed, and prune unused volumes periodically to avoid cluttering the host storage.
Users can also preliminarily create and populate volumes before running the container. The following example uses the podman create volume command to create the volume mounted to the Nginx document root and then populates it with a test index.html file:
$ podman volume create custom_nginx
$ echo "Hello World!" >> $(podman volume inspect custom_nginx –format "{{ .Mountpoint }}")/index.html
We can now run a new Nginx container using the pre-populated volume:
$ podman run -d -p 8080:80 --name nginx_volume3 -v custom_nginx:/usr/share/nginx/html docker.io/library/nginx
The HTTP test shows the updated contents:
$ curl localhost:8080
Hello World!
This time, the volume, which was not empty in the beginning, obfuscated the container target directory with its contents.
Mounting volumes with the --mount option
As with bind mounts, we can freely choose between the -v|--volume and the --mount options. The following example runs an Nginx container using the --mount flag:
$ podman run -d -p 8080:80 --name nginx_volume4 --mount type=volume,src=custom_nginx,dst=/usr/share/nginx/html docker.io/library/nginx
While the -v|--volume option is compact and widely adopted, the advantage of the --mount option is a more clear and expressive syntax, along with an exact statement of the mount type.
Volume drivers
The preceding volume examples are all based on the same local volume driver, which is used to manage volume in the local filesystem of the host. Additional volume drivers can be configured in the /usr/share/containers/containers.conf file in the [engine.volume_plugins] section by passing the plugin name followed by the file or socket path.
The local volume driver can also be used to mount NFS shares in the host running the container. This result cannot be achieved with rootless containers anyway. The following example shows how to create a volume backed by an NFS share and mount it inside a MongoDB container on its /data/db directory:
$ sudo podman volume create --driver local --opt type=nfs --opt o=addr=nfs-host.example.com,rw,context="system_u:object_r:container_file_t:s0" --opt device=:/opt/nfs-export nfs-volume
$ sudo podman run -d -v nfs-volume:/data/db docker.io/library/mongo
A prerequisite of the preceding example is the preliminary configuration of the NFS server, which should be accessible by the host running the container.
Volumes in builds
Volumes can be pre-defined during the image build process. This lets image maintainers define which container directories will be automatically attached to volumes. To understand this concept, let's inspect this minimal Dockerfile:
FROM docker.io/library/nginx:latest
VOLUME /usr/share/nginx/html
The only change made to the docker.io/library/nginx image is a VOLUME directive, which defines which directory should be externally mounted as an anonymous volume in the host. This is simply metadata, and the volume will be created only at runtime when a container is started from this image.
If we build the image and run a container based on the example Dockerfile, we can see an automatically created anonymous volume:
$ podman build -t my_nginx .
$ podman run -d --name volumes_from_build my_nginx
$ podman inspect volumes_from_build --format "{{ .Mounts }}"
[{volume 4d6ac7edcb4f01add205523b7733d61ae4a5772786eacca68e49 72b20fd1180c /home/packt/.local/share/containers/storage/volumes/4d6ac7edcb4f01add205523b7733d61ae4a5772786eacca68e4972 b20fd1180c/_data /usr/share/nginx/html local [nodev exec nosuid rbind] true rprivate}]
Without an explicit volume creation option, Podman has already created and mounted the container volume. This automatic volume definition at build time is a common practice in all containers that are expected to persist data, like databases.
For example, the docker.io/library/mongo image is already configured to create two volumes, one for /data/configdb and one for /data/db. The same behavior can be identified in the most common databases, including PostgreSQL, MariaDB, and MySQL.
It is possible to define how pre-defined anonymous volumes should be mounted when the container is started. The default ID bind, which means that new volumes are created and bind-mounted in the container, but it is possible to use tmpfs or ignore the mount altogether with the --image-volume option. The following example starts a MongoDB container with its default volumes mounted as tmpfs:
$ podman run -d --image-volume tmpfs docker.io/library/mongo
In Chapter 6, Meet Buildah – Building Containers from Scratch, we will cover the build process in greater detail. We now close this subsection with an example of how to mount volumes across multiple containers.
Mounting volumes across containers
One of the greatest advantages of volumes is their flexibility. For example, a container can mount volumes from an already running container to share the same data. To accomplish this result, we can use the --volumes-from option. The following example starts a MongoDB container and then cross mounts its volumes on a Fedora container:
$ podman run -d --name mongodb01 docker.io/library/mongo
$ podman run -it --volumes-from=mongodb01 docker.io/library/fedora
The second container drops an interactive root shell we can use to inspect the filesystem content:
[root@c10420016687 /]# ls -al /data
total 20
drwxr-xr-t. 4 root root 4096 Oct 17 15:36 .
dr-xr-xr-x. 19 root root 4096 Oct 17 15:36 ..
drwxr-xr-x. 2 999 999 4096 Sep 20 22:20 configdb
drwxr-xr-x. 4 999 999 4096 Oct 17 15:36 db
As expected, we can find the MongoDB volumes mounted in the Fedora container. If we stop and even remove the first mongodb01 container, the volumes remain active and mounted inside the Fedora container.
Until now, we have seen basic use cases with no specific segregation between containers or mounted resources. If the host has SELinux enabled and in enforcing mode, some extra considerations must be applied.
SELinux considerations for mounts
SELinux recursively applies labels to files and directories to define their context. Those labels are usually stored as extended filesystem attributes. SELinux uses contexts to manage policies and define which processes can access a specific resource.
The ls command is used to see the type context of a resource:
$ ls -alZ /etc/passwd
-rw-r--r--. 1 root root system_u:object_r:passwd_file_t:s0 2965 Jul 28 21:00 /etc/passwd
In the preceding example, the passwd_file_t label defines the type context of the /etc/passwd file. Depending on the type context, a program can or cannot access a file while SELinux is running in enforcing mode.
Processes also have their type context – containers run with the label container_t and have read/write access to files and directories labeled with container_file_t type context, and read/execute access to container_share_t labeled resources.
Other host directories accessible by default are /etc as read-only and /usr as read/execute. Also, resources under /var/lib/containers/overlay/ are labeled as container_share_t.
What happens if we try to mount a directory not correctly labeled?
Podman still executes the container without complaining about the wrong labeling, but the mounted directory or file will not be accessible from a process running inside the containers, which are labeled with the container_t context type. The following example tries to mount a custom document root for an Nginx container without respecting the labeling constraints:
$ mkdir ~/custom_docroot
$ echo "Hello World!" > ~/custom_docroot/index.html
$ podman run -d
--name custom_nginx
-p 8080:80
-v ~/custom_docroot:/usr/share/nginx/html
docker.io/library/nginx
Apparently, everything went fine – the container started properly and the processes inside it are running, but if we try to contact the Nginx server, we see the error:
$ curl localhost:8080
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.21.3</center>
</body>
</html>
403 – Forbidden shows that the Nginx process cannot access the index.html page. To fix this error, we have two options – put SELinux in permissive mode or relabel the mounted resources. By putting SELinux in permissive mode, it continues to track down the violations without blocking them. Anyway, this is not a good practice and should be used only when we cannot correctly troubleshoot access issues and need to put SELinux out of the equation. The following command sets SELinux to permissive mode:
$ sudo setenforce 0
Important Note
Permissive mode is not equal to disabling SELinux entirely. When working in this mode, SELinux still logs AVC denials without blocking. System admins can immediately switch between permissive and enforcing modes without rebooting. Disabling, on the other hand, implies a full system reboot.
The second preferred option is to simply relabel the resources we need to mount. To achieve this result, we could use SELinux command-line tools. As a shortcut, Podman offers a simpler way – the :z and :Z suffixes applied to the volume mount arguments. The difference between the two suffixes is subtle:
- The :z suffix tells Podman to relabel the mounted resources in order to enable all containers to read and write it. It works with both volumes and bind mounts.
- The :Z suffix tells Podman to relabel the mounted resources in order to enable only the current container to read and write it exclusively. This also works with both volumes and bind mounts.
To test the difference, let's try to run the container again with the :z suffix and see what happens:
$ podman run -d
--name custom_nginx
–p 8080:80
-v ~/custom_docroot:/usr/share/nginx/html:z
docker.io/library/nginx
Now, the HTTP calls return the expected results since the process was able to access the index.html file without being blocked by SELinux:
$ curl localhost:8080
Hello World!
Let's look at the SELinux file context automatically applied to the mounted directory:
$ ls -alZ ~/custom_docroot
total 20
drwxrwxr-x. 2 packt packt system_u:object_r:container_file_t:s0 4096 Oct 16 15:53 .
drwxrwxr-x. 74 packt packt unconfined_u:object_r:user_home_dir_t:s0 12288 Oct 16 16:32 ..
-rw-rw-r--. 1 packt packt system_u:object_r:container_file_t:s0 13 Oct 16 15:53 index.html
Let's focus on the system_u:object_r:container_file_t:s0 label. The final s0 field is a Multi-Level Security (MLS) sensitivity level, which means that all processes with the same sensitivity level will have read/write access to the resource. Therefore, other containers that run with the s0 sensitivity level will be able to mount the resource with read/write access privileges. This also represents a security issue since a malicious container on the same host would be able to attack other containers by stealing or overwriting data.
The solution to this problem is called Multi-Category Security (MCS). SELinux uses MCS to configure additional categories, which are plaintext labels applied to the resources along with the other SELinux labels. MCS-labeled objects are then accessible only to processes with the same categories assigned.
When a container is started, processes inside it are labeled with MCS categories, following the pattern cXXX,cYYY, where XXX and YYY are randomly picked integers.
Podman automatically applies MCS categories to mounted resources when Z (uppercase) is passed. To test this behavior, let's run the Nginx container again with the :Z suffix:
$ podman run -d
--name custom_nginx
–p 8080:80
-v ~/custom_docroot:/usr/share/nginx/html:Z
docker.io/library/nginx
We can immediately see that the mounted folder has been relabeled with MCS categories:
$ ls -alZ ~/custom_docroot
total 20
drwxrwxr-x. 2 packt packt system_u:object_r:container_file_t:s0:c16,c898 4096 Oct 16 15:53 .
drwxrwxr-x. 74 packt packt unconfined_u:object_r:user_home_dir_t:s0 12288 Oct 16 21:12 ..
-rw-rw-r--. 1 packt packt system_u:object_r:container_file_t:s0:c16,c898 13 Oct 16 15:53 index.html
A simple test will return the expected Hello World! text, proving that the processes inside the container are allowed to access the target resources:
$ curl localhost:8080
Hello World!
What happens if we run a second container with the same approach, by applying :Z again to the same bind mount?
$ podman run -d
--name custom_nginx2
–p 8081:80
-v ~/custom_docroot:/usr/share/nginx/html:Z
docker.io/library/nginx
This time, we run the HTTP test on port 8081 and HTTP GET still works correctly:
$ curl localhost:8081
Hello World!
However, if we test once again the container mapped to port 8080, we will get an unexpected 403 Forbidden message:
$ curl localhost:8080
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.21.3</center>
</body>
</html>
Not surprisingly, the second container was executed with the :Z suffix and relabeled the directory with a new pair of MCS categories, thus making the first container unable to access the previously available content.
Important Note
The previous examples were conducted with bind mounts, but applied to volumes in the same way. Use these techniques with caution to avoid unwanted relabels of a bind mounted system or home directories.
In this subsection, we demonstrated the power of SELinux to manage containers and resource isolation. Let's conclude this chapter with an overview of other types of storage that can be attached to containers.
Attaching other types of storage to a container
Along with bind mounts and volumes, it is possible to attach other types of storage to containers, more specifically, of the kinds tmpfs, image, and devpts.
Attaching tmpfs storage
Sometimes, we need to attach storage to containers that is not meant to be persistent (for example, cache usage). Using volumes or bind mounts would clutter the host local disk (or any other backend if using different storage drivers). In those particular cases, we can use a tmpfs volume.
tmpfs is a virtual memory filesystem, which means that all its contents are created inside the host virtual memory. A benefit of tmpfs is that it provides faster I/O since all the read/write operations mostly happen in the RAM.
To attach a tmpfs volume to a container, we can use the --mount option or the --tmpfs option.
The --mount flag has the great advantage of being more verbose and expressive regarding the storage type, source, destination, and extra mount options. The following example runs an httpd container with a tmpfs volume attached to the container:
$ podman run –d –p 8080:80
--name tmpfs_example1
--mount type=tmpfs,tmpfs-size=512M,destination=/tmp
docker.io/library/httpd
The preceding command creates a tmpfs volume of 512 MB and mounts it on the /tmp folder of the container. We can test the correct mount creation by running the mount command inside the container:
$ podman exec -it tmpfs_example1 mount | grep '/tmp'
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,context="system_u:object_r:container_file_t:s0:c375,c804",size=524288k,uid=1000,gid=1000,inode64
This demonstrates that the tmpfs filesystem has been correctly mounted inside the container. Stopping the container will automatically discard tmpfs:
$ podman stop tmpfs_example1
The following example mounts a tmpfs volume using the --tmpfs option:
$ podman run –d –p 8080:80
--name tmpfs_example2
--tmpfs /tmp:rw,size= 524288k,mode=1777
docker.io/library/httpd
This example provides the same results as the previous one: a running container with a 512 MB tmpfs volume mounted on the /tmp directory in read/write mode and 1777 permissions.
By default, the tmpfs volume is mounted inside the container with the following mount options – rw, noexec, nosuid, and nodev.
Another interesting feature is the automatic MCS labeling from SELinux. This provides automatic segregation of the filesystem and prevents any other container from accessing the data in memory.
Attaching images
OCI images are the base that provides layers and metadata to start containers, but they can also be attached to a container filesystem at runtime. This can be useful for troubleshooting purposes or for attaching binaries that are available in a foreign image. When an OCI image is mounted inside a container, an extra overlay is created. This implies that even when the image is mounted with read/write permissions, users never alter the original image but the upper overlay only.
The following example mounts a busybox image with read/write permissions inside an Alpine container:
$ podman run -it
--mount type=image,src=docker.io/library/busybox,dst=/mnt,rw=true
alpine
Important Note
The mounted image must already be cached in the host. Podman only pulls the base container image if it is available when a container is created, but it expects the mounted images to already be available. A preliminary pull of the images will solve the issue.
Attaching devpts
This option is useful for attaching a pseudo terminal slave (PTS) to a container. This feature was introduced in Podman 2.1.0 to support containers that need to mount /dev/ from the host into the container, while still creating a terminal. The /dev pseudo filesystem of the host enables containers to gain direct access to the machine's physical or virtual devices.
To create a container with the /dev filesystem and a devpts device attached, run the following command:
$ sudo podman run -it
-v /dev/:/dev:rslave
--mount type=devpts,destination=/dev/pts
docker.io/library/fedora
To check the result of the mount option, we require an extra tool inside the container. For this reason, we can install it with the following command:
[root@034c8a61a4fc /]# dnf install -y toolbox
The resulting container has an extra, non-isolated, devpts device mounted on /dev/pts:
# mount | grep '/dev/pts'
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,context="system_u:object_r:container_file_t:s0:c299,c741",gid=5,mode=620,ptmxmode=666)
The preceding output was extracted by running the mount command inside the container.
Summary
In this chapter, we have completed a journey on container storage and Podman features offered to manipulate it. The material in this chapter is crucial to understanding how Podman manages both ephemeral and persistent data and provides best practices to users to manipulate their data.
In the first section, we learned why container storage matters and how it should be correctly managed both in single host and orchestrated, multi-host environments.
In the second section, we took a deep dive into container storage features and storage drivers, with a special focus on overlayfs.
In the third section, we learned how to copy files to and from a container. We also saw how changes could be committed to a new image.
The fourth section described the different possible scenarios of storage attached to a container, covering bind mounts, volumes, tmpfs, images, and devpts. This section was also a perfect fit to discuss SELinux interaction with storage management and see how we can use it to isolate storage resources across containers on the same host.
In the next chapter, we will learn a very important topic for both developers and operations teams, which is how to build OCI images with both Podman and Buildah, an advanced and specialized image-building tool.
Further reading
Refer to the following resources for more information:
- Containers Storage project page: https://github.com/containers/storage
- Container Labeling: https://danwalsh.livejournal.com/81269.html
- Why you should be using Multi-Category Security for your Linux containers: https://www.redhat.com/en/blog/why-you-should-be-using-multi-category-security-your-linux-containers
- Udica: Generate SELinux policies: https://github.com/containers/udica
- Overlay source code: https://github.com/containers/storage/blob/main/drivers/overlay/overlay.go