Chapter 6. Troubleshooting and Monitoring

In this chapter, we are going to look at commands which will come in useful when troubleshooting your containers, all the commands we will look at are part of the core Docker Engine, we will also look at a way by which you can debug your Dockerfiles.

Once we have finished with the Troubleshooting commands, we will look at how we can monitor our containers using cAdvisor with a Prometheus backend fronted by a Grafana dashboard – don't worry, it is not as complicated as it sounds.

Note

As we are going to be exposing services, some using default credentials I would recommend that you use your local Docker installation for this chapter.

Troubleshooting containers

Computer programs (software) sometimes fail to behave as expected. This is due to faulty code or due to the environmental changes between the development, testing, and deployment systems. Docker container technology eliminates the environmental issues between development, testing, and deployment as much as possible by containerizing all the application dependencies. Nonetheless, there could still be anomalies due to faulty code or variations in the kernel behavior, which needs debugging. Debugging is one of the most complex processes in the software engineering world and it becomes much more complex in the container paradigm because of the isolation techniques. In this section, we are going to learn a few tips and tricks to debug a containerized application using the tools native to Docker, as well as the tools provided by external sources.

Initially, many people in the Docker community individually developed their own debugging tools, but later Docker started supporting native tools, such as exec, top, logs, events, and many more. In this section, we will dive deep into the following Docker tools:

  • exec
  • ps
  • top
  • stats
  • events
  • logs
  • attach

We shall also consider debugging a Dockerfile.

The exec command

The docker container exec command provided the much-needed help to users, who are deploying their own web servers or other applications running in the background.

Now, it is not necessary to log in to run the SSH daemon in the container.

First, launch a container:

docker container run -d --name trainingapp training/webapp:latest 
 
The exec command

Second, run the docker container ps command to get the container ID. Now you have the container ID you can run the docker container exec command to log in to the container using either the container ID or as we have named it trainingapp you can use that:

docker container exec -it 32005e837724 bash

Note

Please note, not every container will have bash installed, some such Alpine Linux don't have bash out of the box but instead uses sh, which bash was based on.

It is important to note that the docker container exec command can only access the running containers, so if the container stops functioning then you need to restart the stopped container to proceed. The docker container exec command spawns a new process in the target containers namespace using the Docker API and CLI.

A containers name space is what separates the containers from each other, for example you can have several containers all running the same process, but because the processes have been launched within each of the containers namespace they are isolated from one another. A good example of this is are MySQL processes, on a traditional server trying to run more than one MySQL server process will mean that you need to start the process on different ports, use different lock, PID and log files as well as different init scripts.

As Docker is isolating each MySQL server process all you need to worry about is that if you are exposing the MySQL port on the host machine is that you don't assign it on the same port as another container.

So, if you run the ps -aef command inside the target container, it looks like this:

The exec command

Here, python app.y is the application that is already running in the target container, and the docker container exec command has added the bash process inside the container. If you run kill -9 59 (replacing the 59 with the PID of your own bash process), you will be automatically logged out of the container.

It is recommended that you use the docker container exec command only for monitoring and diagnostic purposes, and I personally believe in the concept of one process per container, which is one of the best practices widely accentuated.

The ps command

The ps command, which is available inside the container, is used to see the status of the process. This is like the standard ps command in the Linux environment and is not a dockercontainerps command that we run on the Docker host machine.

This command runs inside the Docker container:

The ps command

Use ps --help <simple|list|output|threads|misc|all> or ps --help <s|l|o|t|m|a> for additional help text.

The top command

You can run the top command from the Docker host machine using the following command:

docker container top CONTAINER [ps OPTIONS]

This gives a list of the running processes of a container without logging into the container, as follows:

The top command

The within the container the top command provides information about the CPU, memory, and swap usage just like any normal Linux host:

The top command

In case you get the error as error - TERM environment variable not set while running the top command inside the container, perform the following steps to resolve it.

Run echo$TERM and if you get the result dumb, then, run the following command:

export TERM=dumb 

This will resolve your error and you can run the top command.

The stats command

The docker container stats command provides you with the capability to view the memory, CPU, and the network usage of a container from a Docker host machine, as illustrated here. Running the following command:

docker container stats 32005e837724

Gives you the following:

The stats command

You can run the stats command to also view the usage for multiple containers:

docker container stats 32005e837724 5e33f02f5fd2 7c9cf27ff46a
The stats command

Since Docker 1.5, you have been able to access to container statistics read only parameters. This will streamline the CPU, memory, network IO, and block IO of your containers.

This helps you choose the resource limits and in profiling. The Docker stats utility provides you with these resource usage details only for running containers.

You can get detailed information using the endpoint APIs at the following URL https://docs.docker.com/engine/api/v1.26/.

The Docker events command

Docker containers will report the following real-time events: create, destroy, die, export, kill, omm, pause, restart, start, stop, and unpause. Let's pause and unpause our container:

The Docker events command

If you specify an image it will also report the untag and delete events.

Using multiple filters will be handled as an AND operation, for example:

docker events --filter container=32005e837724 --filter event=pause --filter event=unpause --since 12h 

Preceding will display all pause and unpause events for the container a245253db38b for the last 12 hours:

The Docker events command

Currently, the supported filters are container, event, and image.

The logs command

This command fetches the log of a container without logging into the container. It batch-retrieves logs present at the time of execution. These logs are the output of STDOUT and STDERR. The general usage is shown in:

docker container logs [OPTIONS] CONTAINERID

The --follow option will continue to provide the output till the Docker logs command is terminated printing any new log entries to the screen in real time,-t will provide the timestamp, and --tail=<number of lines> will show the number of lines of the log messages of your container:

docker container logs 32005e837724
The logs command
docker container logs -t 32005e837724
The logs command

We also used the docker container logs command in previous chapters to view the logs of our database containers.

The attach command

This command attaches the running container and it is very helpful when you want to see what is written in stdout in real time, let's launch new test container which outputs something to stdout:

docker container run -d --name=newtest alpine /bin/sh -c "while true; do sleep 2; df -h; done"

Now we can attach to the container using the following command to see the output;

docker container attach newtest

By default, this command attaches stdin and proxies signals to the remote process. Options are available to control both behaviors. To detach from the process, use the default Ctrl + Q sequence.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset