Like cAdvisor there are several ways you can launch Prometheus. To start with, we will launch a container and inject our own configuration file so that Prometheus knows where our cAdvisor endpoint is:

docker run 
  --detach=true 
  --volume=/monitoring_docker/Chapter03/prometheus.yml:/etc/prometheus/prometheus.yml 
  --publish=9090:9090 
  --name=prometheus 
prom/prometheus:latest

Once you have launched the container, Prometheus will be accessible on the following URL: http://192.168.33.10:9090. When you first load the URL, you will be taken to a status page; this gives some basic information on the Prometheus installation. The important part of this page is the list of targets. This lists the URL that Prometheus will be scrapping to capture metrics; you should see your cAdvisor URL listed with a state of HEALTHY, as shown in the following screenshot:

Another information page contains the following:

As we only have a few containers up and running at the moment, let's launch one that runs Redis so we can start to look at the query language built into Prometheus.

We will use the official Redis image for this and as we are only going to use this as an example we won't need to pass it any user variables:

docker run --name my-redis-server -d redis

We now have a container called my-redis-server running. cAdvisor should already be exposing metrics about the container to Prometheus; let's go ahead and see. In the Prometheus web interface, go to the Graph link in the menu at the top of the page. Here, you will be presented with a text box into which you can enter your query. To start with, let's look at the CPU usage of the Redis container.

In the box, enter the following:

container_cpu_usage_seconds_total{job="cadvisor",name="my-redis-server"}

Then, after clicking on Execute, you should have two results returned, listed in the Console tab of the page. If you remember, cAdvisor records the CPU usage of each of the CPU cores that the container has access to, which is why we have two values returned, one for "cpu00" and one for "cpu01". Clicking on the Graph link will show you results over a period of time:

As you can see in the preceding screenshot, we now have access to the usage graphs for the last 25 minutes, which is about how long ago I launched the Redis instance before generating the graph.

Also, when creating one of the graphs using the query tool in the main application, you can install a separate Dashboard application. This runs in a second container that connects to your main Prometheus container using the API as a data source.

Before we start the Dashboard container, we should initialize a SQLite3 database to store our configuration. To ensure that the database is persistent, we will store this on the host machine in /tmp/prom/file.sqlite3:

docker run 
  --volume=/tmp/prom:/tmp/prom 
  -e DATABASE_URL=sqlite3:/tmp/prom/file.sqlite3 
prom/promdash ./bin/rake db:migrate

Once we have initialized the database, we can launch the Dashboard application properly:

docker run 
  --detach=true 
  --volume=/tmp/prom:/tmp/prom 
  -e DATABASE_URL=sqlite3:/tmp/prom/file.sqlite3 
  --publish=3000:3000  
  --name=promdash 
prom/promdash

The application should now be accessible at http://192.168.33.10:3000/. The first thing we need to do is set up the data source. To do this, click on the Servers link at the top of the screen and then click on New Server. Here, you will be asked to provide the details of your Prometheus server. Name the server and enter the following URL:

Once you click on Create Server, you should receive a message saying Server was successfully created. Next up, you need to create a directory; this is where your dashboards will be stored.

Click on the Dashboards link in the top menu and then click on New directory and create one called Test directory. Now, you are ready to start creating Dashboards. Click on New Dashboard, call it My Dashboard, place it in Test directory. Once you click on Create Dashboard, you will be taken to the preview screen.

From here, you can build up dashboards using the control in the top right-hand side of each section. To add data, you simply enter the query you would like to see in the dashboard section:

At the moment, we are running Prometheus in a single container and its data is being stored within that same container. This means, if for any reason the container is terminated, our data is lost; it also means that we can't upgrade without loosing out data. To get around this problem, we can create a data volume container.

First of all, let's make sure we have removed all the running Prometheus containers:

docker stop prometheus&&dockerrm Prometheus

Next up, let's create a data container called promdata:

docker create 
  --volume=/promdata 
  --name=promdata 
prom/prometheus /bin/true

Finally, launch Prometheus again, this time, using the data container:

docker run 
  --detach=true 
  --volumes-from promdata 
  --volume=/monitoring_docker/Chapter03/prometheus.yml:/etc/prometheus/prometheus.yml 
  --publish=9090:9090 
  --name=prometheus 
prom/prometheus

This will ensure that, if you have to upgrade or relaunch your container, the metrics you have been capturing are safe and sound.

We have only touched on the basics of using Prometheus in this section of the module; for further information on the application, I recommend the following links as a good starting point: