Since Prometheus only pulls metrics from endpoints it knows, we have to explicitly tell it where we'd like to collect data from. Under the /config path is a page that lists the current configured targets to pull. By default, there would be one job that runs against Prometheus itself, and this can be found in the conventional scraping path, /metrics. If we are connecting to the endpoint, we would see a very long text page, as shown in the following:

$ kubectl exec -n monitoring <prometheus_pod_name> -- 
    wget -qO - localhost:9090/metrics

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.5657e-05
go_gc_duration_seconds{quantile="0.25"} 1.9756e-05
go_gc_duration_seconds{quantile="0.5"} 2.4567e-05
go_gc_duration_seconds{quantile="0.75"} 2.8386e-05
...

This is the Prometheus metrics format we've mentioned several times before. Next time we see a page like this, we will know that it's a metrics endpoint. The job to scrape Prometheus is a static target in the default configuration file. However, due to the fact that containers in Kubernetes are created and destroyed dynamically, it is really difficult to find out the exact address of a container, let alone set it in Prometheus. In some cases, we may utilize the service DNS as a static metrics target, but this still cannot solve all cases. For instance, if we'd like to know how many requests are coming to each pod behind a service individually, setting a job to scrape the service might get a result from random pods instead of from all of them. Fortunately, Prometheus helps us overcome this problem with its ability to discover services inside Kubernetes.

To be more specific, Prometheus is able to query Kubernetes about the information of running services. It can then add them to or delete them from the target configuration accordingly. Five discovery mechanisms are currently supported:

• The node discovery mode creates one target per node. The target port would be kubelet's HTTPS port (10250) by default.
• The service discovery mode creates a target for every Service object. All defined target ports in a service would become a scraping target.
• The pod discovery mode works in a similar way to the service discovery role; it creates a target per pod and it exposes all the defined container ports for each pod. If there is no port defined in a pod's template, it would still create a scraping target with its address only.
• The endpoints mode discovers the Endpoint objects created by a service. For example, if a service is backed by three pods with two ports each, we'll have six scraping targets. In addition, for a pod, not only ports that expose to a service, but also other declared container ports would be discovered.
• The ingress mode creates one target per Ingress path. As an Ingress object can route requests to more than one service, and each service might have own metrics set, this mode allows us to configure all those targets at once.

The following diagram illustrates four discovery mechanisms. The left-hand ones are the resources in Kubernetes, and those on the right are the targets created in Prometheus:

Generally speaking, not all exposed ports are served as a metrics endpoint, so we certainly don't want Prometheus to grab everything it discovers in our cluster, but instead to only collect marked resources. To achieve this in Prometheus, a conventional method is to utilize annotations on resource manifests to distinguish which targets are to be grabbed, and then we can filter out those non-annotated targets using the relabel module in the Prometheus configuration. Consider this example configuration:

...

kubernetes_sd_configs:
- role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_mycom_io_scrape]
    action: keep
    regex: true
...

This tells Prometheus to keep only targets with the __meta_kubernetes_pod_annotation_{name} label and the value true. The label is fetched from the annotation field on the pod's specification, as shown in the following snippet:

...
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        mycom.io/scrape: "true"
...

Note that Prometheus would translate every character that is not in the range [a-zA-Z0-9_] to _, so we can also write the previous annotation as mycom-io-scrape: "true". 

By combining those annotations and the label filtering rule, we can precisely control the targets that need to be collected. Some commonly-used annotations in Prometheus are listed as follows:

• prometheus.io/scrape: "true"
• prometheus.io/path: "/metrics"
• prometheus.io/port: "9090"
• prometheus.io/scheme: "https"
• prometheus.io/probe: "true"

Those annotations can be seen at Deployment objects (for their pods) and Service objects. The following template snippet shows a common use case:

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/monitoring"
    prometheus.io/scheme: "http"
    prometheus.io/port: "9090"

By applying the following configuration, Prometheus will translate the discovered target in endpoints mode into http://<pod_ip_of_the_service>:9090/monitoring:

- job_name: 'kubernetes-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__,__meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::d+)?;(d+)
    replacement: $1:$2
    target_label: __address__
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)

We can use the prometheus.io/probe annotation in Prometheus to denote whether a service should be added to the probing target or not. The probing task would be executed by the Blackbox exporter (https://github.com/prometheus/blackbox_exporter).

Occasionally, we might want the metrics from any single pod under a service, not from all pods of a service. Since most endpoint objects are not created manually, the endpoint discovery mode uses the annotations inherited from a service. This means that if we annotate a service, the annotation will be visible in both the service discovery and endpoint discovery modes simultaneously, which prevents us from distinguishing whether the targets should be scraped per endpoint or per service. To solve this problem, we could use prometheus.io/scrape: "true" to denote endpoints that are to be scraped, and use another annotation like prometheus.io/scrape_service_only: "true" to tell Prometheus to create exactly one target for this service. 

The prom-config-k8s.yml template under our example repository contains some basic configurations to discover Kubernetes resources for Prometheus. Apply it as follows:

$ kubectl apply -f prometheus/configs/prom-config-k8s.yml

Because the resource in the template is a ConfigMap, which stores data in the etcd consensus storage, it takes a few seconds to become consistent. Afterward, we can reload Prometheus by sending a SIGHUP to the process:

$ kubectl exec -n monitoring <prometheus_pod_name> -- kill -1 1

The provided template is based on this example from Prometheus' official repository. You can find out further uses at the following link, which also includes the target discovery for the Blackbox exporter: https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml. We have also passed over the details of how the actions in the configuration actually work; to find out more, consult the official documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration.