Problem

The concept of a daemon in software systems exists at many levels. At an operating system level, a daemon is a long-running, self-recovering computer program that runs as a background process. In Unix, the names of daemons end in “d,” such as httpd, named, and sshd. In other operating systems, alternative terms such as services-started tasks and ghost jobs are used.

Regardless of what they are called, the common characteristics among these programs are that they run as processes and usually do not interact with the monitor, keyboard, and mouse, and are launched at system boot time. A similar concept exists at the application level too. For example, in the JVM daemon threads run in the background and provide supporting services to the user threads. These daemon threads have a low priority, run in the background without a say in the life of the application, and perform tasks such as garbage collection or finalization.

Similarly, there is also the concept of a DaemonSet in Kubernetes. Considering that Kubernetes is a distributed platform spread across multiple nodes and with the primary goal of managing application Pods, a DaemonSet is represented by Pods that run on the cluster nodes and provide some background capabilities for the rest of the cluster.

Solution

ReplicaSet and its predecessor ReplicationController are control structures responsible for making sure a specific number of Pods are running. These controllers constantly monitor the list of running Pods and make sure the actual number of Pods always matches the desired number. In that regard, a DaemonSet is a similar construct and is responsible for ensuring that a certain number of Pods are always running. The difference is that the first two run a specific number of Pods, usually driven by the application requirements of high availability and user load, irrespective of the node count.

On the other hand, a DaemonSet is not driven by consumer load in deciding how many Pod instances to run and where to run. Its main purpose is to keep running a single Pod on every node or specific nodes. Let’s see such a DaemonSet definition next in Example 9-1.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: random-refresher
spec:
  selector:
    matchLabels:
      app: random-refresher
  template:
    metadata:
      labels:
        app: random-refresher
    spec:
      nodeSelector:                        
        feature: hw-rng
      containers:
      - image: k8spatterns/random-generator:1.0
        name: random-generator
        command:
        - sh
        - -c
        - >-
          "while true; do
          java -cp / RandomRunner /host_dev/random 100000;
          sleep 30; done"
        volumeMounts:                      
        - mountPath: /host_dev
          name: devices
      volumes:
      - name: devices
        hostPath:                          
          path: /dev

Given this behavior, the primary candidates for a DaemonSet are usually infrastructure-related processes such as log collectors, metric exporters, and even kube-proxy, that perform cluster-wide operations. There are many differences in how DaemonSet and ReplicaSet are managed, but the main ones are the following:

• By default, a DaemonSet places one Pod instance to every node. That can be controlled and limited to a subset of nodes by using the nodeSelector field.

• A Pod created by a DaemonSet already has nodeName specified. As a result, the DaemonSet doesn’t require the existence of the Kubernetes scheduler to run containers. That also allows using a DaemonSet for running and managing the Kubernetes components.

• Pods created by a DaemonSet can run before the scheduler has started, which allows them to run before any other Pod is placed on a node.

• Since the scheduler is not used, the unschedulable field of a node is not respected by the DaemonSet controller.

• Pods managed by a DaemonSet are supposed to run only on targeted nodes, and as a result, are treated with higher priority and differently by many controllers. For example, the descheduler will avoid evicting such Pods, the cluster autoscaler will manage them separately, etc.

Typically a DaemonSet creates a single Pod on every node or subset of nodes. Given that, there are several ways for Pods managed by DaemonSets to be reached:

Service

Create a Service with the same Pod selector as a DaemonSet, and use the Service to reach a daemon Pod load-balanced to a random node.

DNS

Create a headless Service with the same Pod selector as a DaemonSet that can be used to retrieve multiple A records from DNS containing all Pod IPs and ports.

NodeIP with hostPort

Pods in the DaemonSet can specify a hostPort and become reachable via the node IP addresses and the specified port. Since the combination of hostIp and hostPort and protocol must be unique, the number of places where a Pod can be scheduled is limited.

Push

The application in the DaemonSets Pod can push data to a well-known location or service that’s external to the Pod. No consumer needs to reach the DaemonSets Pods.

Static Pods can be used to spin off a containerized version of Kubernetes system processes or other containers. But DaemonSets are better integrated with the rest of the platform and recommended over static Pods.

Discussion

In this book, we describe patterns and Kubernetes features primarily used by developers rather than platform administrators. A DaemonSet is somewhere in the middle, inclining more toward the administrator toolbox, but we include it here because it also has applicability to application developers. DaemonSets and CronJobs are also perfect examples of how Kubernetes turns single-node concepts such as Crontab and daemon scripts into multinode clustered primitives for managing distributed systems. These are new distributed concepts developers must also be familiar with.