Kubernetes provides several mechanisms for global discovery across the cluster. If your storage cluster is not managed by Kubernetes, you still need to tell Kubernetes pods how to find it and access it. There are two main methods:
- DNS
- Environment variables
In some cases, you may want to use both where environment variables can override DNS.
The DNS approach is simple and straightforward. Assuming your external storage cluster is load balanced and can provide a stable endpoint, then pods can just hit directly that endpoint and connect to the external cluster.
Another simple approach is to use environment variables to pass connection information to an external storage cluster. Kubernetes offers the ConfigMap resource as a way to keep configuration separate from the container image. The configuration is a set of key-value pairs. The configuration information can be exposed as an environment variable inside the container as well as volumes. You may prefer to use secrets for sensitive connection information.
The following configuration file will create a configuration file that keeps a list of addresses:
apiVersion: v1 kind: ConfigMap metadata: name: db-config namespace: default data: db-ip-addresses: 1.2.3.4,5.6.7.8 > kubectl create -f .configmap.yaml configmap "db-config" created
The data section contains all the key value pairs. In this case, just a single pair with a key name of db-ip-addresses. It will be important later when consuming the configmap in a pod. You can check out the content to make sure it's OK:
> kubectl get configmap db-config -o yaml apiVersion: v1 data: db-ip-addresses: 1.2.3.4,5.6.7.8 kind: ConfigMap metadata: creationTimestamp: 2017-01-09T03:14:07Z name: db-config namespace: default resourceVersion: "551258" selfLink: /api/v1/namespaces/default/configmaps/db-config uid: aebcc007-d619-11e6-91f1-3a7ae2a25c7d
There are other ways to create ConfigMap. You can directly create them using the --from-value or --from-file command line arguments.
When you are creating a pod, you can specify a ConfigMap and consume its values in several ways. Here is how to consume our configuration map as an environment variable:
apiVersion: v1 kind: Pod metadata: name: some-pod spec: containers: - name: some-container image: busybox command: [ "/bin/sh", "-c", "env" ] env: - name: DB_IP_ADDRESSES valueFrom: configMapKeyRef: name: db-config key: db-ip-addresses restartPolicy: Never
This pod runs the busybox minimal container and executes an env bash command and immediately exists. The db-ip-addresses key from the db-config map is mapped to the environment variable, DB_IP_ADDRESSES, and is reflected in the output:
> kubectl logs some-pod HUE_REMINDERS_SERVICE_PORT=80 HUE_REMINDERS_PORT=tcp://10.0.0.238:80 KUBERNETES_PORT=tcp://10.0.0.1:443 KUBERNETES_SERVICE_PORT=443 HOSTNAME=some-pod SHLVL=1 HOME=/root HUE_REMINDERS_PORT_80_TCP_ADDR=10.0.0.238 HUE_REMINDERS_PORT_80_TCP_PORT=80 HUE_REMINDERS_PORT_80_TCP_PROTO=tcp DB_IP_ADDRESSES=1.2.3.4,5.6.7.8 HUE_REMINDERS_PORT_80_TCP=tcp://10.0.0.238:80 KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443 HUE_REMINDERS_SERVICE_HOST=10.0.0.238 PWD=/ KUBERNETES_SERVICE_HOST=10.0.0.1
In some cases, you may want to keep a transient state in memory. Distributed caching is a common case. Time-sensitive information is another one. For these use cases, there is no need for persistent storage and multiple pods accessed through a service may be just the right solution. We can use standard Kubernetes techniques such as labeling to identify pods that belong to the store redundant copies of the same state and expose it through a service. If a pod dies, Kubernetes will create a new one and until it catches up the other pods will serve the state. We can even use the pod anti-affinity Alpha feature to ensure that pods who maintain redundant copies of the same state are not scheduled to the same node.
Some stateful applications such as distributed databases or queues manage their state redundantly and sync their nodes automatically (we'll take a very deep look into Cassandra later). In these cases, it is important that pods are scheduled to separate nodes. It is also important that pods are scheduled to nodes with particular hardware configuration or are even dedicated for the stateful application. The DaemonSet feature is perfect for this use case. We can label a set of nodes and make sure that the stateful pods are scheduled on a one-by-one basis to the selected group of nodes.
If the stateful application can use effectively shared persistent storage, then using a persistent volume claim in each pod is the way to go, as we demonstrated in Chapter 7, Handling Kubernetes Storage. The stateful application will be presented with a mounted volume that looks just like a local filesystem.
The StatefulSet controller is a relatively new addition to Kubernetes (introduced as StatefulSet in Kubernetes 1.3 and renamed to StatefulSet in Kubernetes 1.5). It is especially designed to support distributed stateful applications where the identities of the members is important and if a pod is restarted it must retain its identity in the set. It provides ordered deployment and scaling. Unlike regular pods, the pods of a stateful set are associated with persistent storage.
StatefulSet is great for applications that require one or more of the following:
- Stable, unique network identifiers
- Stable, persistent storage
- Ordered, graceful deployment, and scaling
- Ordered, graceful deletion, and termination
There are several pieces that need to be configured correctly in order to have a working StatefulSet:
- A headless service responsible for managing the network identity of the StatefulSet pods
- The StatefulSet itself with a number of replicas
- Persistent storage provision dynamically or by an administrator
Here is an example of a service called nginx that will be used for a StatefulSet:
apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx
Now, the StatefulSet configuration file will reference the service:
apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: web spec: serviceName: "nginx" replicas: 3 template: metadata: labels: app: nginx
The next part is the pod template that includes a mounted volume named www:
spec: terminationGracePeriodSeconds: 10 containers: - name: nginx image: gcr.io/google_containers/nginx-slim:0.8 ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html
Last but not least, the volumeClaimTemplates use a claim named www matching the mounted volume. The claim requests 1Gib of storage with ReadWriteOnce access:
volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gib