Tackling pod disruptions

Ideally, we'd like to keep the availability of our service as high as we can. However, there're always lots of events that cause the pods that are backing our service to go up and down, either voluntarily or involuntarily. Voluntary disruptions include Deployment rollouts, planned node maintenance, or the accidental killing of a pod with the API. On the whole, every operation that goes through the Kubernetes master counts. On the other hand, any unexpected outage that leads to the termination of our service belongs to the category of involuntary disruptions.

In previous chapters, we discussed how to prevent involuntary disruptions by replicating pods with Deployment and StatefulSet, appropriately configuring resource requests and limits, scaling an application's capacity with the autoscaler, and distributing pods to multiple locations with affinities and anti-affinities. Since we've already put a lot of effort into our service, what could go wrong when it comes to these expected voluntary disruptions? In fact, because they're events that are likely to happen, we ought to pay more attention to them.

In Deployment and other similar objects, we can use the maxUnavailable and maxSurge fields that help us roll out our updates in a controlled manner. As for other cases, such as node maintenance tasks performed by cluster administrators who don't have domain knowledge about all the applications run in the cluster, the service owner can utilize PodDisruptionBudget to tell Kubernetes how many pods are required for a service to meet its service level.

A pod disruption budget has the following syntax:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: <pdb name>
spec:
maxUnavailable: <desired number or percentage of pods>
minAvailable: <desired number or percentage of pods>
selector:
<matchLabels> or <matchExpressions>

There are two configurable fields in a pod disruption budget, but they can't be used together. The selector is identical to the one in Deployment or other places. Note that a pod disruption budget is immutable, which means it can't be updated after its creation. The minAvailable and maxUnavailable fields are mutually exclusive, but they're the same in some ways. For example, maxUnavailable:0 means zero tolerance of any pod losses, and it's roughly equivalent to minAvailable:100%, which means that all pods should be available.

Pod disruption budgets work by evicting events such as draining nodes or pod preemption. They don't interfere with the rolling update process performed by controllers such as Deployment or StatefulSet. Suppose that we want to temporarily remove one node from the cluster with kubectl drain, but this would violate certain pod disruption budgets of running applications. In this case, the draining operation would be blocked unless all pod disruption budgets can be satisfied. However, if the Kubernetes scheduler is going to evict a victim pod to fulfill high priority pods, the scheduler would only try to meet all of the pod disruption budgets if possible. If the scheduler can't find a target without breaking any pod disruption budgets, it would still pick a pod with the lowest priority.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset