Computing resource management is so important in any infrastructure. We should know our application well and preserve enough CPU and memory capacity in order to prevent running out of resources. In this section, we'll introduce how to manage node capacity in the Kubernetes nodes. Furthermore, we'll also describe how to manage pod computing resources.
Before you start managing computing resources, you should know your applications well in order to know the maximum resources they need. Before we start, check out the current node capacity using the kubectl
command described in Chapter 1, Building Your Own Kubernetes:
// check current node capacity # kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, capacity: .status.capacity}' { "name": "kube-node1", "capacity": { "cpu": "1", "memory": "1019428Ki", "pods": "40" } } { "name": "kube-node2", "capacity": { "cpu": "1", "memory": "1019428Ki", "pods": "40" } }
You should know currently, we have two nodes with 1
CPU and 1019428
bytes memory. The node capacity of the pods are 40
for each. Then, we can start planning. How much computing resource capacity is allowed to be used on a node? How much computing resource is used in running our containers?
When the Kubernetes scheduler schedules a pod running on a node, it will always ensure that the total limits of the containers are less than the node capacity. If a node runs out of resources, Kubernetes will not schedule any new containers running on it. If no node is available when you launch a pod, the pod will remain pending, since the Kubernetes scheduler will be unable to find any node that could run your desired pod.
Sometimes, we want to explicitly preserve some resources for other processes or future usage on the node. Let's say we want to preserve 200 MB on all my nodes. First, we'll need to create a pod and run the pause
container in Kubernetes. Pause is a container for each pod for forwarding the traffic. In this scenario, we'll create a resource reserver pod, which is basically doing nothing with a limit of 200 MB:
// configuration file for resource reserver # cat /etc/kubernetes/reserve.yaml apiVersion: v1 kind: Pod metadata: name: resource-reserver spec: containers: - name: resource-reserver image: gcr.io/google_containers/pause:0.8.0 resources: limits: memory: 200Mi
Since it's a pod infra container, we will not use kubectl to launch it. Note that we put it in the /etc/kubernetes/
folder. You could put it under different paths; write down the path and we'll need to add it into the kubelet config file to launch it. Find the kubelet config file you specified in the Configuring nodes recipe in Chapter 1, Building Your Own Kubernetes and add the following argument when launching kubelet: --config=/etc/kubernetes/reserve.yaml
. Restart kubelet. After we restart kubelet, we will see the kubelet log in the node:
I0325 20:44:22.937067 21306 kubelet.go:1960] Starting kubelet main sync loop. I0325 20:44:22.937078 21306 kubelet.go:2012] SyncLoop (ADD): "resource-reserver-kube-node1_default" I0325 20:44:22.937138 21306 kubelet.go:2012] SyncLoop (ADD): "mynginx-e09bu_default" I0325 20:44:22.963425 21306 kubelet.go:2012] SyncLoop (ADD): "resource-reserver-kube-node1_default" I0325 20:44:22.964484 21306 manager.go:1707] Need to restart pod infra container for "resource-reserver-kube-node1_default" because it is not found
kubelet will check whether the infra container exists and create it accordingly:
// check pods list # kubectl get pods NAME READY STATUS RESTARTS AGE resource-reserver-kube-node1 1/1 Running 0 3m
The Kubernetes master is aware that the resource reserver pod has been created. Let's describe the details to get deeper insight:
# kubectl describe pods resource-reserver-kube-node1 Name: resource-reserver-kube-node1 Namespace: default Image(s): gcr.io/google_containers/pause:0.8.0 Node: kube-node1/10.0.0.224 Start Time: Fri, 25 Mar 2016 20:44:24 +0000 Labels: <none> Status: Running IP: 192.168.99.3 Replication Controllers: <none> Containers: resource-reserver: ... QoS Tier: memory: Guaranteed Limits: memory: 200Mi Requests: memory: 200Mi State: Running Started: Fri, 25 Mar 2016 20:44:24 +0000 Ready: True
We can find the limits and requests that are all set as 200Mi
; it means that this container has been reserved a minimum of 200 MB and a maximum of 200 MB. Repeat the same steps in your other nodes and check the status via the kubectl
command:
# kubectl get pods NAME READY STATUS RESTARTS AGE resource-reserver-kube-node1 1/1 Running 0 11m resource-reserver-kube-node2 1/1 Running 0 42m
Limits or requests?
The Kubernetes scheduler schedules a pod running on a node by checking the remaining computing resources. We could specify the limits or requests for each pod we launch. Limit means the maximum resources this pod can occupy. Request means the minimum resources this pod needs. We could use the following inequality to represent their relation: 0 <= request <= the resource this pod occupies <= limit <= node capacity.
The concept for managing the capacity in a pod or node is similar. They both specify the requests or limits under the container resource spec.
Let's create an nginx pod with certain requests and limits using kubectl create -f nginx-resources.yaml
to launch it:
# cat nginx-resources.yaml apiVersion: v1 kind: Pod metadata: name: nginx labels: name: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 resources: requests: cpu: 250m memory: 32Mi limits: cpu: 500m memory: 64Mi // create the pod # kubectl create -f nginx-resources.yaml pod "nginx" created
Following are the available resources for this pod:
Please note that the minimum CPU limit is set to 10 millicore. You cannot specify a value less than the minimum limit. Let's get more details via kubectl:
# kubectl describe pod nginx Name: nginx Namespace: default Image(s): nginx Node: kube-node1/10.0.0.224 Start Time: Fri, 25 Mar 2016 21:12:43 +0000 Labels: name=nginx Status: Running Reason: Message: IP: 192.168.99.4 Replication Controllers: <none> Containers: nginx: ... QoS Tier: cpu: Burstable memory: Burstable Limits: memory: 64Mi cpu: 500m Requests: cpu: 250m memory: 32Mi State: Running Started: Fri, 25 Mar 2016 21:12:44 +0000 Ready: True Restart Count: 0
Everything is expected. QoS Tier is Burstable
. Compared with Guaranteed
, Burstable
has a buffer to burst to the limits; however, Guaranteed
will always reserve certain resources for the pod. Please note that if you specify too many pods with Guaranteed
, cluster utilization would be poor, since it wastes the resources if the containers are not reaching the limits all the time.
In this section, you learned how to constrain computing resources in Kubernetes. We give more control to our containers. Check out the following recipes: