Even though a pod is the most fine-grained unit that serves as a placeholder to run microservices, securing Kubernetes pods is a vast topic as it should cover the entire DevOps flow: build, deployment, and runtime.
In this chapter, we choose to narrow our focus to the build and runtime stages. To secure Kubernetes pods in the build stage, we will talk about how to harden a container image and configure the security attributes of pods (or pod templates) to reduce the attack surface. Although some of the security attributes of workloads, such as AppArmor and SELinux labels, take effect in the runtime stage, security control has already been defined for the workload. To clarify matters further, we're trying to secure Kubernetes workloads by configuring the runtime effect security attributes in the build stage. To secure Kubernetes pods in the runtime stage, we will introduce a PodSecurityPolicy with examples along with the facilitating tool, kube-psp-advisor.
Later chapters will go into more detail regarding runtime security and response. Also note that exploitation of the application may lead to pods getting compromised. However, we don't intend to cover application in this chapter.
In this chapter, we will cover the following topics:
Container image hardening means to follow security best practices or baselines to configure a container image in order to reduce the attack surface. Image scanning tools only focus on finding publicly disclosed issues in applications bundled inside the image. But, following the best practices along with secure configuration while building the image ensures that the application has a minimal attack surface.
Before we start talking about the secure configuration baseline, let's look at what a container image is, as well as a Dockerfile, and how it is used to build an image.
A container image is a file that bundles the microservice binary, its dependencies, and configurations of the microservice, and so on. A container is a running instance of an image. Nowadays, application developers not only write code to build microservices; they also need to build the Dockerfile to containerize the microservice. To help build a container image, Docker offers a standardized approach, known as a Dockerfile. A Dockerfile contains a series of instructions, such as copy files, configure environment variables, configure open ports, and container entry points, which can be understood by the Docker daemon to construct the image file. Then, the image file will be pushed to the image registry from where the image is then deployed in Kubernetes clusters. Each Dockerfile instruction will create a file layer in the image.
Before we look at an example of a Dockerfile, let's understand some basic Dockerfile instructions:
Now, let's take a look at an example of a Dockerfile:
FROM ubuntu
# install dependencies
RUN apt-get install -y software-properties-common python
RUN add-apt-repository ppa:chris-lea/node.js
RUN echo "deb http://us.archive.ubuntu.com/ubuntu/ precise universe" >> /etc/apt/sources.list
RUN apt-get update
RUN apt-get install -y nodejs
# make directory
RUN mkdir /var/www
# copy app.js
ADD app.js /var/www/app.js
# set the default command to run
CMD ["/usr/bin/node", "/var/www/app.js"]
From the preceding Dockerfile, we can tell that the image was built on top of ubuntu. Then, it ran a bunch of apt-get commands to install the dependencies, and created a directory called /var/www. Next, copy the app.js file from the current directory to /var/www/app.js in the filesystem of the image. Finally, configure the default command to run this Node.js application. I believe you will see how straightforward and powerful Dockerfile is when it comes to helping you build an image.
The next question is any security concern, as it looks like you're able to build any kind of image. Next, let's talk about CIS Docker benchmarks.
Center for Internet Security (CIS) put together a guideline regarding Docker container administration and management. Now, let's take a look at the security recommendations from CIS Docker benchmarks regarding container images:
If you follow the security recommendations from the preceding CIS Docker benchmarks, you will be successful in hardening your container image. This is the first step in securing pods in the build stage. Now, let's look at the security attributes we need to pay attention to in order to secure a pod.
As we mentioned in the previous chapter, application developers should be aware of what privileges a microservice must have in order to perform tasks. Ideally, application developers and security engineers work together to harden the microservice at the pod and container level by configuring the security context provided by Kubernetes.
We classify the major security attributes into four categories:
By employing such a means of classification, you will find them easy to manage.
The following attributes in the pod specification are used to configure the use of host namespaces:
The following is an example of how to configure the use of host namespaces at the pod level in an ubuntu-1 pod YAML file:
apiVersion: v1
kind: Pod
metadata:
name: ubuntu-1
labels:
app: util
spec:
containers:
- name: ubuntu
image: ubuntu
imagePullPolicy: Always
hostPID: true
hostNetwork: true
hostIPC: true
The preceding workload YAML configured the ubuntu-1 pod to use a host-level PID namespace, network namespace, and IPC namespace. Keep in mind that you shouldn't set these attributes to true unless necessary. Setting these attributes to true also disarms the security boundaries of other workloads in the same worker node, as has already been mentioned in Chapter 5, Configuring Kubernetes Security Boundaries.
Multiple containers can be grouped together inside the same pod. Each container can have its own security context, which defines privileges and access controls. The design of a security context at a container level provides a more fine-grained security control for Kubernetes workloads. For example, you may have three containers running inside the same pod and one of them has to run in privileged mode, while the others run in non-privileged mode. This can be done by configuring a security context for individual containers.
The following are the principal attributes of a security context for containers:
You may add extra capabilities or drop some of the defaults by configuring this attribute. Capabilities such as CAP_SYS_ADMIN and CAP_NETWORK_ADMIN should be added with caution. For the default capabilities, you should also drop those that are unnecessary.
Since you now understand what these security attributes are, you may come up with your own hardening strategy aligned with your business requirements. In general, the security best practices are as follows:
Now, let's take a look at an example of configuring SecurityContext for containers:
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod
labels:
app: web
spec:
hostNetwork: false
hostIPC: false
hostPID: false
containers:
- name: nginx
image: kaizheh/nginx
securityContext:
privileged: false
capabilities:
add:
- NETWORK_ADMIN
readOnlyRootFilesystem: true
runAsUser: 100
runAsGroup: 1000
The nginx container inside nginx-pod runs as a user with a UID of 100 and a GID of 1000. In addition to this, the nginx container gains extra NETWORK_ADMIN capability and the root filesystem is set to read-only. The YAML file here only shows an example of how to configure the security context. Note that adding NETWORK_ADMIN is not recommended for containers running in production environments.
A security context is used at the pod level, which means that security attributes will be applied to all the containers inside the pod.
The following is a list of the principal security attributes at the pod level:
Notice that the attributes runAsUser, runAsGroup, runAsNonRoot, and seLinuxOptions are available both in SecurityContext at the container level and PodSecurityContext at the pod level. This gives users both the flexibility and extreme importance of security control. fsGroup and sysctls are not as commonly used as the others, so only use them when you have to.
An AppArmor profile usually defines what Linux capabilities the process owns, what network resources and files can be accessed by the container, and so on. In order to use an AppArmor profile to protect pods or containers, you will need to update the annotation of the pod. Let's look at an example, assuming you have an AppArmor profile to block any file write activities:
#include <tunables/global>
profile k8s-apparmor-example-deny-write flags=(attach_disconnected) {
#include <abstractions/base>
file,
# Deny all file writes.
deny /** w,
}
Note that AppArmor is not a Kubernetes object, like a pod, deployment, and so on. It can't be operated through kubectl. You will have to SSH to each node and load the AppArmor profile into the kernel so that the pod may be able to use it.
The following is the command for loading the AppArmor profile:
cat /etc/apparmor.d/profile.name | sudo apparmor_parser -a
Then, put the profile into enforce mode:
sudo aa-enforce /etc/apparmor.d/profile.name
Once the AppArmor profile is loaded, you can update the annotation of the pod to use the AppArmor profile to protect your container. Here is an example of applying an AppArmor profile to containers:
apiVersion: v1
kind: Pod
metadata:
name: hello-apparmor
annotations:
# Tell Kubernetes to apply the AppArmor profile
# "k8s-apparmor-example-deny-write".
container.apparmor.security.beta.kubernetes.io/hello:
localhost/k8s-apparmor-example-deny-write
spec:
containers:
- name: hello
image: busybox
command: [ "sh", "-c", "echo 'Hello AppArmor!' && sleep 1h" ]
The container inside hello-apparmor does nothing but sleep after echoing the Hello AppArmor! message. When it is running, if you launch a shell from a container and write to any file, it will be blocked by AppArmor. Even though writing a robust AppArmor profile is not easy, you can still create some basic restrictions, such as denying writing to certain directories, denying accepting raw packets, and making certain files read-only. Also, test the profile first before applying it to the production cluster. Open source tools such as bane can help create AppArmor profiles for containers.
We do not intend to dive into the seccomp profile in this book since writing a seccomp profile for a microservice is not easy. Even an application developer doesn't have knowledge of what system calls are legitimate for the microservice they developed. Although you can turn the audit mode on to avoid breaking the microservice's functionality, building a robust seccomp profile is still a long way off. Another reason is that this feature is still in the alpha stage up to version 1.17. According to Kubernetes' official documentation, being alpha means it is disabled by default, perhaps buggy, and only recommended to run in a short-lived testing cluster. When there are any new updates on seccomp, we may come back to introduce seccomp in more detail at a later date.
We've covered how to secure Kubernetes pods in the build time. Next, let's look at how we can secure Kubernetes pods during runtime.
A Kubernetes PodSecurityPolicy is a cluster-level resource that controls security-sensitive aspects of the pod specification through which the access privileges of a Kubernetes pod are limited. As a DevOps engineer, you may want to use a PodSecurityPolicy to restrict most of the workloads run in limited access privileges, while only allowing a few workloads to be run with extra privileges.
In this section, we will first take a closer look at a PodSecurityPolicy, and then we will introduce an open source tool, kube-psp-advisor, which can help build an adaptive PodSecurityPolicy for the running Kubernetes cluster.
You can think of a PodSecurityPolicy as a policy to evaluate the security attributes defined in the pod's specification. Only those pods whose security attributes meet the requirements of PodSecurityPolicy will be admitted to the cluster. For example, PodSecurityPolicy can be used to block the launch of most privileged pods, while only allowing those necessary or limited pods access to the host filesystem.
The following are the principal security attributes that are controlled by PodSecurityPolicy:
Now, let's take a look at an example of a PodSecurityPolicy:
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: example
spec:
allowedCapabilities:
- NET_ADMIN
- IPC_LOCK
allowedHostPaths:
- pathPrefix: /dev
- pathPrefix: /run
- pathPrefix: /
fsGroup:
rule: RunAsAny
hostNetwork: true
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- hostPath
- secret
This PodSecurityPolicy allows the NET_ADMIN and IPC_LOCK capabilities, mounts /, /dev, and /run from the host and Kubernetes' secret volumes. It doesn't enforce any filesystem group ID or supplemental groups and it also allows the container to run as any user, access the host network namespace, and run as a privileged container. No SELinux policy is enforced in the policy.
To enable this Pod Security Policy, you can run the following command:
$ kubectl apply -f example-psp.yaml
Now, let's verify that the Pod Security Policy has been created successfully:
$ kubectl get psp
The output will appear as follows:
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
example true NET_ADMIN, IPC_LOCK RunAsAny RunAsAny RunAsAny RunAsAny false hostPath,secret
After you have created the Pod Security Policy, there is one more step required in order to enforce it. You will have to grant the privilege of using the PodSecurityPolicy object to the users, groups, or service accounts. By doing so, the pod security policies are entitled to evaluate the workloads based on the associated service account. Here is an example of how to enforce a PodSecurityPolicy. First, you will need to create a cluster role that uses the PodSecurityPolicy:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: use-example-psp
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- example
Then, create a RoleBinding or ClusterRoleBinding object to associate the preceding ClusterRole object created with the service accounts, users, or groups:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: use-example-psp-binding
roleRef:
kind: ClusterRole
name: use-example-psp
apiGroup: rbac.authorization.k8s.io
subjects:
# Authorize specific service accounts:
- kind: ServiceAccount
name: test-sa
namespace: psp-test
The preceding use-example-pspbinding.yaml file created a RoleBinding object to associate the use-example-psp cluster role with the test-sa service account in the psp-test namespace. With all of these set up, any workloads in the psp-test namespace whose service account is test-sa will run through the PodSecurityPolicy example's evaluation. And only those that meet the requirements will be admitted to the cluster.
From the preceding example, think of there being different types of workloads running in your Kubernetes cluster, and each of them may require different privileges to access different types of resources. It would be a challenge to create and manage pod security policies for different workloads. Now, let's take a look at kube-psp-advisor and see how it can help create pod security policies for you.
Kubernetes PodSecurityPolicy Advisor (also known as kube-psp-advisor) is an open source tool from Sysdig. It scans the security attributes of running workloads in the cluster and then, on this basis, recommends pod security policies for your cluster or workloads.
First, let's install kube-psp-advisor as a kubectl plugin. If you haven't installed krew, a kubectl plugin management tool, please follow the instructions (https://github.com/kubernetes-sigs/krew#installation) in order to install it. Then, install kube-psp-advisor with krew as follows:
$ kubectl krew install advise-psp
Then, you should be able to run the following command to verify the installation:
$ kubectl advise-psp
A way to generate K8s PodSecurityPolicy objects from a live K8s environment or individual K8s objects containing pod specifications
Usage:
kube-psp-advisor [command]
Available Commands:
convert Generate a PodSecurityPolicy from a single K8s Yaml file
help Help about any command
inspect Inspect a live K8s Environment to generate a PodSecurityPolicy
Flags:
-h, --help help for kube-psp-advisor
--level string Log level (default "info")
To generate pod security policies for workloads in a namespace, you can run the following command:
$ kubectl advise-psp inspect --grant --namespace psp-test
The preceding command generates pod security policies for workloads running inside the psp-test namespace. If the workload uses a default service account, no PodSecurityPolicy will be generated for it. This is because the default service account will be assigned to the workload that does not have a dedicated service account associated with it. And you certainly don't want to have a default service account that is able to use a PodSecurityPolicy for privileged workloads.
Here is an example of output generated by kube-psp-advisor for workloads in the psp-test namespace, including Role, RoleBinding, and PodSecurityPolicy in a single YAML file with multiple pod security policies. Let's take a look at one of the recommended PodSecurityPolicy:
# Pod security policies will be created for service account 'sa-1' in namespace 'psp-test' with following workloads:
# Kind: ReplicaSet, Name: busy-rs, Image: busybox
# Kind: Pod, Name: busy-pod, Image: busybox
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
creationTimestamp: null
name: psp-for-psp-test-sa-1
spec:
allowedCapabilities:
- SYS_ADMIN
allowedHostPaths:
- pathPrefix: /usr/bin
readOnly: true
fsGroup:
rule: RunAsAny
hostIPC: true
hostNetwork: true
hostPID: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- configMap
- secret
- hostPath
Following is the Role generated by kube-psp-advisor:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
creationTimestamp: null
name: use-psp-by-psp-test:sa-1
namespace: psp-test
rules:
- apiGroups:
- policy
resourceNames:
- psp-for-psp-test-sa-1
resources:
- podsecuritypolicies
verbs:
- use
---
Following is the RoleBinding generated by kube-psp-advisor:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: null
name: use-psp-by-psp-test:sa-1-binding
namespace: psp-test
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: use-psp-by-psp-test:sa-1
subjects:
- kind: ServiceAccount
name: sa-1
namespace: psp-test
---
The preceding section is the recommended PodSecurityPolicy, psp-for-psp-test-sa-1, for the busy-rs and busy-pod workloads, since these two workloads share the same service account, sa-1. Hence, Role and RoleBinding are created to use the Pod Security Policy, psp-for-psp-test-sa-1, respectively. The PodSecurityPolicy is generated based on the aggregation of the security attributes of workloads using the sa-1 service account:
---
# Pod security policies will NOT be created for service account 'default' in namespace 'psp-test' with following workdloads:
# Kind: ReplicationController, Name: busy-rc, Image: busybox
---
The preceding section mentions that the busy-rc workload uses a default service account, so there is no Pod Security Policy created for it. This is a reminder that if you want to generate pod security policies for workloads, don't use the default service account.
Building a Kubernetes PodSecurityPolicy is not straightforward, although it would be ideal if a single restricted PodSecurityPolicy was to apply to the entire cluster and all workloads complied with it. DevOps engineers need to be creative in order to build restricted pod security policies while not breaking workloads' functionalities. kube-psp-advisor makes the implementation of Kubernetes pod security policies simple, adapts to your application requirements and, specifically, is fine-grained for each one to allow only the privilege of least access.
In this chapter, we covered how to harden a container image with CIS Docker benchmarks, and then we gave a detailed introduction to the security attributes of Kubernetes workloads. Next, we looked at the PodSecurityPolicy in detail and introduced the kube-psp-advisor open source tool, which facilitates the establishment of pod security policies.
Securing Kubernetes workloads is not a one-shot thing. Security controls need to be applied from the build, deployment, and runtime stages. It starts with hardening container images, and then configuring security attributes of Kubernetes workloads in a secure way. This happens at the build stage. It is also important to build adaptive pod security policies for different Kubernetes workloads. The goal is to restrict most of the workloads to run with limited privileges, while allowing only a few workloads to run with extra privileges, and without breaking workload availability. This happens at the runtime stage. kube-psp-advisor is able to help build adaptive pod security policies.
In the next chapter, we will talk about image scanning. It is critical in helping to secure Kubernetes workloads in the DevOps workflow.
You can refer to the following links for more information on the topics covered in this chapter: