Authentication and authorization play a very vital role in securing applications. These two terms are often used interchangeably but are very different. Authentication validates the identity of a user. Once the identity is validated, authorization is used to check whether the user has the privileges to perform the desired action. Authentication uses something the user knows to verify their identity; in the simplest form, this is a username and password. Once the application verifies the user's identity, it checks what resources the user has access to. In most cases, this is a variation of an access control list. Access control lists for the user are compared with the request attributes to allow or deny an action.
In this chapter, we will discuss how a request is processed by authentication, authorization modules, and admission controllers before it is processed by kube-apiserver. We'll walk through the details of different modules and admission controllers and highlight the recommended security configurations.
We will finally look at Open Policy Agent (OPA), which is an open source tool that can be used to implement authorization across microservices. In Kubernetes, we will look at how it can be used as a validating admission controller. Many clusters require a more granular level of authorization than what is already provided by Kubernetes. With OPA, developers can define custom authorization policies that can be updated at runtime. There are several open source tools that leverage OPA, such as Istio.
In this chapter, we will discuss the following topics:
In Kubernetes, the kube-apiserver processes all requests to modify the state of the cluster. The kube-apiserver first verifies the origin of the request. It can use one or more authentication modules, including client certificates, passwords, or tokens. The request passes serially from one module to the other. If the request is not rejected by all the modules, it is tagged as an anonymous request. The API server can be configured to allow anonymous requests.
Once the origin of the request is verified, it passes through the authorization modules to check whether the origin of the request is permitted to perform the action. The authorization modules allow the request if a policy permits the user to perform the action. Kubernetes supports multiple authorization modules, such as Attribute-Based Access Control (ABAC), Role-Based Access Control (RBAC), and webhooks. Similar to authentication modules, a cluster can use multiple authorizations:
After passing through the authorization and authentication modules, admission controllers modify or reject the requests. Admission controllers intercept requests that create, update, or delete an object in the admission controller. Admission controllers fall into two categories: mutating or validating. Mutating admission controllers run first; they modify the requests they admit. Validating admission controllers run next. These controllers cannot modify objects. If any of the admission controllers reject a request, an error is returned to the user and the request will not be processed by the API server.
All requests in Kubernetes originate from external users, service accounts, or Kubernetes components. If the origin of the request is unknown, it is treated as an anonymous request. Depending on the configuration of the components, anonymous requests can be allowed or dropped by the authentication modules. In v1.6+, anonymous access is allowed to support anonymous and unauthenticated users for the RBAC and ABAC authorization modes. It can be explicitly disabled by passing the --anonymous-auth=false flag to the API server configuration:
$ps aux | grep api
root 3701 6.1 8.7 497408 346244 ? Ssl 21:06 0:16 kube-apiserver --advertise-address=192.168.99.111 --allow-privileged=true --anonymous-auth=false
Kubernetes uses one or more of these authentication strategies. Let's discuss them one by one.
Using X509 Certificate Authority (CA) certificates is the most common authentication strategy in Kubernetes. It can be enabled by passing --client-ca-file=file_path to the server. The file passed to the API server has a list of CAs, which creates and validates client certificates in the cluster. The common name property in the certificate is often used as the username for the request and the organization property is used to identify the user's groups:
kube-apiserver --advertise-address=192.168.99.104 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/var/lib/minikube/certs/ca.crt
To create a new certificate, the following steps need to be taken:
openssl genrsa -out priv.key 4096
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
CN = test
O = dev
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment extendedKeyUsage=serverAuth,clientAuth
You can generate a CSR using openssl:
openssl req -config ./csr.cnf -new -key priv.key -nodes -out new.csr
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: mycsr
spec:
groups:
- system:authenticated
request: ${BASE64_CSR}
usages:
- digital signature
- key encipherment
- server auth
- client auth
The certificate-signing request generated earlier is used with the preceding YAML specification to generate a new Kubernetes certificate-signing request:
$ export BASE64_CSR=$(cat ./new.csr | base64 | tr -d ' ')
$ cat csr.yaml | envsubst | kubectl apply -f -
Once this request is created, it needs to be approved by the cluster administrators to generate the certificate:
kubectl certificate approve mycsr
kubectl get csr mycsr -o jsonpath='{.status.certificate}'
| base64 --decode > new.crt
Next, we will look at static tokens, which are a popular mode of authentication in development and debugging environments but should not be used in production clusters.
The API server uses a static file to read the bearer tokens. This static file is passed to the API server using --token-auth-file=<path>. The token file is a comma-separated file consisting of secret, user, uid, group1, and group2.
The token is passed as an HTTP header in the request:
Authorization: Bearer 66e6a781-09cb-4e7e-8e13-34d78cb0dab6
The tokens persist indefinitely, and the API server needs to be restarted to update the tokens. This is not a recommended authentication strategy. These tokens can be easily compromised if the attacker is able to spawn a malicious pod in a cluster. Once compromised, the only way to generate a new token is to restart the API server.
Next, we will look at basic authentication, a variation of static tokens that has been used as a method for authentication by web services for many years.
Similar to static tokens, Kubernetes also supports basic authentication. This can be enabled by using basic-auth-file=<path>. The authentication credentials are stored in a CSV file as password, user, uid, group1, and group2.
The username and password are passed as an authentication header in the request:
Authentication: Basic base64(user:password)
Similar to static tokens, basic authentication passwords cannot be changed without restarting the API server. Basic authentication should not be used in production clusters.
Bootstrap tokens are an improvisation over the static tokens. Bootstrap tokens are the default authentication method used in Kubernetes. They are dynamically managed and stored as secrets in kube-system. To enable bootstrap tokens, do the following:
$ps aux | grep api
root 3701 3.8 8.8 497920 347140 ? Ssl 21:06 4:58 kube-apiserver --advertise-address=192.168.99.111 --allow-privileged=true --anonymous-auth=true --authorization-mode=Node,RBAC --client-ca-file=/var/lib/minikube/certs/ca.crt --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true
$ ps aux | grep controller
root 3693 1.4 2.3 211196 94396 ? Ssl 21:06 1:55 kube-controller-manager --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-name=mk --cluster-signing-cert-file=/var/lib/minikube/certs/ca.crt --cluster-signing-key-file=/var/lib/minikube/certs/ca.key --controllers=*,bootstrapsigner,tokencleaner
Authorization: Bearer 123456.aa1234fdeffeeedf
The first part of the token is the TokenId value and the second part of it is the TokenSecret value. TokenController ensures that expired tokens are deleted from the system secrets.
The service account authenticator is automatically enabled. It verifies signed bearer tokens. The signing key is specified using --service-account-key-file. If this value is unspecified, the Kube API server's private key is used:
$ps aux | grep api
root 3711 27.1 14.9 426728 296552 ? Ssl 04:22 0:04 kube-apiserver --advertise-address=192.168.99.104 ... --secure-port=8443 --service-account-key-file=/var/lib/minikube/certs/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/var/lib/minikube/certs/apiserver.crt --tls-private-key-file=/var/lib/minikube/certs/apiserver.key
docker 4496 0.0 0.0 11408 544 pts/0 S+ 04:22 0:00 grep api
Service accounts are created by the kube-apiserver and are associated with the pods. This is similar to instance profiles in AWS. The default service account is associated with a pod if no service account is specified.
To create a service account test, you can use the following:
kubectl create serviceaccount test
The service account has associated secrets, which includes the CA of the API server and a signed token:
$ kubectl get serviceaccounts test -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2020-03-29T04:35:58Z"
name: test
namespace: default
resourceVersion: "954754"
selfLink: /api/v1/namespaces/default/serviceaccounts/test
uid: 026466f3-e2e8-4b26-994d-ee473b2f36cd
secrets:
- name: test-token-sdq2d
If we enumerate the details, we can see the certificate and the token:
$ kubectl get secret test-token-sdq2d -o yaml
apiVersion: v1
data:
ca.crt: base64(crt)
namespace: ZGVmYXVsdA==
token: base64(token)
kind: Secret
Next, we will talk about webhook tokens. Some enterprises have a remote authentication and authorization server, which is often used across all services. In Kubernetes, developers can use webhook tokens to leverage the remote services for authentication.
In webhook mode, Kubernetes makes a call to a REST API outside the cluster to determine the user's identity. Webhook mode for authentication can be enabled by passing --authorization-webhook-config-file=<path> to the API server.
Here is an example of a webhook configuration. In this, authn.example.com/authenticate is used as the authentication endpoint for the Kubernetes cluster:
clusters:
- name: name-of-remote-authn-service
cluster:
certificate-authority: /path/to/ca.pem
server: https://authn.example.com/authenticate
Let's look at another way that a remote service can be used for authentication.
kube-apiserver can be configured to identify users using the X-Remote request header. You can enable this method by adding the following arguments to the API server:
--requestheader-username-headers=X-Remote-User
--requestheader-group-headers=X-Remote-Group
--requestheader-extra-headers-prefix=X-Remote-Extra-
Each request has the following headers to identify them:
GET / HTTP/1.1
X-Remote-User: foo
X-Remote-Group: bar
X-Remote-Extra-Scopes: profile
The API proxy validates the requests using the CA.
Cluster administrators and developers can use user impersonation to debug authentication and authorization policies for new users. To use user impersonation, a user must be granted impersonation privileges. The API server uses impersonation the following headers to impersonate a user:
Once the impersonation headers are received by the API server, the API server verifies whether the user is authenticated and has the impersonation privileges. If yes, the request is executed as the impersonated user. kubectl can use the --as and --as-group flags to impersonate a user:
kubectl apply -f pod.yaml --as=dev-user --as-group=system:dev
Once the authentication modules verify the identity of a user, they parse the request to check whether the user is allowed to access or modify the request.
Authorization determines whether a request is allowed or denied. Once the origin of the request is identified, active authorization modules evaluate the attributes of the request against the authorization policies of the user to allow or deny a request. Each request passes through the authorization module sequentially and if any module provides a decision to allow or deny, it is automatically accepted or denied.
Authorization modules parse a set of attributes in a request to determine whether the request should be parsed, allowed, or denied:
Now, let's look at the different authorization modes that use these request attributes to determine whether the origin is allowed to initiate the request.
Let's look at the different authorization modes available in Kubernetes.
Node authorization mode grants permissions to kubelets to access services, endpoints, nodes, pods, secrets, and persistent volumes for a node. The kubelet is identified as part of the system:nodes group with a username of system:node:<name> to be authorized by the node authorizer. This mode is enabled by default in Kubernetes.
The NodeRestriction admission controller, which we'll learn about later in this chapter, is used in conjunction with the node authorizer to ensure that the kubelet can only modify objects on the node that it is running. The API server uses the --authorization-mode=Node flag to use the node authorization module:
$ps aux | grep api
root 3701 6.1 8.7 497408 346244 ? Ssl 21:06 0:16 kube-apiserver --advertise-address=192.168.99.111 --allow-privileged=true --anonymous-auth=true --authorization-mode=Node,RBAC --client-ca-file=/var/lib/minikube/certs/ca.crt --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
Node authorization is used in conjunction with ABAC or RBAC, which we will look at next.
With ABAC, requests are allowed by validating policies against the attributes of the request. ABAC authorization mode can be enabled by using --authorization-policy-file=<path> and --authorization-mode=ABAC with the API server.
The policies include a JSON object per line. Each policy consists of the following:
An example policy is as follows:
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user": "kubelet", "namespace": "*", "resource": "pods", "readonly": true}}
This policy allows a kubelet to read any pods. ABAC is difficult to configure and maintain. It is not recommended that you use ABAC in production environments.
With RBAC, access to resources is regulated using roles assigned to users. RBAC is enabled by default in many clusters since v1.8. To enable RBAC, start the API server with --authorization-mode=RBAC:
$ ps aux | grep api
root 14632 9.2 17.0 495148 338780 ? Ssl 06:11 0:09 kube-apiserver --advertise-address=192.168.99.104 --allow-privileged=true --authorization-mode=Node,RBAC ...
RBAC uses Role, which is a set of permissions, and RoleBinding, which grants permissions to users. Role and RoleBinding are restricted to namespaces. If a role needs to span across namespaces, ClusterRole and ClusterRoleBinding can be used to grant permissions to users across namespace boundaries.
Here is an example of a Role property that allows a user to create and modify pods in the default namespace:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: default
name: deployment-manager
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
The corresponding RoleBinding can be used with Role to grant permissions to the user:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: binding
namespace: default
subjects:
- kind: User
name: employee
apiGroup: ""
roleRef:
kind: Role
name: deployment-manager
apiGroup: ""
Once RoleBinding is applied, you can switch the context to see whether it worked correctly:
$ kubectl --context=employee-context get pods
NAME READY STATUS RESTARTS AGE
hello-node-677b9cfc6b-xks5f 1/1 Running 0 12m
However, if you try to view the deployments, it will result in an error:
$ kubectl --context=employee-context get deployments
Error from server (Forbidden): deployments.apps is forbidden: User "employee" cannot list resource "deployments" in API group "apps" in the namespace "default"
Since roles and role bindings are restricted to the default namespace, accessing the pods in a different namespace will result in an error:
$ kubectl --context=employee-context get pods -n test
Error from server (Forbidden): pods is forbidden: User "test" cannot list resource "pods" in API group "" in the namespace "test"
$ kubectl --context=employee-context get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "test" cannot list resource "pods" in API group "" in the namespace "kube-system"
Next, we will talk about webhooks, which provide enterprises with the ability to use remote servers for authorization.
Similar to webhook mode for authentication, webhook mode for authorization uses a remote API server to check user permissions. Webhook mode can be enabled by using --authorization-webhook-config-file=<path>.
Let's look at a sample webhook configuration file that sets https://authz.remote as the remote authorization endpoint for the Kubernetes cluster:
clusters:
- name: authz_service
cluster:
certificate-authority: ca.pem
server: https://authz.remote/
Once the request is passed by the authentication and authorization modules, admission controllers process the request. Let's discuss admission controllers in detail.
Admission controllers are modules that intercept requests to the API server after the request is authenticated and authorized. The controllers validate and mutate the request before modifying the state of the objects in the cluster. A controller can be both mutating and validating. If any of the controllers reject the request, the request is dropped immediately and an error is returned to the user so that the request will not be processed.
Admission controllers can be enabled by using the --enable-admission-plugins flag:
$ps aux | grep api
root 3460 17.0 8.6 496896 339432 ? Ssl 06:53 0:09 kube-apiserver --advertise-address=192.168.99.106 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/var/lib/minikube/certs/ca.crt --enable-admission-plugins=PodSecurityPolicy,NamespaceLifecycle,LimitRanger --enable-bootstrap-token-auth=true
Default admission controllers can be disabled using the --disable-admission-plugins flag.
In the following sections, we will look at some important admission controllers.
This admission controller allows all the pods to exist in the cluster. This controller has been deprecated since 1.13 and should not be used in any cluster. With this controller, the cluster behaves as if no controllers exist in the cluster.
This controller ensures that new pods always force image pull. This is helpful to ensure updated images are used by pods. It also ensures that private images can only be used by users who have the privileges to access them since users without access cannot pull images when a new pod is started. This controller should be enabled in your clusters.
Denial-of-service attacks are common in infrastructure. Misbehaving objects can also cause high consumption of resources, such as the CPU or network, resulting in increased cost or low availability. EventRateLimit is used to prevent these scenarios.
The limit is specified using a config file, which can be specified by adding a --admission-control-config-file flag to the API server.
A cluster can have four types of limits: Namespace, Server, User and SourceAndObject. With each limit, the user can have a maximum limit for the Queries Per Second (QPS), the burst and cache size.
Let's look at an example of a configuration file:
limits:
- type: Namespace
qps: 50
burst: 100
cacheSize: 200
- type: Server
qps: 10
burst: 50
cacheSize: 200
This adds the qps, burst, and cacheSize limits to all API servers and namespaces.
Next, we will talk about LimitRanger, which prevents the overutilization of resources available in the cluster.
This admission controller observes the incoming request and ensures that it does not violate any of the limits specified in the LimitRange object.
An example of a LimitRange object is as follows:
apiVersion: "v1"
kind: "LimitRange"
metadata:
name: "pod-example"
spec:
limits:
- type: "Pod"
max:
memory: "128Mi"
With this limit range object, any pod requesting memory of more than 128 Mi will fail:
pods "range-demo" is forbidden maximum memory usage per Pod is 128Mi, but limit is 1073741824
When using LimitRanger, malicious pods cannot consume excess resources.
This admission controller restricts the pods and nodes that a kubelet can modify. With this admission controller, a kubelet gets a username in the system:node:<name> format and is only able to modify the node object and pods running on its own node.
This admission controller adds validations for the PersistentVolumeClaimResize requests.
This admission controller runs on the creation or modification of pods to determine whether the pods should be run based on the security-sensitive configuration of the pods. The set of conditions in the policy is checked against the workload configuration to verify whether the workload creation request should be allowed. A PodSecurityPolicy can check for fields such as privileged, allowHostPaths, defaultAddCapabilities, and so on. You'll learn more about PodSecurityPolicy in the next chapter.
This is the recommended admission controller to use if PodSecurityPolicy is not enabled. It restricts the settings of security-sensitive fields, which can cause privilege escalation, such as running a privileged pod or adding Linux capabilities to a container:
$ ps aux | grep api
root 3763 6.7 8.7 497344 345404 ? Ssl 23:28 0:14 kube-apiserver --advertise-address=192.168.99.112 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/var/lib/minikube/certs/ca.crt --enable-admission-plugins=SecurityContextDeny
It is recommended that PodSecurityPolicy is enabled by default in a cluster. However, due to the administrative overhead, SecurityContextDeny can be used until PodSecurityPolicy is configured for the cluster.
ServiceAccount is an identity of the pod. This admission controller implements ServiceAccount; it should be used if the cluster uses service accounts.
Similar to webhook configurations for authentication and authorization, webhooks can be used as admission controllers. MutatingAdmissionWebhook modifies the workload's specifications. These hooks execute sequentially. ValidatingAdmissionWebhook parses the incoming request to verify whether it is correct. Validating hooks execute simultaneously.
Now, we have looked at authentication, authorization, and admission control of resources in Kubernetes. Let's look at how developers can implement fine-grained access control in their clusters. In the next section, we talk about OPA, an open source tool that is used extensively in production clusters.
OPA is an open source policy engine that allows policy enforcement in Kubernetes. Several open source projects, such as Istio, utilize OPA to provide finer-grained controls. OPA is an incubating project hosted by Cloud Native Computing Foundation (CNCF).
OPA is deployed as a service alongside your other services. To make authorization decisions, the microservice makes a call to OPA to decide whether the request should be allowed or denied. Authorization decisions are offloaded to OPA, but this enforcement needs to be implemented by the service itself. In Kubernetes environments, it is often used as a validating webhook:
To make a policy decision, OPA needs the following:
Let's look at an example of how OPA can be leveraged to deny the creation of pods with a busybox image. You can use the official OPA documentation (https://www.openpolicyagent.org/docs/latest/kubernetes-tutorial/) to install OPA on your cluster.
Here is the policy that restricts the creation and updating of pods with the busybox image:
$ cat pod-blacklist.rego
package kubernetes.admission
import data.kubernetes.namespaces
operations = {"CREATE", "UPDATE"}
deny[msg] {
input.request.kind.kind == "Pod"
operations[input.request.operation]
image := input.request.object.spec.containers[_].image
image == "busybox"
msg := sprintf("image not allowed %q", [image])
}
To apply this policy, you can use the following:
kubectl create configmap pod —from-file=pod-blacklist.rego
Once configmap is created, kube-mgmt loads these policies out of configmap in the opa container, both kube-mgmt and opa containers are in the opa pod. Now, if you try to create a pod with the busybox image, you get the following:
$ cat busybox.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- name: sec-ctx-demo
image: busybox
command: [ "sh", "-c", "sleep 1h" ]
This policy checks the request for the busybox image name and denies creation of pods with the busybox image with an image not allowed error:
admission webhook "validating-webhook.openpolicyagent.org" denied the request: image not allowed "busybox"
Similar to the admission controller that we discussed previously, further finer-grained admission controllers can be created using OPA in the Kubernetes cluster.
In this chapter, we looked at the importance of authentication and authorization in Kubernetes. We discussed the different modules available for authentication and authorization and discussed these modules in detail, as well as going through detailed examples of how each module is used. When looking at authentication, we discussed user impersonation, which can be used by cluster administrators or developers to test permissions. Next, we talked about admission controllers, which can be used to validate or mutate requests after authentication and authorization. We also discussed some admission controllers in detail. Finally, we looked at OPA, which can be used in Kubernetes clusters to perform a more fine-grained level of authorization.
Now, you should be able to devise appropriate authentication and authorization strategies for your cluster. You should be able to figure out which admission controllers work for your environment. In many cases, you'll need more granular controls for authorization, which can be provided by using OPA.
In the next chapter, we will take a deep dive into securing pods. The chapter will cover some of the topics that we covered in this chapter in more detail, such as PodSecurityPolicy. Securing pods is essential to securing application deployment in Kubernetes.
You can refer to the following links for more information: