6

Securing Microservices Communication

Istio secures communication between microservices without microservices requiring any code changes. In Chapter 4, we briefly touched upon the topic of security. We configured transport layer security by exposing our sockshop application over HTTPS. We created certificates and configured the Istio Ingress gateway to bind those certificates to hostnames in SIMPLE TLS mode. We also implemented TLS-based security for multiple hosts managed by a single Ingress gateway.

In this chapter, we will dive deeper into some advanced topics of security. We will start by understanding Istio security architecture. We will implement mutual TLS for service communication with other services in the mesh, and we will also implement mutual TLS with downstream clients outside the mesh. We will then perform various hands-on exercises to create custom security policies for authentication and authorization. We will go through these topics in the following order:

• Istio security architecture
• Authenticating using mutual TLS
• How to configure a custom authentication and authorization policy

Important note

The technical prerequisites for this chapter are the same as Chapters 4 and 5.

Understanding Istio security architecture

In Chapter 3, we discussed how the Istio control plane is responsible for the injection of sidecars and establishing trust so that sidecars can communicate with the control plane securely and security policies are eventually enforced by the sidecar. When deployed in Kubernetes, Istio relies on Kubernetes service accounts to identify the roles of workloads in a Service Mesh. The Istio CA watches the Kubernetes API server for the addition/deletion/modification of any service accounts in the namespace with Istio injection enabled. It creates a key and certificates for each service account and, during Pod creation, the certificate and key are mounted onto the sidecar. The Istio CA is responsible for managing the life cycle of the certificates distributed to the sidecars, including the rotation and management of private keys. Using the Secure Production Identity Framework for Everyone (SPIFFE) format identities, Istio provides a strong identity to each service along with service naming, which represents the role that can be taken up by the identity assigned to the service.

SPIFFE is a set of open source standards for software identity. SPIFFE provides platform-agnostic interoperable software identities along with interfaces and documents required to obtain and validate cryptographic identity in a fully automated fashion.

In Istio, each workload is automatically assigned an identity represented in the X.509 certificate format. The creation and signing of certificate signing request (CSRs) are managed by the Istio control plane, as discussed in Chapter 3. The X.509 certificate follows the SPIFFE format.

Let’s redeploy the envoydummy service and inspect the envoydummy Pods:

$ kubectl apply -f Chapter6/01-envoy-dummy.yaml
$ istioctl proxy-config all po/envoydummy-2-7488b58cd7-m5vpv -n utilities -o json | jq -r '.. |."secret"?' | jq -r 'select(.name == "default")' | jq -r '.tls_certificate.certificate_chain.inline_bytes' | base64 -d - | step certificate inspect  --short
X.509v3 TLS Certificate (RSA 2048) [Serial: 3062...1679]
  Subject:     spiffe://cluster.local/ns/utilities/sa/default
  Issuer:
  Valid from:  2022-09-11T22:18:13Z
          to:  2022-09-12T22:20:13Z

step CLI

You will need to install step CLI to be able to run the preceding command. To install it, please follow the documentation at https://smallstep.com/docs/step-cli.

In the output of the preceding command, you will notice that the Subject Alternative Name (SAN) is spiffe://cluster.local/ns/utilities/sa/default. This is the SPIFFE ID, which functions as the unique name:

• spifee is the URI scheme
• cluster.local is the trust domain
• /ns/utilities/sa/default is the URI identifying the service account associated with the workload:
  • ns stands for namespace
  • sa stands for service account

The value default for service accounts comes from the service account attached to the workload. In our example of envoydummy, we didn’t associate any service accounts so, by default, Kubernetes associated the default service account. You can find the service account name associated with a Pod using the following command:

kubectl get po/envoydummy-2-7488b58cd7-m5vpv -n utilities -o json | jq .spec.serviceAccountName
"default"

You will notice that default is the default name for service accounts associated with all Pods in all namespaces, such as sock-shop, utilities, and so on. Kubernetes creates a service account named default in every namespace:

% kubectl get sa -n utilities
NAME      SECRETS   AGE
default   1         13d
% kubectl get sa -n sock-shop
NAME      SECRETS   AGE
default   1         27d

Kubernetes service accounts

A service account is an identity assigned to workloads in Kubernetes. When processes running inside a workload try to access other Kubernetes resources, they are identified and authenticated as per the details of their service accounts. You can find more details about service accounts at https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/.

Secure Naming is a technology that decouples the name of services from the identities the services are running as. In the previous example, spiffe://cluster.local/ns/utilities/sa/default is the identity of the service presented during mutual TLS presented by the istio-proxy sidecar in envoydummy-2-7488b58cd7-m5vpv workload. From the SPIFFE ID, the other party (istio-proxy in another Pod) in the MTLS session can validate that the endpoint has the identity of a service account named default in the utilities namespace. The Istio control plane propagates the secure naming information to all sidecars in the mesh and during mutual TLS, the sidecar not only verifies that the identity is correct but also that the respective service is assuming the correct identity.

The following diagram summarizes the Istio security architecture:

Figure 6.1 – Istio security architecture

These are the key concepts to remember:

• The Istio CA manages keys and certificates and the SANs in certificates are in SPIFFE format
• Istiod distributes authentication and authorization security policies to all sidecars in the mesh
• Sidecars enforce authentication and authorization as per security policies distributed by Istiod

Reminder

Make sure to clean up Chapter6/01-envoy-dummy.yaml to avoid conflict in upcoming exercises.

In the next section, we will read about how to secure data in transit between microservices in a Service Mesh.

Authentication using mutual TLS

Mutual TLS (mTLS) is a technique for authenticating two parties at each end of a network connection. Through mTLS, each party can verify that the other party is what they are claiming to be. Certificate authorities play a critical role in mTLS, and hence we had the previous section on Istio security architecture describing certificate authorities and secure naming in Istio.

mTLS is one of the most frequently used authentication mechanisms for implementing the zero-trust security framework, in which no party trusts another party by default, irrespective of where the other party is placed in the network. Zero trust assumes that there are no traditional network edges and boundaries and hence every party needs to be authenticated and authorized. This helps to eliminate many security vulnerabilities that arise because of the assumption-based trust model.

In the following two subsections, we will look at how Istio helps you implement mTLS for service-to-service authentication inside a mesh, also called east-west traffic, and mTLS between client/downstream systems that are outside the mesh, with services in the mesh called north-south communication.

Service-to-service authentication

Istio provides service-to-service authentication by using mTLS for transport authentication. During traffic processing, Istio performs the following:

• All outbound traffic from Pods is rerouted to istio-proxy.
• istio-proxy starts an mTLS handshake with the server-side istio-proxy. During the handshake, it also does a secure naming check to verify that the service account presented in the server certificate can run the Pod.
• The server-side istio-proxy verifies the client-side istio-proxy in the same fashion and if all is okay, a secure channel is established between the two proxies.

Istio provides the following two options when implementing mTLS:

• Permissive mode: In permissive mode, Istio allows traffic in both mTLS and non-mTLS mode. This feature is primarily to improve the onboarding of clients to mTLS. Clients who are not yet ready to communicate over mTLS can continue communicating over TLS with the view that they will eventually migrate to mTLS whenever they are ready.
• Strict mode: In strict mode, Istio enforces strict mTLS and any non-mTLS traffic is not allowed.

Mutual TLS traffic can be established between clients outside of a mesh trying to access a workload within the mesh, as well as clients within the mesh trying to access other workloads in the mesh. For the former, we will discuss the details in the next section. For the latter set of clients, we will go through some examples in this section.

Let’s set up service-to-service communication using mTLS:

1. Create a namespace called chapter6 with Istio injection enabled and deploy the httpbin service:

   $ kubectl apply -f Chapter6/01-httpbin-deployment.yaml

Most of the config in this deployment is the usual, except that we have also created a default Kubernetes service account called httpbin in the Chapter6 namespace:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: chapter6

The httpbin identity is then assigned to an httpbin Pod by following these specs:

  Spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

2. Next, we will create a client in the form of a curl Pod to access the httpbin service. Create a utilities namespace with Istio injection disabled, and create a curl Deployment with its own service account:

   $ kubectl apply -f Chapter6/01-curl-deployment.yaml

Make sure the istio-injection label is not applied. If it is, you can remove it using the following command:

$ kubectl label ns utilities istio-injection-

3. From the curl Pod, try to access the httpbin Pod and you should get a response back:

   $ kubectl exec -it curl -n utilities – curl -v http://httpbin.chapter6.svc.cluster.local:8000/get
   {
     "args": {},
     "headers": {
       "Accept": "*/*",
       "Host": "httpbin.chapter6.svc.cluster.local:8000",
       "User-Agent": "curl/7.87.0-DEV",
       "X-B3-Sampled": "1",
       "X-B3-Spanid": "a00a50536c3ec2f5",
       "X-B3-Traceid": "49b6942c85c7c1f2a00a50536c3ec2f5"
     },
     "origin": "127.0.0.6",
     "url": "http://httpbin.chapter6.svc.cluster.local:8000/get"
4. So far, we have the httpbin Pod running in the mesh, but by default, in permissive TLS mode. We will now create a PeerAuthentication policy to enforce STRICT mTLS. The PeerAuthentication policy defines how traffic will be tunneled via sidecars:

   apiVersion: security.istio.io/v1beta1
   kind: PeerAuthentication
   metadata:
     name: "httpbin-strict-tls"
     namespace: chapter6
   spec:
     mtls:
       mode: STRICT
     selector:
       matchLabels:
         app: httpbin

In the PeerAuthentication policy, we defined the following configuration parameters:

• mtls: This defines the mTLS setting. If not specified, the value is inherited from the default mesh-wide setting. It has one field called mode, which can have the following values:
  • UNSET: With this value, the mTLS settings are inherited from the parent and if the parent does not have any settings, then the value is set to PERMISSIVE.
  • MTLS: With this value, the sidecar accepts both mTLS and non-mTLS connections.
  • STRICT: This enforces strict mTLS – any non-mTLS connection will be dropped.
  • DISABLE: mTLS is disabled and connections are not tunneled.
• Selector: This defines the criteria that need to be satisfied by a workload to be part of this per authentication policy. It has a field named matchLabels, which takes label information in the key:value format.

To summarize the configuration, we have created httpbin-strict-tls, which is a PeerAuthentication policy in the Chapter6 namespace. The policy enforces string mTLS for all workloads that have a label of app=httpbin. The configuration is available at Chapter6/02-httpbin-strictTLS.yaml.

5. Apply the changes via the following command:

   $ kubectl apply -f Chapter6/02-httpbin-strictTLS.yaml
   peerauthentication.security.istio.io/httpbin-strict-tls created
6. Now try to connect to the httpbin service from the curl Pod:

   $ kubectl exec -it curl -n utilities – curl -v http://httpbin.chapter6.svc.cluster.local:8000/get
   * Connected to httpbin.chapter6.svc.cluster.local (172.20.147.104) port 8000 (#0)
   > GET /get HTTP/1.1
   > Host: httpbin.chapter6.svc.cluster.local:8000
   > User-Agent: curl/7.87.0-DEV
   > Accept: */*
   >
   * Recv failure: Connection reset by peer
   * Closing connection 0
   curl: (56) Recv failure: Connection reset by peer
   command terminated with exit code 56

curl is not able to connect because the curl Pod is running in a namespace with Istio injection disabled, whereas the httpbin Pod is running in the mesh with the PeerAuthentication policy enforcing STRICT mTLS. One option is to manually establish an mTLS connection, which is equivalent to modifying your application code to perform mTLS. In this case, as we are trying to simulate service communication within the mesh, we can simply turn on Istio injection and let Istio take care of client-side mTLS as well.

7. Enable Istio injection for the curl Pod using the following steps:
   1. Delete the resource created by Chapter6/01-curl-deployment.yaml.
   2. Modify the value of Istio injection to be enabled.
   3. Apply the updated configuration.

Once the curl Pod is in the RUNNING state, along with the istio-proxy sidecar, you can perform curl on the httpbin service and you will see the following output:

$ kubectl exec -it curl -n utilities -- curl -s http://httpbin.chapter6.svc.cluster.local:8000/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.chapter6.svc.cluster.local:8000",
    "User-Agent": "curl/7.85.0-DEV",
    "X-B3-Parentspanid": "a35412ed46b7ec46",
    "X-B3-Sampled": "1",
    "X-B3-Spanid": "0728b578e88b72fb",
    "X-B3-Traceid": "830ed3d5d867a460a35412ed46b7ec46",
    "X-Envoy-Attempt-Count": "1",
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/chapter6/sa/
httpbin;Hash=b1b88fe241c557bd1281324b458503274eec3f04b1d439758508842d6d5b7018;Subject="";URI=spiffe://cluster.local/ns/utilities/sa/curl"
  },
  "origin": "127.0.0.6",
  "url": "http://httpbin.chapter6.svc.cluster.local:8000/get"
}

In the response from the httpbin service, you will notice all the headers that were received by the httpbin Pod. The most interesting header is X-Forwarded-Client-Cert, also called XFCC. There are two parts of the XFCC header value that shed light on mTLS:

• By: This is filled with the SAN, which is the SPIFFE ID of the istio-proxy’s client certificate of the httpbin Pod (spiffe://cluster.local/ns/chapter6/sa/httpbin)
• URI: This contains the SAN, which is the SPIFFE ID of the curl Pod’s client certificate presented during mTLS (spiffe://cluster.local/ns/utilities/sa/curl)

There is also Hash, which is the SHA256 digest of istio-proxy’s client certificate of the httpbin Pod.

You can selectively apply mTLS configuration at the port level also. In the following configuration, we are implying that mTLS is enforced strictly for all ports except port 8080, which should allow permissive connections. The configuration is available at Chapter6/03-httpbin-strictTLSwithException.yaml:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: "httpbin-strict-tls"
  namespace: chapter6
spec:
  portLevelMtls:
    8080:
      mode: PERMISSIVE
    8000:
      mode: STRICT
  selector:
    matchLabels:
      app: httpbin

So, in this section, we learned how to perform mTLS between services inside the mesh. mTLS can be enabled at the service level as well as at the port level. In the next section, we will read about performing mTLS with clients outside the mesh.

Reminder

Make sure to clean up Chapter6/01-httpbin-deployment.yaml, Chapter6/01-curl-deployment.yaml and Chapter6/02-httpbin-strictTLS.yaml to avoid conflict in upcoming exercises.

Authentication with clients outside the mesh

For clients outside the mesh, Istio supports mTLS with the Istio Ingress gateway. In Chapter 5, we configured HTTPS at the Ingress gateway. In this section, we will extend that configuration to also support mTLS.

We will now configure mTLS for the httpbin Pod. Notice that the first five steps are very similar to steps 1-5 of Exposing Ingress over HTTPS of Chapter 5. The steps are as follows:

1. Create a CA. Here, we are creating a CA with a Common Name (CN) of sock.inc:

   $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=sock Inc./CN=sock.inc' -keyout sock.inc.key -out sock.inc.crt
2. Generate a CSR for httpbin.org. Here, we are generating a Certificate Signing Request (CSR) for httpbin.org, which also generates a private key:

   $ openssl req -out httpbin.org.csr -newkey rsa:2048 -nodes -keyout httpbin.org.key -subj "/CN=httpbin.org/O=sockshop.inc"
   Generating a 2048 bit RSA private key
   .........+++
   ..+++
   writing new private key to 'httpbin.org.key'
3. Sign the CSR using the CA created in step 1:

   $ openssl x509 -req -sha256 -days 365 -CA sock.inc.crt -CAkey sock.inc.key -set_serial 0 -in httpbin.org.csr -out httpbin.org.crt
   Signature ok
   subject=/CN=httpbin.org/O=sockshop.inc
   Getting CA Private Key
4. Load the certificate and private key as a Kubernetes secret along with the CA certificate against which client certificates must be verified:

   $ kubectl create -n istio-system secret generic httpbin-credential --from-file=tls.key=httpbin.org.key --from-file=tls.crt=httpbin.org.crt --from-file=ca.crt=sock.inc.crt
   secret/httpbin-credential created
5. Configure the Ingress gateway to enforce mTLS for all incoming connections and use the secret created in step 4 as the secret containing TLS certificate and the CA certificate:

     tls:
         mode: MUTUAL
         credentialName: httpbin-credential
6. Deploy the httpbin Pod, the Ingress gateway, and the virtual service:

   $ kubectl apply -f Chapter6/02-httpbin-deployment-MTLS.yaml
7. To perform mTLS, you also need to generate client certificates that can be used to prove the client’s identity. For that, perform the following steps:

   $ openssl req -out bootstrapistio.packt.com.csr -newkey rsa:2048 -nodes -keyout bootstrapistio.packt.com.key -subj "/CN= bootstrapistio.packt.com/O=packt.com"
   $ openssl x509 -req -sha256 -days 365 -CA sock.inc.crt -CAkey sock.inc.key -set_serial 0 -in bootstrapistio.packt.com.csr -out bootstrapistio.packt.com.crt
8. Test the connection to httpbin.org by passing client certificates in the request:

   % curl -v -HHost:httpbin.org --connect-to "httpbin.org:443:a816bb2638a5e4a8c990ce790b47d429-1565783620.us-east-1.elb.amazonaws.com" --cacert sock.inc.crt --cert bootstrapistio.packt.com.crt --key bootstrapistio.packt.com.key https://httpbin.org:443/get

Reminder

Don’t forget to clean up using kubectl delete -n istio-system secret httpbin-credential and kubectl delete -f Chapter6/02-httpbin-deployment-MTLS.yaml.

Configuring RequestAuthentication

Like service-to-service authentication, Istio can also authenticate an end user or validate that an end user has been authenticated based on assertions presented by the end user. The RequestAuthentication policy is used to specify what authentication methods are supported by a workload. This policy identifies the authenticated identity but doesn’t enforce whether the request should be allowed or denied. Rather, it provides information about the authenticated identity to the authorization policy, which we will go through in the next section. In this section, we will learn how to make use of the Istio RequestAuthentication policy to validate an end user who has been authenticated by Auth0 and is providing a bearer token as security credentials to Istio. If you are not familiar with OAuth then you can read more about it at https://auth0.com/docs/authenticate/protocols/oauth.

We will follow the hands-on steps to configure Auth0 and perform an OAuth flow while at the same time demystifying all that is happening under the hood:

1. Sign up for Auth0:

Figure 6.2 – Auth0 signup

2. After signing up, create an application in Auth0:

Figure 6.3 – Create application in Auth0

3. After creating the application, you will need to create an API. You can provide the Ingress URL as the identifier:

Figure 6.4 – Create an API in Auth0

4. Declare permissions the consumer of this API needs to have to be able to access the API:

Figure 6.5 – API scopes in Auth0

5. Enable RBAC for the API from General Settings:

Figure 6.6 – Enable RBAC for the API in Auth0

6. After creating the API, go back to the application and authorize the application for access to the EnvoyDummy API, and while doing so, please also configure scopes:

Figure 6.7 – Grant permission to the application in Auth0

7. As the last step, go to the application page to get the request you can use to get the access token:

Figure 6.8 – Quickstart example to get the access token in Auth0

Copy the curl string, including client_id, client_secret, and so on, and with this, we have completed all the steps in Auth0.

Now, using the curl string you copied in the previous steps, get the access token from the terminal:

$ curl --request POST --url https://dev-0ej7x7k2.us.auth0.com/oauth/token --header 'content-type:application/json' --data '{"client_id":"XXXXXX-id","client_secret":"XXXXX-secret"," "audience":"http://a816bb2638a5e4a8c990ce790b47d429-1565783620.us-east-1.elb.amazonaws.com/","grant_type":"client_credentials"}'
{"access_token":"xxxxxx-accesstoken" "scope":"read:dummyData","expires_in":86400,"token_type":"Bearer"}%

Once we have received the access token, we will apply the RequestAuthentication policy. The RequestAuthentication policy specifies the details of how to validate the JWT provided during authentication. Following is the RequestAuthentication policy:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: "auth0"
  namespace: chapter6
spec:
  selector:
    matchLabels:
      name: envoydummy
  jwtRules:
  - issuer: "https://dev-0ej7x7k2.us.auth0.com/"
    jwksUri: "https://dev-0ej7x7k2.us.auth0.com/.well-known/jwks.json"

In the preceding configuration, also available in Chapter6/01-requestAuthentication.yaml, we are declaring a RequestAuthentication policy with the name auth0 in the chapter6 namespace with the following specifications:

• issuer: This is the value of the domain of the Auth0 application. You can fetch the value from the following screen:

Figure 6.9 – Application domain

• jwksUri: This is the JWKS endpoint, which can be used by Istio to verify the signature. Auth0 exposes a JWKS endpoint for each tenant, which is found at https://DOMAIN/.well-known/jwks.json. This endpoint will contain the JWK used to verify all Auth0-issued JWTs for this tenant. Replace the DOMAIN value with the value in the application.

When using the RequestAuthentication policy, it is best practice to also configure the RequestAuthentication and AuthorizationPolicy together and enforce a rule that any request with an empty principal should not be allowed. Following is an example of a sample authorization policy – you will read more about authorization policies in the next section:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: auth0-authz
  namespace: chapter6
spec:
  action: DENY
  selector:
    matchLabels:
      name: envoydummy
  rules:
  - from:
    - source:
        notPrincipals: ["*"]

Configuring RequestAuthorization

In the previous section, we configured a RequestAuthentication policy, which verifies a JWT token against the issuer and JWK details as per the JWKS location. We configured Auth0 as the authentication provider and the one that generates the bearer token. In this section, we will learn about how to make use of the information provided by authentication policies such as peer authentication and request authentication to authorize client access to the server (the requested resource, Pod, workload, service, etc.).

We will first focus on implementing an authorization policy in conjunction with the RequestAuthentication policy from the previous section.

To let curl access the envoy dummy using the access token issued by Auth0, we need to create an AuthorizationPolicy:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "envoydummy-authz-policy"
  namespace: utilities
spec:
  action: ALLOW
  selector:
    matchLabels:
      name: envoydummy
  rules:
  - when:
    - key: request.auth.claims[permissions]
      values: ["read:profile"]

AuthorizationPolicy contains the following data:

• action: This defines the type of action to be taken when the request matches the defined rule. Possible values for action are ALLOW, DENY, and AUDIT.
• selector: This defines what workload this policy should be applied to. Here, you provide a set of labels that should match the workload’s label to be part of a selection.
• rules: Here, we are defining a set of rules that should be matched with the request. Rules contain the following sub-configurations:
  • source: This provides the rule about the origin of the request.
  • to: This provides rules about the request such as to what host it was addressed, what the method name is, and what resource identified by the URI is requested.
  • when: This specifies a list of additional conditions. You can find a detailed list of all parameters at https://istio.io/latest/docs/reference/config/security/conditions/.

In this example, we are defining an authorization policy that allows access to Pods with the label name:envoydummy if the request contains an authenticated JWT token with a claim of read:profile.

Before we apply the changes, make sure that you can access the dummy data and make sure you have the Ingress gateway and envoydummy Pods deployed in the utilities namespace – if not, you can do that by applying the following commands:

$ curl -Hhost:mockshop.com http://a816bb2638a5e4a8c990ce790b47d429-1565783620.us-east-1.elb.amazonaws.com/
V1----------Bootstrap Service Mesh Implementation with Istio----------V1%

Go ahead and apply both of the policies:

% kubectl apply -f Chapter6/01-requestAuthentication.yaml
requestauthentication.security.istio.io/auth0 created
% kubectl apply -f Chapter6/02-requestAuthorization.yaml
authorizationpolicy.security.istio.io/envoydummy-authz-policy created

Check that you are able to access mockshop.com:

% curl -Hhost:mockshop.com http://a816bb2638a5e4a8c990ce790b47d429-1565783620.us-east-1.elb.amazonaws.com/
RBAC: access denied

The access is denied because we need to provide a valid access token as part of the request. Copy the access token you got from the previous request and try again in the following fashion:

$ curl -Hhost:mockshop.com -H "authorization: Bearer xxxxxx-accesstoken " http://a816bb2638a5e4a8c990ce790b47d429-1565783620.us-east-1.elb.amazonaws.com/
RBAC: access denied%

Although the JWT verification succeeded, the request failed due to RBAC controls. The error is deliberate because instead of providing read:dummyData in Chapter6/02-requestAuthorization.yaml, we provide read:profile. The changes are updated in Chapter6/03-requestAuthorization.yaml. Apply the changes and test the APIs:

% kubectl apply -f Chapter6/03-requestAuthorization.yaml
authorizationpolicy.security.istio.io/envoydummy-authz-policy configured
$ curl -Hhost:mockshop.com -H "authorization: Bearer xxxxxx-accesstoken " http://a816bb2638a5e4a8c990ce790b47d429-1565783620.us-east-1.elb.amazonaws.com/
V2----------Bootstrap Service Mesh Implementation with Istio----------V2%

To summarize, we did the following, including the previous section:

1. We configured Auth0 as the authentication provider and OAuth server.
2. We created a RequestAuthentication policy to validate the bearer token provided in the request.
3. We created an AuthorizationPolicy to verify claims presented in the JWT token and whether the claim matched the desired value, then let the request go through the upstream.

Next, we will learn how to configure request authorization in conjunction with PeerAuthentication, which we configured in the Service-to-service authentication section.

We will modify the curl Pod to use a different service account, and let’s call it chapter6sa:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: chapter6sa
  namespace: utilities

As you cannot change the service account of an existing Pod, you need to delete the previous deployment and redeploy with a new service account:

Kubectl delete -f Chapter6/01-curl-deployment.yaml
kubectl apply -f Chapter6/02-curl-deployment.yaml

You can check that the curl Pod is running with the identity of the chapter6sa service account. After this, let’s create an AuthorizationPolicy to allow a request to the httpbin Pod if the principal of the requestor is cluster.local/ns/utilities/sa/curl:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "httpbin-authz-policy"
  namespace: chapter6
spec:
  action: ALLOW
  selector:
    matchLabels:
      app: httpbin
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/utilities/sa/curl"]
    to:
    - operation:
        methods: ['*']

Previously, we looked at AuthorizationPolicy and you will be familiar with most of the configuration in this example. In this example, we are building AuthorizationPolicy on top of peer authentication rather than request authentication. The most interesting part is the source field in the rules section. In the source configuration, we define the source identities of the request. All fields in the source request need to match for the rule to be successful.

The following fields can be defined in the source configuration:

• principals: This is a list of accepted identities that are derived from the client certificate during mTLS. The values are in the <TRUST_DOMAIN NAME >/ns/<NAMESPACE NAME>/sa/<SERVICE_ACCOUNT NAME> format. In this example, the value of principals will be cluster.local/ns/utilities/sa/curl.
• notPrincipals: This is a list of identities from which the request will not be accepted. The values are derived the same way as principals.
• requestPrincipals: This is a list of accepted identities where the request principal is derived from JWT and is in the format <ISS>/<SUB>.
• notRequestPrincipals: This is a list of identities from which the request will not be accepted. The principal is derived from the JWT and is in the format <ISS>/<SUB>.
• namespaces: This is a list of namespaces from which the request will be accepted. The namespaces are derived from the peer certificate details.
• notNamespaces: This is a list of namespaces from which a request will not be allowed. The namespaces are derived from the peer certificate details.
• ipBlocks: This is a list of IPs or CIDR blocks from which a request will be accepted. The IP is populated from the source address of the IP packet.
• notIpBlocks: This is a list of IP blocks from which a request will be rejected.
• remoteIpBlocks: This is a list of IP blocks, populated from the X-Forwarded-For header or proxy protocol.
• notRemoteIpBlocks: This is a negative list of remoteIpBlocks.

Go ahead and apply the configuration and test whether you are able to curl to httpbin:

$ kubectl apply -f Chapter6/04-httpbinAuthorizationForSpecificSA.yaml
authorizationpolicy.security.istio.io/httpbin-authz-policy configured
$ kubectl exec -it curl -n utilities -- curl -v http://httpbin.chapter6.svc.cluster.local:8000/headers
*   Trying 172.20.152.62:8000...
* Connected to httpbin.chapter6.svc.cluster.local (172.20.152.62) port 8000 (#0)
> GET /headers HTTP/1.1
> Host: httpbin.chapter6.svc.cluster.local:8000
> User-Agent: curl/7.85.0-DEV
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< content-length: 19
< content-type: text/plain
< date: Tue, 20 Sep 2022 02:20:39 GMT
< server: envoy
< x-envoy-upstream-service-time: 15
<
* Connection #0 to host httpbin.chapter6.svc.cluster.local left intact
RBAC: access denied%

Istio denies the request from the curl Pod to httpbin because the peer certificate presented by the curl Pod contains cluster.local/ns/utilities/sa/chapter6sa instead of cluster.local/ns/utilities/sa/curl as the principal. Although the curl Pod is part of the mesh and contains a valid certificate, it is not authorized to access the httpbin Pod.

Go ahead and fix the problem by assigning the correct service account to the curl Pod.

Tip

You can use the following commands to fix the problem:

$ kubectl delete -f Chapter6/02-curl-deployment.yaml
$ kubectl apply -f Chapter6/03-curl-deployment.yaml

We will implement one more authorization policy, but this time the policy will enforce that using the utilities or curl service account, the requestor can access only /headers of httpbin.

Following is the authorization policy:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "httpbin-authz-policy"
  namespace: chapter6
spec:
  action: ALLOW
  selector:
    matchLabels:
      app: httpbin
  rules:
  - from:
    - source:
        requestPrincipals: ["cluster.local/ns/utilities/sa/curl"]
  - to:
    - operation:
        methods: ["GET"]
        paths: ["/get"]

In this policy, we have defined the HTTP method and HTTP paths in the to field on the rule. The to field contains a list of operations on which the rules will be applied. The operation field supports the following parameters:

• hosts: This specifies a list of hostnames for which the request will be accepted. If it’s not set, then any host is allowed.
• notHosts: This is a negative list of hosts.
• ports: This is a list of ports for which a request will be accepted.
• notPorts: This is a list of negative matches of ports.
• methods: This is a list of methods as specified in the HTTP request. If not set, any method is allowed.
• notMethods: This is a list of negative matches of methods as specified in the HTTP request.
• paths: This is a list of paths as specified in the HTTP request. Paths are normalized as per https://istio.io/latest/docs/reference/config/security/normalization/.
• notPaths: This is a list of negative matches of paths.

Apply the changes:

kubectl apply -f Chapter6/05-httpbinAuthorizationForSpecificPath.yaml

And then try to access httpbin:

kubectl exec -it curl -n utilities -- curl -X GET -v http://httpbin.chapter6.svc.cluster.local:8000/headers
……
RBAC: access denied%

Access to the request is denied because the authorization policy only allows the /get request made using the HTTP GET method. The following is the correct request:

kubectl exec -it curl -n utilities -- curl -X GET -v http://httpbin.chapter6.svc.cluster.local:8000/get

This concludes our lesson on how you can build custom policies for performing request authentication and request authorization. To get more familiar with them, I suggest going through the examples in this chapter a few times and maybe building your own variations to learn how to use these policies effectively.

Summary

In this chapter, we read about how Istio provides authentication and authorization. We also read about how to implement service-to-service authentication using mutual TLS within a Service Mesh using the PeerAuthentication policy, as well as mutual TLS with clients external to a Service Mesh by using the mutual TLS mode at the Ingress gateway. We then read about end user authentication using the RequestAuthentication policy. We configured Auth0 to gain some real-life experience in using authentication and identity providers.

To finish off, we then read about AuthorizationPolicy and how it can be used to enforce various authorization checks to ensure that the authenticated identity is authorized to access the requested resources.

In the next chapter, we will read about how Istio helps in making microservices observable and how various observability tools and software can be integrated with Istio.