Mutual TLS states that all services must use TLS when communicating with other services. This uncovers one of the big security holes in Kubernetes. A bad actor who has access to the cluster, even if they don't have access to the namespace, can send commands to any pod, pretending to be a legitimate service. If given enough rights, they can also operate as the man in the middle between services, grabbing JSON Web Tokens (JWTs). Implementing TLS between services reduces the chances of man-in-the-middle attacks between services:

1. To enable mutual TLS globally, run the following command:

cat <<EOF | kubectl apply -f -
apiVersion: "authentication.istio.io/v1alpha1"
kind: "MeshPolicy"
metadata:
  name: "default"
spec:
  peers:
  - mtls: {}
EOF

Since it is named default, it specifies that all workloads in the mesh will only accept encrypted requests using TLS.

2. Now run the following command:

for from in "foo" "bar"; do for to in "foo" "bar"; do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/ip -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}
"; done; done

Those systems with sidecars will fail when running this command and will receive a 503 code, as the client is still using plain text. It might take a few seconds for MeshPolicy to take effect. The following is the output:

sleep.foo to httpbin.foo: 503
sleep.foo to httpbin.bar: 503
sleep.bar to httpbin.foo: 503
sleep.bar to httpbin.bar: 503

3. We will set the destination rule to use a * wildcard that is similar to the mesh-wide authentication policy. This is required to configure the client side:

cat <<EOF | kubectl apply -f -
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "default"
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF

Running the preceding command will make all the pods with the sidecar communicate via TLS.

4. We can check this by running the same command again:

for from in "foo" "bar"; do for to in "foo" "bar"; do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/ip -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}
"; done; done

This time, the returned codes will be 200:

sleep.foo to httpbin.foo: 200
sleep.foo to httpbin.bar: 200
sleep.bar to httpbin.foo: 200
sleep.bar to httpbin.bar: 200

5. We can also check that the pods without the istio sidecar cannot access any services in the foo or bar namespaces by running the following command:

for from in "legacy"; do for to in "foo" "bar"; do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.${to}:8000/ip -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}
"; done; done

The result will be as follows:

sleep.legacy to httpbin.foo: 000
command terminated with exit code 56
sleep.legacy to httpbin.bar: 000
command terminated with exit code 56

Using some simple commands, we were able to dramatically increase the security of our applications without changing the application code. We achieved the goal of setting system-wide policies with operators without having developers be worried about them.