Parlaying with kubectl

As noted in the context of troubleshooting Pilot in Chapter 7, Meshery and istioctl are powerful tools in assessing the synchronization of and coordination between Istio’s control-plane components and its data-plane service proxies. In Kubernetes environments, Meshery and istioctl both use kubectl exec under the hood. The invocation of kubectl exec involves execution over one of two HTTP streaming protocols from your local kubectl CLI to the Kubernetes API server on to the kubelet local to the node running the service proxy being interrogated by either Meshery or the istioctl command.

The specific mechanics of these two HTTP streaming protocols depend on which version of Kubernetes and which container runtime you’re using; the Kubernetes API server supports the SPDY protocol (now deprecated) and HTTP/2 WebSockets. (If WebSockets are unfamiliar, simply think of WebSockets as a protocol that transforms HTTP into a bidirectional byte-streaming protocol.) On top of that stream, the Kubernetes API server introduces an additional multiplexed streaming protocol. The reason for this is that, for many use cases, it is quite useful for the API server to be able to service multiple independent byte streams. Consider, for example, executing a command within a container. In this case, there are actually three streams that need to be maintained: stdin, stderr, and stdout.

When invoked, kubectl exec performs a sequence of actions. Initially, it issues an HTTP POST request to the Kubernetes API server at

/api/v1/namespaces/$NAMESPACE/pods/$NAME/exec

with a query string defining the command(s) to execute in which container and whether to establish multiplexed bidirectional streaming of stdin, stdout, and stderr:

?command=<command-syntax>&container=<name>&stderr=true&stdout=true

These query parameters are self-explanatory, indicating the command to run, if stdin should be enabled, if stdout should be enabled, if stderr should be enabled, and the container’s name. Using these query string parameters, a WebSocket connection is established and kube-apiserver will start streaming data between Meshery/istioctl and the respective kubelet, as shown in Figure 11-3.

Figure 11-3. How Meshery and istioctl parlay with kubectl to retrieve mesh configuration from service proxies

When you write to the WebSocket, the data will be passed to standard input (stdin) and on the receiving end of the WebSocket will be standard output (stdout) and error (stderr). The API defines a simple protocol to multiplex stdout and stderr over a single connection. Every message passed through the WebSocket is prefixed by a single byte that defines which stream the message belongs to (see Table 11-1).

The kube-apiserver establishes a connection to the kubelet located on the node where the pod in question resides. From there, the kubelet generates a short-lived token and issues a redirect to the container runtime interface (CRI). The CRI handles the kubectl exec request and issues a docker exec API call. The basic protocol channels of stdin, stdout, and stderr are specified as input, output, and error to the docker exec API call, respectively. kubectl exec/attach/logs/proxy might require long-running connections to the kube-apiserver since any of these commands might need data streamed over time rather than an immediate, one-time response.

Application Configuration

Avoid UID 1337. Ensure that your pods do not run applications as a user with the user ID (UID) value of 1337. The Istio service proxy uses 1337 as its UID. You’ll need to avoid this conflict.

Network Traffic and Ports

HTTP/1.1 or HTTP/2.0 is required. Applications must use either the HTTP/1.1 or HTTP/2.0 protocols for all its HTTP traffic; HTTP/1.0 is not supported.

Service ports must be named. To use Istio’s traffic routing, ensure each service has port name key/value pairs following this syntax: name: <protocol>[-<suffix>]. The value used for <protocol> should match one of the following types (strings):

• grpc

• http

• http2

• https

• mongo

• redis

• tcp

• tls

• udp

By default, Istio will treat traffic as TCP. Unless the port explicitly uses udp to signify UDP traffic, or unless the beginning of a service’s port name doesn’t match one of these prefixes, Istio will treat traffic on the port as TCP. Consequently, traffic on unnamed ports is treated as TCP, as well. Examples of valid port names are http2-myservice or http2; however, http2myservice is not valid.

Pods must include an explicit list of the ports on which each container listens. Use a containerPortconfiguration in the container specification for each port. Any unlisted ports bypass the Istio proxy.

Services and Deployments

Associate all pods to at least one service. Whether it exposes a port or not, all pods must belong to at least one Kubernetes service. For pods belonging to multiple Kubernetes services, ensure that each service defines the same type of protocol when referencing the same port number (for instance, HTTP and TCP).

Meaningful telemetry with Kubernetes labels. With app and version labels deployments, we recommend adding an explicit app label and version label. Add the labels to the deployment specification of pods deployed using the Kubernetes Deployment. The app and version labels add contextual information to the telemetry Istio collects:

An app label

Each deployment specification should have a distinct app label with a meaningful value.

A version label

Indicates the version of the application corresponding to the particular deployment.

These labels are helpful as context propagated in distributed traces.

Pods

Pod configuration must allow NET_ADMIN capability. If your cluster enforces pod security policies, pods must allow the NET_ADMIN capability. As described in Chapter 5, while injecting the service proxy sidecar, Istio uses an init container to manipulate the iptables rules of the pod in order to intercept requests to application containers. Although the service proxy sidecar does not require root to run, the short-lived init container does require cap_net_admin privileges to install iptables rules in each pod just prior to starting the pod’s primary containers onboard to the service mesh.

Manipulating iptables rules is an action that requires elevated access through the NET_ADMIN capability in the Linux kernel. Pods that have this kernel-level capability enabled can manipulate with the networking configuration of the other pods as well as can manipulate networking configuration on their host node. Most Kubernetes operators avoid allowing tenant pods to have this capability or at least those that operate a shared-tenant cluster.

If you use the Istio CNI Plugin, the NET_ADMIN capability requirement does not apply, because the CNI plug-in (a privileged pod) will perform administrative functions on behalf of the istio-init container. The following shows a check as to whether pod security policies are enabled in your cluster:

$ kubectl get psp
No resources found.

This example shows a cluster that has no pod security policies defined. If your cluster does have pod security policies defined, look for NET_ADMIN or * in the list of capabilities of the allowed policies for your given service account. Unless a Kubernetes service account is specified in your pods’ deployment, the pods run as the default service account in the namespace in which the pods are deployed. To check which capabilities are allowed for the service account of your pods, run the following command:

$ for psp in $(kubectl get psp -o
     jsonpath="{range .items[*]}{@.metadata.name}{'
'}{end}");
     do if [ $(kubectl auth can-i use psp/$psp --as=system:serviceaccount:<your
     namespace>:<your service account>) = yes ]; then kubectl get psp/$psp
     --no-headers -o=custom-columns=NAME:.metadata.name,CAPS:.spec.allowed
     Capabilities; fi; done

For more on the NET_ADMIN capability, see Istio’s Required Pod Capabilities page.

Installation

Simply rerun your deployment script. Initial installation challenges are sometimes overcome by reapplying your installation YAML file to your Kubernetes cluster (rerunning your installation commands). Whether due to network traffic loss or scheduled resources (e.g., CRDs) not yet having been fully instantiated by kube-api, sometimes you’ll find that not all Istio components have instantiated successfully the first time you applied them to the cluster.

In that event, you might see a message like this:

unable to recognize "install/kubernetes/istio-demo-auth.yaml": no
      matches for kind

You will need to ensure that CRDs have been applied by running the following command:

$ kubectl apply -f install/kubernetes/helm/istio/templates/crds.yamlt

Upgrade

Istio upgrades can follow various paths. Let’s look at two of those paths: Helm with Tiller, and without Tiller.

$ helm install install/kubernetes/helm/istio --name istio
       --namespace istio-system

$ helm upgrade istio install/kubernetes/helm/istio
       --namespace istio-system

$ helm template install/kubernetes/helm/istio --name istio --namespace
      istio-system > istio.yaml
$ kubectl apply -f istio.yaml

Uninstallation

Istio might not uninstall cleanly, leaving residual artifacts of its presence. You might find the following artifacts deposited.

$ for i in install/kubernetes/helm/istio-init/files/crd*yaml;
     do kubectl delete -f $i; done

$ kubectl get crds | grep istio

Using ControlZ (ctrlz)

Alternatively, you can turn on Mixer debugging using ControlZ on Mixer on port 9876. To do so, port forward to ControlZ:

$ kubectl --namespace istio-system port-forward istio-[policy/telemetry]-<pod#>
     9876:9876

Open your browser to http://localhost:9876.

Envoy’s Administrative Console

Envoy’s administrative interface runs on port 15000 (see Chapter 4 for an exhaustive list of ports). You can access Envoy’s administrative interface by using kubectl to port-forward from your local machine to any given pod sidecarred with Envoy:

$ kubectl port-forward <pod> 15000:15000 &

To stop a port-forwarding job running in the background, run kill %1 (assuming you have no other background jobs running). For a list of active jobs, run jobs.

To see a YAML-formatted printout of that Envoy’s configuration run the following:

$ curl http://localhost:15000/config_dump | yq r -

Or, bring up the full administrative console in your browser at http://localhost:15000. Reference http://localhost:15000/help for a description of administrative actions available. See Envoy’s operations and admin docs for a full description of all its administrative functions.

503 or 404 Requests

If you’re seeing either of these two error codes when trying to access your service, know that although there are different causes, the most common culprits are:

• There’s a disconnect between Envoy and Pilot. Here are the steps to remediate:

  1. Confirm Pilot is running. See “Troubleshooting Pilot”.

  2. Use istioctl proxy-status to confirm communication status between Pilot and Envoy. During normal operation, each xDS will show a status of SYNCED.

• There is a missing or incorrect network protocol in your Kubernetes service manifest. To remediate, configure your service manifest with the appropriate name for your service’s exposed ports. For a list of protocol names, see “Workload Preparedness”.

• The Envoy configuration is capturing the route in the wrong upstream cluster. The VirtualService configuration is incorrect. Here are the steps to remediate:

  1. Identify an edge service (adjacent to an ingress gateway).

  2. Inspect its Envoy logs or use a tool such as Jaeger to confirm where failures are occurring.

Sidecar Injection

If you’re having difficulty with sidecar injection, there may be a few different issues in your environment. Three factors could account for whether a sidecar is injected:

• The webhook namespaceSelector

• The default policy

• The per-pod override annotation

Verify that the following items are true:

error: unable to recognize "istio.yaml": no matches for
      admissionregistration.k8s.io/, Kind=MutatingWebhookConfiguration

$ kubectl get namespace -L istio-injection
NAME              STATUS   AGE   ISTIO-INJECTION
default           Active   39d   enabled
istio-system      Active   13h
kube-node-lease   Active   39d
kube-public       Active   39d
kube-system       Active   39d

$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml
       | grep "namespaceSelector:" -A5

  namespaceSelector:
    matchLabels:
      istio-injection: enabled
  rules:
  - apiGroups:
    - ""

 namespaceSelector:
    matchExpressions:
    - key: istio-injection
      operator: NotIn
      values:
      - disabled
  rules:
  - apiGroups:
    - ""
Debugging Citadel

$ kubectl logs -l istio=citadel -n istio-system
$ kubectl describe pod -l istio=citadel -n istio-system

$ kubectl get pod -l istio=citadel -n istio-system
NAME                             READY   STATUS    RESTARTS   AGE
istio-citadel-678b7c5cd4-ndn4n   1/1     Running   0          13h