In chapter 2, you deployed your first application in OpenShift and reviewed many of the components that were created. You confirmed that all the components worked together to deliver two deployments of the Image Uploader application. In this chapter, we’ll discuss those relationships in depth, and how OpenShift recovers when those relationships are altered.
When you deployed the Image Uploader application in chapter 2, one pod was created for each deployment. If that pod crashed, the application would be temporarily unavailable until a new pod was created to replace it. If your application became more popular, you wouldn’t be able to support new users past the capacity of a single pod. To solve this problem and provide scalable applications, OpenShift deploys each application with the ability to scale up and down. The application component that handles scaling application pods is called the replication controller (RC).
The RC’s main function is to ensure that the desired number of identical pods is running at all times. If a pod exits or fails, the RC deploys a new one to ensure a healthy application is always available (see figure 4.1).
You can think of the RC as a pod-monitoring agent that ensures certain requirements are met across the entire OpenShift cluster. You can check the current status of the RCs for the app-cli deployment by running the oc describe command (listing 4.1). Note that in listing 4.1, the individual deployment (app-cli-1) is specified, not the name of the application.
The information tracked about the RC helps to establish its relationship to the other components that make up the application:
The labels and selectors in the next listing are key-value pairs that are associated with all OpenShift components. They’re used to create and maintain the relationships and interactions between applications. We’ll discuss them in more depth in the next section.
$ oc describe rc $(oc get rc -l app=app-cli -o=jsonpath='{.items[].metadata.name}') Name: app-cli-1 1 Namespace: image-uploader Selector: app=app-cli,deployment=app-cli-1,deploymentconfig=app-cli 2 Labels: app=app-cli openshift.io/deployment-config.name=app-cli Annotations: openshift.io/deployer-pod.name=app-cli-1-deploy openshift.io/deployment-config.latest-version=1 openshift.io/deployment-config.name=app-cli openshift.io/deployment.phase=Complete openshift.io/deployment.replicas= openshift.io/deployment.status-reason=config change openshift.io/encoded-deployment-config= {"kind":"DeploymentConfig","apiVersion":"v1","metadata": {"name":"app-cli","namespace": "image-uploader","selfLink": "/apis/apps.openshift.io/v1/namespaces/image-up... Replicas: 1 current / 1 desired Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed 3 Pod Template: Labels: app=app-cli deployment=app-cli-1 deploymentconfig=app-cli Annotations: openshift.io/deployment-config.latest-version=1 openshift.io/deployment-config.name=app-cli openshift.io/deployment.name=app-cli-1 openshift.io/generated-by=OpenShiftNewApp Containers: app-cli: Image: docker-registry.default.svc: 4 5000/image-uploader/app-cli@sha256: 4 cef79b2eaf6bb7bf495fb16e9f720d5728299673dfec1d8f16472f1871633ebc 4 Port: 8080/TCP Environment: <none> Mounts: /opt/app-root/src/uploads from volume-mddzb (rw) 5 Volumes: volume-mddzb: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: app-cli ReadOnly: false Events: <none>
By using the oc describe command, you can quickly look at the information about the component, but not all the fields are shown. If you want to see every attribute for the component, use the oc get command, which can be output in multiple formats, including YAML syntax. An example is oc get rc app-cli-1 -o yaml.
The RC doesn’t keep track of a specific list of pods that it manages. This is by design. In keeping with Kubernetes design philosophies, OpenShift API objects are loosely coupled. RCs use label selectors to constantly query the pods that they manage. If pods are created or deleted outside of the normal process, the RC can immediately take action to put the system back into the desired state. Listing 4.1 included the selector field; the RC will manage any pod that has the three labels shown in that selector field. This loosely coupled philosophy is demonstrated throughout OpenShift, including many of the API objects discussed in the first three chapters:
The RC and other objects track pods through labels and selectors, and this can be used in troubleshooting. For instance, a pod may exhibit odd behavior that requires extensive debugging. Instead of taking the system offline, an OpenShift administrator can modify the pod’s labels so it’s no longer included as part of the RC. The old pod is quarantined and can be debugged via a command shell without affecting end users or services. The RC will then notice that the desired number of pods is different from the current number of pods and start a new pod in place of the one that’s in quarantine.
As we go forward, it’s important that you understand the following regarding how labels and selectors are used in OpenShift:
Let’s examine this in more depth, using the app-cli application you deployed in chapter 2. In the next example, you’ll remove a label from a deployed pod. This is one of the few times we’ll ask you to do something to intentionally break an application. Removing a label from a pod will break the relationship with the RC and other application components. The purpose of this example is to demonstrate the RC in action—and to do that, you need to create a condition that it needs to remedy.
Using listing 4.1 as an example, the selectors and labels for the app-cli-1 RC and the app-cli-1 pods are shown in figure 4.2. Selectors in an application component are the labels it uses to interact with other components. The app-cli RC will create and monitor applications pods with the following labels:
The app-cli-1 RC will always ensure that there is the configured number of replica applications pods with those three labels. When you deployed app-cli, you didn’t specify a specific replica count, so the RC ensures that one replica is running. In the next section, you’ll manipulate the number of replicas for the app-cli deployment. First, let’s discuss what happens when an application pod no longer meets the criteria defined by the RC.
As an exercise, remove the app=app-cli label from the app-cli pod:
If you click each pod individually, you’ll see that one pod has all three labels, and one pod has only two labels (because you deleted one). The RC immediately detected your label deletion and deployed a new pod that had all the labels to meet its requirement of one pod with the proper labels defined by the selectors in the RC (see figure 4.4).
The original pod from which you removed a label is now considered abandoned, because there isn’t an RC with matching selectors. If it dies, a new pod won’t be started in its place. The new pod is being managed by the app-cli-1 RC, which means if that pod dies or is deleted, then a replacement pod will be started quickly.
The fastest way to delete the pods for the app-cli deployment is through the command line. This process shouldn’t be part of your normal application workflow, but it can come in handy when you’re troubleshooting issues in an active cluster. From the command line, run the following oc delete command to delete all the pods for the app-cli application. The -l parameter specifies a label to look for when performing the delete:
$ oc delete pod -l app=app-cli
Switching back to the web console, you can see that a single pod for app-cli was recreated. By navigating down to the pod details, you can see that the new pod, which was started by the RC in response to you deleting the app=app-cli label, has all three labels specified by the RC’s selectors.
You may be wondering whether the abandoned pod will still receive traffic from users. It turns out that the service object, responsible for network traffic, also works on the concepts of labels and selectors. To determine whether the abandoned pod would have served traffic, you need to look at the Selector field in the service object. You can get the selector information about the app-cli service by running the following oc describe command. There’s a lot of output, so passing it into a grep command to look for only the information you want can be helpful:
$ oc describe svc app-cli | grep Selector Selector: app=app-cli,deploymentconfig=app-cli
The output shows that the service component has two selectors:
Because you deleted the app=app-cli label from the original pod, its labels are no longer a match for the app-cli service selectors and would no longer receive traffic requests.
Kubernetes was born out of many lessons learned at Google from running containers at scale for 10+ years. The main two orchestration engines internally at Google during this time have been Borg and its predecessor, Omega. One of the primary lessons learned from these two systems was that control loops of decoupled API objects were far preferable to a large, centralized, stateful orchestration. This type of design is often called control through choreography. Here are just a few of the ways it was implemented in Kubernetes:
By running through control loops instead of maintaining a large state diagram, the resiliency of the system is considerably improved. If a controller crashes, it reruns the loop when it’s restarted, whereas a state machine can become brittle when there are errors or the system starts to grow in complexity. In our specific examples, this holds true because the RC loops through pods with the labels in its Selector field as opposed to maintaining a list of pods that it’s supervising.
You can find more information about the Kubernetes structure at https://kubernetes.io.
Replication controllers ensure that properly configured application pods are always available in the proper number. Additionally, the desired replica counts can be modified manually or automatically. In the next section, we’ll discuss how to scale application deployments.
An application can consist of many different pods, all communicating together to do work for the application’s users. Because different pods need to be scaled independently, each collection of identical pods is represented by a service component, as we initially discussed in chapter 1. More complex applications can consist of multiple services of independently scaled pods.
A standard application design uses three tiers to separate concerns in the application:
Figure 4.5 shows how a three-tier application is deployed in OpenShift. The application code runs in the pods for each application layer. That code is accessed directly from the routing layer. Each application’s service communicates with the routing layer and the pods it manages. This design results in the fewest network hops between the user and the application. This design also allows each layer in the three-tier design to be independently scaled to handle its workload effectively without making any changes to the overall application configuration.
Figure 4.5 is a simple example. In a large, enterprise application, dozens (sometimes hundreds) of services often work together to build an application. The design methodology behind building out multiple small, decoupled applications that work together to act together is often called microservices. When you start designing applications that are made up of multiple smaller, independent pods, the act of scaling those pods up and down becomes more important. In the next section, we’ll discuss different ways to scale application pods in OpenShift.
Extreme examples of the microservice paradigm exist in large tech companies like Google, Amazon, and Netflix. These companies often connect to hundreds of different services on the backend of a single user session. Not many other IT organizations may need to scale to the level of Netflix, but the need for a platform to handle modern web application designs is nearly universal.
To scale an application, begin at the deployment configuration page. Follow these steps:
1. Choose Applications > Deployments on the left panel of your project overview. This takes you to a list of all your OpenShift deployments in the current project.
2. Click the link for your app-cli deployment. The History tab opens, with a list of changes to the app-cli deployment.
3. Click Configuration to see details of the deployment configuration. On this page, you can change the number of replicas to meet the demands of your application.
4. Click Edit next to the current replica count (see figure 4.6), and change the number of replicas to three. Changing this number will change the deployment config object, which will update the RC behind the scenes.
Figure 4.6. Editing the number of replica pods by editing the deployment config
Changing the number of pods will take a few seconds to complete. When that’s done, you’ll have scaled the app-cli deployment from one pod to three (see figure 4.6).
The deployment config component makes changes to the RC in the background. When you instruct the deployment config to add replicas to an application, it in turn tells the RC to create the additional pods.
The same functionality is available on the command line. To scale the number of pods back down to one, run the following oc scale command:
$ oc scale dc app-cli-1 --replicas=1
At this point, you’ve done the following:
The up and down arrows on the project overview page also scale applications by modifying the deployment config component the same way. The only difference is that if you use the arrows, you can add or remove only one pod at a time. If you needed to scale to 20 or even 100 pods, you wouldn’t want to click the console for every change in replica count.
In the real world, manually editing deployment configs to scale applications isn’t practical for many scenarios. In chapter 5, we’ll discuss ways to automate application scaling.
In most situations, application pods run into issues because the code in the pod stops responding or begins to respond in ways that aren’t desired. The first step in building a resilient application is to run automated health and status checks on your pods, restarting them when necessary without manual intervention. Creating probes to run the needed checks on applications to make sure they’re healthy is built into OpenShift. The first type of probe we’ll look at is the liveness probe.
In OpenShift, you define a liveness probe as a parameter for specific containers in the deployment config. The liveness probe configuration then propagates down to the individual containers created in pods running as part of the DC. A service on each node running the container is responsible for running the liveness probe that’s defined in the deployment config. If the liveness probe was created as a script, then it’s run inside the container. If the liveness probe was created as an HTTP response or TCP socket-based probe, then it’s run by the node connecting to the container. If a liveness probe fails for a container, then the pod is restarted.
The service that executes liveness probe checks is called the kubelet service. This is the primary service that runs on each application node in OpenShift.
In OpenShift, the liveness probe component is a simple, powerful concept that checks to be sure an application pod is running and healthy. Liveness probes can check container health three ways:
An HTTP response code is a three-digit number supplied by the server as part of the HTTP response headers in a web request. A 2xx response indicates a successful connection, and a 3xx response indicates an HTTP redirect. You can find a full description of all response codes on the IETF website at http://mng.bz/XfMi.
As a best practice, always create a liveness probe unless your application is intelligent enough to exit when it hits an unhealthy state. Create liveness probes that not only check the internal components of the application, but also isolate problems from external service dependencies. For example, a container shouldn’t fail its liveness probe because another service that it needs isn’t functional. Modern applications should have code to gracefully handle missing service dependencies. If you need an application to wait for a missing service dependency, you can use readiness probes, which are covered later in this chapter. For legacy applications that require an ordered startup sequence of replicated pods, you can take advantage of a concept called stateful sets, which we’ll cover in chapter 8.
To make creating probes easier, a health check wizard is built into the OpenShift web interface. Using the wizard will help you avoid formatting issues that can result from creating the raw YAML template by hand.
To add a new liveness probe to a deployed application, follow these steps:
1. Choose Applications > Deployments on the left panel of your project view.
2. Click your deployment, app-cli.
3. Click the Configuration tab. You should see an alert that shows you haven’t yet configured any health checks (see figure 4.7).
Figure 4.7. The Deployments page, showing the app-cli deployment config link
4. Choose Add Health Checks > Add Liveness Probe.
5. On the page that opens, set the following values for your probe, and click Save (see figure 4.8):
The values you specify to create a liveness probe are as follows:
After you create the liveness probe, you’ll be redirected back to the Deployments page for app-cli. When you added the liveness probe, the deployment config automatically created a new deployment for app-cli that includes the liveness probe. The deployment config also automatically migrated the app-cli route to the new deployment. Click #2 to access the app-cli deployment config overview for the newly deployed application.
Scroll down to the bottom of the page to see information about the newly deployed pod. If the deployment has failed for some reason, a large error with a link to the events and logs to help begin troubleshooting will appear at the top of the Deployments page.
Click Pod Name. The default pod view should show that there’s a liveness probe configured for the app-cli deployment (see figure 4.9).
The OpenShift health check wizard was designed for ease of use to cover most situations, but not all. Other parameters can be passed to the liveness probe that aren’t implemented in the web interface, but are available from the command line. One that’s useful is failureThreshold, which defaults to three attempts. As we discussed in the previous section, failureThreshold for a readiness probe (discussed in the next section) or a liveness probe sets the number of times a probe will be attempted before it’s considered a failure.
Imagine a situation where a liveness probe’s initial delay was run before the application started up. You wouldn’t want the container to be killed right away. failureThreshold allows the liveness probe to try again in periodSeconds, another liveness probe parameter that can be manually set. In the current release of OpenShift, periodSeconds defaults to 10 seconds.
To create a probe on the command line, you use the oc set probe command, which we’ll discuss in chapter 5.
You can find the documentation for all probes at http://mng.bz/Yh16
Liveness probes are a good way to ensure that application pods are functioning properly. For app-cli, a new deployment should take less than a minute on your system. But some applications may not be ready to receive traffic that soon after the pod is deployed.
Many applications need to perform any combination of the following before they’re able to receive traffic, which increases the amount of time before an application is ready to do work. Some common tasks include
Fortunately, OpenShift also supports the concept of readiness probes, which ensures that the container is ready to receive traffic before marking the pod as active. Similar to a liveness probe, a readiness probe is run at the container level in a pod and supports the same HTTP(S), container execution, and TCP socket-based checks. Unlike a liveness probe, though, a failed readiness check doesn’t result in a new pod being deployed. If a readiness check fails, the pod remains running while not receiving traffic.
Let’s run through an example of adding a readiness probe to the app-cli application using the command line. For this readiness probe, you’ll tell OpenShift to look for a non-existent endpoint.
Looking for a URL that doesn’t exist in your app-cli deployment will cause the readiness probe to fail. This exercise illustrates how an OpenShift probe works when it runs into an undesired condition. Until a deployment passes a readiness probe, it won’t receive user requests. If it never passes the readiness probe, as in this example, the deployment will fail and never be made available to users.
To create the readiness probe, use the command line and run the oc set probe command:
$ oc set probe dc/app-cli --readiness --get-url=http://:8080/notreal --initial-delay-seconds=5 deploymentconfig "app-cli" updated
The output includes a message that the deployment configuration was updated. Just like a liveness probe, creating a readiness probe triggers the creation of a new app-cli deployment. Check to see whether the new pods were deployed by running the oc get pods command:
$ oc get pods NAME READY STATUS RESTARTS AGE app-cli-1-build 0/1 Completed 0 17d app-cli-2-js7z9 1/1 Running 0 38m 1 app-cli-3-6snpl 0/1 Running 0 1m 2 app-cli-3-deploy 1/1 Running 0 2m app-gui-1-build 0/1 Completed 0 15d app-gui-1-x6mvw 1/1 Running 1 15d
The new app-cli pod is running but not ready, which means it isn’t yet receiving traffic. The previous pod is still running and receiving any incoming requests. Eventually, the readiness probe will fail two more times and will meets the readiness probe failureThreshold metric, which is set to 3 by default. As we discussed in the previous section, failureThreshold for a readiness or liveness probe sets the number of times a probe will be attempted before it’s considered a failure.
The readiness probe will take 10 minutes to trigger a failed deployment. When this happens, the pod will be deleted, and the deployment will roll back to the old working configuration, resulting in a new pod without the readiness probe. You can modify the default timeout parameters by changing the timeoutSeconds parameter as part of dc.spec.strategy.*params in the deployment config object. Deployment strategies are covered in greater detail at the end of chapter 6.
Once all three failures occur, the deployment is marked as failed, and OpenShift automatically reverts back to the previous deployment:
$ oc get pods NAME READY STATUS RESTARTS AGE app-cli-1-build 0/1 Completed 0 17d app-cli-2-js7z9 1/1 Running 0 47m 1 app-cli-3-deploy 0/1 Error 0 10m 2 app-gui-1-build 0/1 Completed 0 15d app-gui-1-x6mvw 1/1 Running 1 15d
The reason for the failure appears as an event in OpenShift that can be shown from the command line or the web console. Because events are easier to read through the web console, let’s check it out there. Click the Overview tab on the left panel to go to your project’s home page. There you’ll see a warning showing that the previous deployment configuration failed (see figure 4.10).
Expand the panel by clicking the down arrow, and then click View Events, as shown in figure 4.11. You’ll see all the events for the current project, including one that says, “Readiness probe failed: HTTP probe failed with statuscode: 404.” HTTP response code 404 means “Page Not Found.” This makes sense, because the readiness probe was configured to check for a nonexistent URL.
In the last example, the readiness probe failed, and after the deployment config time was reached, OpenShift automatically rolled back the application to the previous working development. To manually trigger rollbacks, you can use the console or the command line. An example of using the command line for rollback is executing the following command:
$ oc rollback app-cli
In this chapter, we’ve discussed how replication controllers work and are managed by their corresponding deployment configs. By replicating pods across your OpenShift cluster, you can ensure that your applications are resilient and highly available, and that multipod applications have independently scaling components.