Chapter 9. Authentication and resource access

This chapter covers

• Adding permissions to users by assigning roles
• Managing project resource limits and quotas
• Setting default limits and quotas for projects
• Examining how Linux enforces limits and quotas

Before we get started, let’s face it: this isn’t the most interesting chapter in the book. Setting up authentication sources and configuring project quotas for your applications aren’t topics that will show up on the first slide of anyone’s presentation. They’re essential for an application platform to function correctly, however, so strap in, and let’s dive into the deep, dark reaches of OpenShift.

9.1. Proper permissions vs. the Wild West

Application platforms like OpenShift aren’t effective for multiple users without robust access and permissions management for various applications and OpenShift components. If every user had full access to all of your OpenShift cluster’s resources, it would truly be the Wild West. Conversely, if it was difficult to access resources, OpenShift wouldn’t be good for much, either. OpenShift has a robust authentication and access-control system that provides a good balance of self-service workflows to keep productivity up while limiting users to only what they need to get their job done.

When you first deployed OpenShift, the default configuration allowed any username and non-empty password field to log in. This authentication method uses the allow-all identity provider that comes with OpenShift.

Tip

In OpenShift, an identity provider is a plugin that defines how users can authenticate and the backend service that you want to connect to for managing user information. Although the allow-all provider is good enough when you’re learning to use OpenShift, when you need to enforce access rules, you’ll need to change to a more secure authentication method. In the next section, you’ll replace the allow-all provider with one that uses a local database file.

Appendix D walks you through configuring your OpenShift cluster to use an Apache htpasswd database for user access and set up a few users to use with that authentication source. You’ll create the following users:

• developer—a user with permissions typical to those given to a developer in an OpenShift cluster.
• project-admin—a user with permissions typical of a developer or team lead in an OpenShift cluster. They have administrative control over a single project.
• admin—a user with administrative control over the entire OpenShift cluster.

Please go through that now, and then continue with this chapter. After configuring OpenShift per appendix D, if you attempt to log in with your original dev user, that user can’t be authenticated because it’s not in your htpasswd database. But if you log in using the developer or admin user, you no longer have access to the image-uploader project. That’s because the now locked-out dev user owns the image-uploader project and all the applications deployed in it.

You’ve made your OpenShift cluster more secure by configuring it to use a more secure authentication provider. But in the process, you disabled access to the deployed applications by stopping access for the user you used to deploy your project and applications. In the next section, you’ll correct this and set up more robust permissions by manipulating user roles.

9.2. Working with user roles

Roles are used to define permissions for all users in an OpenShift cluster. In chapter 3, you used the special system:admin user to configure physical volumes on your cluster. The system:admin is a special system account that uses SSL certificates for authentication. In this section, you’ll create users with similar privileges using roles.

To work with roles, you’ll use a new command-line tool named oadm (short for OpenShift administration). It’s installed by default on your master server.

On your master node, the OpenShift deployment program set up the root user with the same admin-level user information that you used in chapter 3 by copying the admin .kubeconfig file to your workstation. You can see all the user information set up for the root user by running the following command as the root user on your master server:

oadm config view

This allows administrators with access to the root user on an OpenShift master server to have cluster administration access by default. It’s useful, but it also means you have to make sure everyone who has root access to your master server should be able to control your OpenShift cluster. For a smaller cluster like the one you’ve built, this will work fine. But for a larger cluster, the people who should have root access to your servers and the people who should be able to administer OpenShift probably won’t match exactly. You can distribute this administrative certificate as needed for your cluster administrator’s workstations.

9.2.1. Assigning new user roles

Remember those users you created? The developer user needs permission to view and add new content to the image-uploader project. To accomplish that, first make sure you’re working in the context of the image-uploader project by running the following command:

oc project image-uploader

In the project, you need to add the edit role to your developer user. This role gives users permission to add to a project and edit existing deployments. Adding a role to a user for a project, or even the entire OpenShift cluster, is called binding a role to a user. You do so by running the following command:

oadm policy add-role-to-user edit developer

To confirm that your new role is applied, log in again through the web UI or the command line as the developer user. You should now have access to the image-uploader project and the deployed applications in it.

That takes care of the developer user. Let’s give your admin user a little more access. In the next section, you’ll give the admin user administrator-level access to your entire OpenShift cluster.

9.2.2. Creating administrators

So far, your OpenShift cluster has a single project. As an OpenShift cluster grows, it typically has dozens, or even hundreds, of projects at any given time. To manage this effectively, you need users who can administer a project, or even multiple projects across the cluster.

Creating a project admin

For the image-uploader project, you’ll make the project-admin user an administrator for the project. You do so much the same way you gave the developer user the ability to edit. Instead of binding the edit role to the project-admin user, however, you need to bind the admin role. This role will give the project-admin user full administrative privileges in the image-uploader project. Run the following command as root on your master server:

oadm policy add-role-to-user admin project-admin

You now have a developer who can work in the image-uploader project and a project-admin user who can administer the project. The next user role you need is one who can manage the entire OpenShift cluster.

Creating a cluster admin

The cluster administrator role is important. To borrow a line from a comic book, “With great power comes great responsibility.” A cluster admin can not only administer projects, but also manage all of OpenShift’s internal configurations. To create a cluster admin, run the following command as root on your master node:

oadm policy add-cluster-role-to-user cluster-admin admin

This command binds the admin role to the admin user you created in the previous section. Instead of binding that role for a single project, it binds it for every project in OpenShift.

Everything you’ve done in this chapter until now will help you edit existing users and make sure they have the correct privileges to access what their job requires. But what happens when you add new users? In the next section, you’ll configure OpenShift to bind the edit role to new users by default when they’re created.

9.2.3. Setting default user roles

OpenShift has three default groups. These groups are configured during OpenShift installation and define whether a user is authenticated. You can use these groups to target users for additional actions, but the groups themselves can’t be modified:

• system:authenticated— Any user who has successfully authenticated through the web UI or command line, or via the API.
• system:authenticted:oauth— Any user who’s been authenticated by OpenShift’s internal oauth server. This excludes system accounts.
• system:unauthenticated— Users who have failed authentication or not attempted to authenticate.

In your cluster, it will be helpful to allow any authenticated user to access the image-uploader project. You can accomplish this by running the following oadm policy command, which binds the edit role for the image-uploader project, specified by the -n option, to the system:authenticated default group:

oadm policy add-role-to-group edit -n image-uploader system:authenticated

Any user who has successfully logged in will now be able to access the image-uploader project.

To confirm that your new default user role has taken effect, add an additional user named user1 to your htpasswd database with the following command:

echo user1 | htpasswd --stdin /etc/origin/master/openshift.htpasswd user1

Log in to your cluster and confirm that your new user can see the image-uploader project by default. user1 should have the ability to work in the image-uploader project from the time of first login.

Any time you have a shared environment, you need processes in place to ensure that one user or project doesn’t take up too many resources in your cluster—either accidentally or on purpose. Limit ranges and resource quotas are the processes that manage this potential problem. In OpenShift, these resource constraints are different for each deployment, depending on whether explicit resource quotas are requested.

For app-gui and app-cli, you didn’t specify specific CPU or memory resources to be allocated for either deployment. These best-effort deployments—deployments that don’t request specific resources and are assigned a best-effort quality of service (QoS)—can govern default values at the project level in OpenShift by using limit ranges. In the next section, we’ll discuss limit ranges in more depth, and you’ll create your own and apply them to the image-uploader project.

9.3. Limit ranges

For each project in OpenShift, a limit range, defined as a LimitRange when working with the OpenShift API, provides resource constraints for most objects that exist in a project (the documentation is at http://mng.bz/tB5m.) The objects are the types of OpenShift components that users deploy to serve applications and data. Limit ranges apply to the maximum CPU, memory, and total object count for each component. The limits for each component are outlined in table 9.1.

Table 9.1. Limit-range resource constraints

Before an application is deployed or scaled up, the project limit ranges are analyzed to confirm that the request is within the limit range. If a project limit range doesn’t allow the desired action, then it doesn’t happen (see figure 9.1). For example, if a project’s limit range defines the memory per pod as between 50 MB and 1,000 MB, a request for a new application deployment with a defined quota of 1,500 MB will fail because it’s outside the pod-memory limit range for that project.

Figure 9.1. Limit ranges are queried before any deployment or scaling request.

Limit ranges have the additional benefit of being able to define default compute resource values for a project. When you deployed app-gui and app-cli, you hadn’t yet defined a limit range for the image-uploader project and didn’t specify the resources for each deployment, so each application’s pod was deployed with no resource constraints. In OpenShift, a deployment with no defined resource quotas is called a best-effort deployment. A best-effort deployment has the lowest priority in an OpenShift cluster and is the first to be killed if a system runs out of memory.

If users start accessing app-gui heavily, it can consume resources to the point that the performance of the app-cli deployment is affected. For a busy cluster with multiple users and running applications, that’s a major problem. With limit ranges, you can define the default compute resources for a project that doesn’t specify a quota and prevent this from happening in your cluster.

9.3.1. Defining resource limit ranges

Limit ranges define the minimum and maximum RAM and CPU an application can be allocated when it’s deployed. In addition to the top and bottom of the range, you can specify default request values and limits. The difference between an application’s requested value and its maximum limit is called the burstable range.

All this is a little abstract, so let’s walk through a few quick examples, using as a reference the limit-range YAML template in listing 9.1:

• If you deploy an application with no requested resources, you’re assigned a default request amount of 200 millicores and 100 MiB of RAM per pod. The default limit is 300 millicores and 200 MiB RAM. That means your application would be deployed with 200 millicores and 100 MiB RAM and could burst up to a maximum of 300 millicores and 200 MiB RAM if it received a spike in traffic.
• If you requested specific resources for an application deployment, you can request any CPU value between 100 millicores and 2,000 millicores (2 CPU), and RAM resources between 4 MiB and 1,000 GiB per pod. For the burstable range, your application can’t exceed 10 times your defined request. This is set by maxLimitRequestRatio in the limit range. That means if you requested 150 millicores of CPU, the maximum your application could ever consume would be 1,500 millicores.

Let’s create a limit range for the image-uploader project using the template in listing 9.1. Create a file called core-resource-limits.yaml using the contents of the template. Then, ensure that you’re logged in as the proper user.

Listing 9.1. Template to create limit-range definitions for image-uploader

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "core-resource-limits"         1
spec:
  limits:
    - type: "Pod"                      2
      max:
        cpu: "2"                       3
        memory: "1Gi"                  4
      min:                             5
        cpu: "100m"
        memory: "4Mi"
    - type: "Container"                6
      max:                             7
        cpu: "2"
        memory: "1Gi"
      min:                             8
        cpu: "100m"
        memory: "4Mi"
      default:                         9
        cpu: "300m"
        memory: "200Mi"
      defaultRequest:                  10
        cpu: "200m"
        memory: "100Mi"
      maxLimitRequestRatio:            11
        cpu: "10"

• 1 Name for the resource limits object
• 2 Limit ranges for pods
• 3 Maximum CPUs for one pod
• 4 Maximum memory for one pod
• 5 Minimum CPU and memory for one pod
• 6 Limit ranges for containers
• 7 Maximum CPU and memory for one container
• 8 Minimum CPU and memory for one container
• 9 Default maximum CPU and memory usable for one container
• 10 Default requested CPU and memory for one container
• 11 Maximum burstable ratio for container CPU utilization

To define a limit range, a user needs to have the cluster-admin role. To log in as the admin user, run the following oc login command:

oc login -u admin -p admin https://ocp-1.192.168.122.100.nip.io:8443

Now, execute the following oc create command to create the new limit range. The -n option can be used to specify the project you want to be affected by an oc command:

oc create -f core-resource-limits.yaml -n image-uploader

Tip

Throughout this chapter and the rest of the book, you’ll need to create YAML template files that are referenced in oc commands. We use relative filename paths to keep the examples easy to read, but if you’re not running oc from the directory where those files are created, be sure to reference the full path when you run the command.

You can use the command line to confirm that the image-uploader limit range was created and to confirm the settings you specified in the YAML template were accurately read:

$ oc get limitrange
NAME                   AGE
core-resource-limits   5m
$ oc describe limitrange core-resource-limits
Name:        core-resource-limits
Namespace:    image-uploader
Type        Resource    Min    Max    Default Request    Default Limit
 Max Limit/Request Ratio
----        --------    ---    ---    ---------------    -------------
 -----------------------
Pod        cpu        100m    2    -        -        -
Pod        memory        4Mi    1Gi    -        -        -
Container    cpu        100m    2    200m        300m        10
Container    memory        4Mi    1Gi    100Mi        200Mi        -

You also can use the web UI to confirm the limit range you just set. Choose Resources > Quotas from the image-uploader project overview page to see the limit range.

Limit ranges act on components in a project. They also provide default resource limits for deployments that don’t provide any specific values themselves. But they don’t provide project-wide limits to specify a maximum resource amount. For that, you’ll need to define a resource quota for image-uploader. In the next section, that’s exactly what you’ll do.

9.4. Resource quotas

Nobody likes a noisy neighbor, and OpenShift users are no different. If one project’s users were able to consume more than their fair share of the resources in an OpenShift cluster, all manner of resource-availability issues would occur. For example, a resource-hungry development project could stop applications in a production-level project in the same cluster from scaling up when their traffic increased. To solve this problem, OpenShift uses project quotas to provide resource caps and limits at the project level.

Whereas limit ranges provide resource minimums and maximums for individual application components, quotas provide maximum resource limits for an entire project. Quotas fall into three primary categories:

• Compute resources
• Storage resources
• Object counts

In chapter 2, we discussed pod lifecycles. Project quotas apply only to pods that aren’t in a terminal (failed or succeeded) phase. Quotas apply to any pod in a pending, running, or unknown state. Before an application deployment is started or a deployed application is changed, OpenShift evaluates the project’s quotas (see figure 9.2).

Figure 9.2. Quotas are evaluated before any new resources are created in a project.

Tip

The full documentation for quotas in OpenShift is available at http://mng.bz/ktZj.

In the next section, you’ll create a compute resource quota for the image-uploader project.

9.4.1. Creating compute quotas

Compute resource quotas apply to CPU and memory allocation. They’re related to limit ranges, because they represent quotas against totals for requests and limits for all applications in a project. You can set the following six values with compute resource quotas:

• cpu, requests.cpu—Total of all CPU requests in a project, typically measured in cores or millicores. cpu and requests.cpu are synonyms and can be used interchangeably.
• memory, requests.memory—Total of all memory requests in a project, typically expressed in mebibytes or gibibytes. memory and requests.memory are synonyms and can be used interchangeably.
• limits.cpu—Total for all CPU limits in a project.
• limits.memory—Total for all memory limits in a project.

In addition to the quotas, you can also specify the scope the quota applies to. There are four quota scopes in OpenShift:

• Terminating—Pods that have a defined lifecycle. Typically, these are builder and deployment pods.
• NotTerminating—Pods that don’t have a defined lifecycle. This scope includes application pods like app-gui and app-cli, and most other applications you’ll deploy in OpenShift.
• BestEffort—Pods that have a best-effort QoS for CPU or memory. Best-effort deployments are those that didn’t specify a request or limit when they were created.
• NotBestEffort—Pods that don’t have a best-effort QoS for CPU or memory. The inverse of best effort, this scope is useful when you have a mixture of low-priority, transient workloads that have been deployed with best-effort QoS, and higher-priority workloads with dedicated resources.

Quotas are defined in YAML templates like all other resources in OpenShift. To create a quota for compute resources with the NotTerminating quota scope, create a file named compute-resources.yaml on your workstation with the content from the following listing.

Listing 9.2. Template to create CPU resource quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    pods: "10"                       1
    requests.cpu: "2"                2
    requests.memory: 2Gi             3
    limits.cpu: "3"                  4
    limits.memory: 3Gi               5
  scopes:
    - NotTerminating                 6

• 1 The project can run a maximum of 10 pods.
• 2 A maximum of two CPU cores can be requested.
• 3 A maximum of two gibibytes of memory can be requested.
• 4 The CPU limit is three cores total.
• 5 The project’s memory limit is three gibibytes.
• 6 Scope that limits the quota to deployed application pods

To create a new quota for a project, cluster-admin level privileges are required. That means you need to be logged in as the admin user to run this command, because the developer user only has the edit role bound to it for the image-uploader project and has no privileges for the rest of the cluster. To log in as the admin user, run the following oc login command:

oc login -u admin -p admin https://ocp-1.192.168.122.100.nip.io:8443

Now, run the following oc create command to create the new CPU resource quota:

oc create -f compute-resources.yaml -n image-uploader

This command applies the compute-resources quota YAML template to the image-uploader project. To confirm that this quota was applied, run the following oc get command:

oc get resourcequota -n image-uploader

In the output, you should see a quota name compute-resources. Next, you’ll create some resource quotas for image-uploader. Resource quotas provide a cap for the number of deployable resources in a project.

9.4.2. Creating resource quotas

Resource quotas track all resources in a project that are deployed by Kubernetes. Core components in OpenShift, like deployment configs and build configs, aren’t covered by quotas; that’s because these components are created on demand for each deployment and controlled by OpenShift.

The components that are managed by resource quotas are the primary resources in an OpenShift cluster that consume storage and compute resources. Keeping track of a project’s resources is important when you need to plan how to grow and manage your OpenShift cluster to accommodate your applications. The following components are tracked by resource quotas:

• Config maps— We discussed config maps in chapter 6.
• Persistent volume claims— Application requests for persistent storage.
• Resource quotas— The total number of quotas in a project.
• Replication controllers— The number of RCs in a project. This is typically equal to the number of deployed applications, but you can also manually deploy applications using different workflows that could make this number change.
• Secrets— We discussed secrets in chapter 6.
• Services— The total number of services in a project.
• Image streams— The total number of image streams in a project.

Most of the items in this list should look familiar. We’ve been discussing them for several chapters now.

The following listing shows the resource quota template that you need to apply to the image-uploader project. To do this, create a file named core-object-counts.yaml on your workstation, and enter the contents of the following listing.

Listing 9.3. Template to create resource quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: core-object-counts             1
spec:
  hard:                                2
    configmaps: "10"
    persistentvolumeclaims: "5"
    resourcequotas: "5"
    replicationcontrollers: "20"
    secrets: "50"
    services: "10"
    openshift.io/Image streams: "10"

• 1 Name for the resource quota
• 2 Limits for resource types

Run the following command to apply the resource quota template to the image-uploader project. As was true for the compute quota you created, you need to be logged in to OpenShift as the admin user to run this command:

oc create -f core-object-counts.yaml -n image-uploader

Next, you can confirm that your quota was created using the command line or the web interface. At the command line, run the same oc get quota command you ran to confirm that your compute quota was applied. You should see both quotas in the command output:

$ oc get quota -n image-uploader           1
NAME                 AGE
compute-resources    9h
core-object-counts   9h

• 1 Gets the names of all active quotas

To see the configuration for each quota, run oc describe resourcequota and complete the command with each quota name:

$ oc describe resourcequota compute-resources                   1
Name:        compute-resources
Namespace:    image-uploader
Scopes:        NotTerminating
 * Matches all pods that do not have an active deadline.
Resource    Used    Hard
--------    ----    ----
limits.cpu    0    2
limits.memory    0    2Gi
pods        2    10                                             2
requests.cpu    0    3
requests.memory    0    3Gi

$ oc describe quota core-object-counts                          3
Name:                core-object-counts
Namespace:            image-uploader
Resource            Used    Hard
--------            ----    ----
configmaps            0    10
openshift.io/imagestreams    2    10
persistentvolumeclaims        1    5
replicationcontrollers        4    20
secrets                9    50
services            2    10

• 1 Describes the compute resource quota
• 2 Pod count quota
• 3 Resource quotas, which are all counted correctly

In the web interface, you can see this information added to the quota page under Resources > Quotas, linked from the image-uploader project overview page.

Looking at the compute quota from the previous output, you can see that you’ve run into a problem: the app-gui and app-cli deployments aren’t counting against the compute quota. The number of pods is reflected, but not the CPU or memory consumption. In the next section, we’ll discuss why that is and how you can work with your newly minted quotas and limits in OpenShift.

9.5. Working with quotas and limits

Now that the image-uploader project has limit ranges and quotas, it’s time to put them through their paces. The compute quota for app-cli and app-gui isn’t being reflected yet, and your first task is to fix that.

9.5.1. Applying quotas and limits to existing applications

When you deployed app-gui and app-cli in chapter 2, no quotas or limits were defined for the image-uploader project. As we mentioned when you were creating limit ranges, back then your cluster was essentially the Wild West, and any deployed application could consume any amount of resources in the cluster.

If an application is created and there are no limit ranges to reference and no resources were requested (as when you deployed the metrics pod in chapter 5), the Linux kernel components that define the resource constraints for each container are created with unlimited values for the resources limits. This is what happened when you deployed app-cli and app-gui and why their CPU and memory quotas aren’t reflected in OpenShift.

Now that you’ve applied limit ranges and quotas to image-uploader, you can have OpenShift re-create the containers for app-gui and app-cli to include these constraints. The easiest way to do this is to delete the current pods for each application. When you run the following oc delete command, OpenShift will automatically deploy new pods that contain the default limit ranges that you defined in the previous section:

$ for i in app-cli app-gui;do oc delete pod -l app=$i;done

This command is a for loop that will iterate through both app-cli and app-gui and variables. For each value, it will delete the pods in OpenShift that have the app=VALUE label.

Because you didn’t specify specific resource values, your new app-gui and app-cli pods inherit the default request values you defined in the core-resource-limits limit range. Each pod was assigned 200 millicores and 100 MiB of RAM. You can see in the previous output that the consumed CPU and memory quotas for the image-uploader project are twice the default request.

Tip

It’s definitely not a best practice to start using projects before you’ve set limits and quotas. But we had to start somewhere, and if chapter 2 was all about quotas, you’d never have gotten to chapter 3. So, for teaching purposes, we decided to begin with using OpenShift before we discussed a proper configuration.

In the next section, you’ll edit the app-cli deployment config to give it more resources than the default limit.

9.5.2. Changing quotas for deployed applications

When you deploy a new application, you can specify limits and quotas as part of its definition. You can also edit the YAML definition for an existing deployment config directly from the command line. To edit the resource limits for your app-cli deployment, run the following oc edit command, which lets you edit the current YAML definition for the application:

oc edit dc/app-cli

This command attempts to open the YAML template from the OpenShift database with the default text editor on your system. To edit the resource limits, you need to find the spec.containers.resources section of the configuration. This section is currently empty because nothing was defined when the application was deployed:

spec:
      containers:
        ...
        resources: {}
        ...

Let’s add memory and CPU requests and limits that are slightly larger than the default request values you defined in your limit range. Replace the {} in the resources section, and edit the file to look like the following listing, adding a larger quota request and limit for CPU and memory resources.

Listing 9.4. Editing the deployment config to increase limits and resource requests

resources:
  requests:
    cpu: "750m"
    memory: "500Mi"
  limits:
    cpu: "1"
    memory: "1000Mi"

Saving the new configuration file will trigger a new deployment for app-cli. This new deployment will incorporate your new resource requests and limits. Once the build completes, your app-cli deployment will be available with more guaranteed resources. You can confirm this by looking at the web interface page for the latest deployment, or from the command line by running the following oc describe command:

oc describe dc/app-cli

You can edit a deployment config to make complex changes to deployed applications, but it’s a manual process. For new application deployments, your projects should use the default limit ranges whenever possible, to inherit default values.

While your resource requests and limit ranges are new and fresh in your mind, let’s dig a little deeper and discuss how these constraints are enforced in OpenShift by the Linux kernel with control groups (cgroups).

9.6. Using cgroups to limit resources

Cgroups are Linux kernel components that provide per-process limits for CPU, memory, network bandwidth, and block-storage bandwidth. In an OpenShift cluster, they enforce the limits and quotas configured for applications and projects.

9.6.1. Cgroups overview

Cgroups are defined in a hierarchy in the /sys/fs/cgroup/ directory on the application node. Within this director is a directory for each type of cgroup controller that’s available. A controller represents a specific system resource that can be controlled by cgroups (see figure 9.3).

Figure 9.3. Cgroup controller hierarchy

In this section, we’re focusing on the cpu and memory cgroup controllers. In the directories for the cpu and memory controllers is a directory named kubepods.slice. Cgroup slices are used to create subdivisions within a cgroup controller. Slices are used as logical dividers in a controller and define resource limits for groups of resources below them in the cgroup hierarchy.

The kubepods slice is where the configurations to enforce OpenShift requests and limits are located (figure 9.4). Within kubepods.slice are two additional slices:

• kubepods-besteffort.slice
• kubepods-burstable.slice

Figure 9.4. kubepods.slice cgroup slices

These two slices are how resource limits for best-effort and burstable QoS levels that we’ve discussed in this chapter are enforced. Because you defined resource requests for app-cli and app-gui, they both will be defined in kubepods-burstable.slice.

Within kubepods-besteffort.slice and kubepods-burstable.slice are multiple additional slices. There isn’t an immediate identifier to tell you which slice contains the resource information for a given container, but you can get that information directly from docker on your application node.

Note

In the following sections, you’ll interact with docker to extract low-level information about the containers that OpenShift creates. Working with docker is outlined in appendix C; if you haven’t already, read appendix C, and then move on to the next section.

9.6.2. Identifying container cgroups

To determine which cgroup slice controls the resources for your app-cli deployment, you need to get the cgroup information from docker. The cgroup slice that each container belongs to is listed in the information from docker inspect. To obtain it, use the -f parameter and specify the {{ .HostConfig.CgroupParent }} object accessor. This limits the output to only the cgroup slice information. In our example cluster, the cgroup slice for app-cli is kubepods-burstable-pod7bdf36bb_a3eb_11e7_a480_001cc4000001.slice, as you can see here:

# docker inspect -f '{{ .HostConfig.CgroupParent }}' 80366fd64c36
kubepods-burstable-pod7bdf36bb_a3eb_11e7_a480_001cc4000001.slice

As we mentioned earlier, app-cli’s cgroup is in the burstable slice. The slice defined in the app-cli docker inspect output is in the burstable slice. Slices don’t define resource constraints for individual containers, but they can set default values for multiple containers. That’s why the hierarchy of slices looks a little excessive here.

You have one more layer to go to get to the resource constraints for the app-cli container. In the lowest slice is a scope directory. Cgroup scopes are where individual process resource constraints are defined. Each scope is named after the full hash that a container’s short ID is based on. In our example, app-cli’s resource constraints are defined in the scope named docker-80366fd64c3630651d80076e5333475438fb6fc8 e34f4525aa94fc99a0e15750.scope. This is how cgroups are mapped to containers in OpenShift (see figure 9.5).

Figure 9.5. Cgroup hierarchy, down to the container-specific scope

Cgroup configurations are created on OpenShift application nodes using this process. It’s a little complex; and because cgroups are listed according to the cgroup controller and not the PID they manage, troubleshooting them can be a challenge on a busy system.

When you need to see the cgroup configuration for a single container, the process is more straightforward. In the next section, we’ll look at how the cgroup information from the host is mounted into each container that’s created.

...
kubepods.slice
kubepods-burstable.slice
kubepods-burstable-pod7bdf36bb_a3eb_11e7_a480_001cc4000001.slice
  |?docker-
 80366fd64c3630651d80076e5333475438fb6fc8e34f4525aa94fc99a0e15750.scope
55965 httpd -D FOREGROUND
56068 /usr/bin/cat
56069 /usr/bin/cat
56070 httpd -D FOREGROUND
56073 httpd -D FOREGROUND
56076 httpd -D FOREGROUND
56091 httpd -D FOREGROUND
56094 httpd -D FOREGROUND
56098 httpd -D FOREGROUND
56102 httpd -D FOREGROUND
56105 httpd -D FOREGROUND
56966 httpd -D FOREGROUND

...

9.6.3. Confirming cgroup resource limits

When docker creates a container, it mounts the cgroup scope that applies to it in the container in the /sys/fs/cgroup directory. It truncates the slices and scope so the container appears to have only a single cgroup controller.

We’re going to focus on the limits that enforce CPU and memory constraints for the app-cli container. Let’s begin with the limits for CPU consumption. To start an interactive shell prompt in your running container, run the following command, edited to reference your container’s short ID:

# docker run -it 80366fd6 bash

This command launches the bash shell prompt in the specified container and provides you with an interactive TTY session. You’ll use this session in the next section.

Verifying container CPU limits

As we discussed earlier in this chapter, CPU resources in OpenShift are allocated in millicores, or one-thousandths of the CPU resources available on the server. For example, if your application node has two CPUs, a total of 2,000 millicores is available for containers on the node.

The ratio expressed here is what’s represented in the cpu cgroup. The actual number isn’t expressed in the same units, but the ratios are always the same.

The app-cli container has a request of 750 millicores, with a limit of 1,000 millicores, or one CPU. You need the following two values from /sys/fs/cgroup/cpu to build a ratio that confirms the limit for the app-cli container:

• cpu.cfs_quota_us—The time, in microseconds, that the cgroup is allowed to access a CPU during the defined period. The period of time is adjustable. For example, if the cfs_quota_us value is 100, the cgroup will be allowed to access the CPU for 100 microseconds during the set period. If that period is also 100, that means the cgroup has unlimited access to a CPU on the system. If the period were set to 1000, the process would have access to the CPU for 100 microseconds out of every 1,000.
• cpu.cfs_period_us—The time period, in microseconds, during which the cgroup’s quota for CPU access is measured and reallocated. This can be manipulated to create different CPU quota ratios for different applications.

For app-cli, this cgroup limits the container’s access to 1 CPU during 100,000 out of every 100,000 microseconds:

bash-4.2$ cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
100000
bash-4.2$ cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000

If you convert these values to a ratio, app-cli is allocated a maximum of 1,000 millicores, or 1 CPU. That’s the limit you set for app-cli. This is how CPU limits are managed for each container in an application deployment. Next, let’s look at how the request values are controlled by cgroups.

Note

The limit for app-gui is 300 millicores, the default limit range for the image-uploader project. Using the same steps, you can verify that its CPU limits match the ratio of 300/1,000 millicores.

The request limit for app-cli is managed by the value in /sys/fs/cgroup/cpu/ cpu.shares. This value is a ratio of CPU resources relative to all the cores on the system.

The CPU request for app-cli is 750 millicores. Because the application node has two CPUs, it should be allocated 750/2,000 of the total CPU capacity for the application node, or 37.5%. The cpu.shares ratio for app-cli should have the same ratio.

The denominator for the cgroup ratio is the total CPU shares available to kubepods .slice on the host. This value is set by multiplying the number of CPUs by 1,024. You can verify this with the cpu.shares value for kubepods.slice, as shown in the following example from the application node:

# cat /sys/fs/cgroup/cpu/kubepods.slice/cpu.shares
2048

To get the numerator for this ratio, run cat /sys/fs/cgroup/cpu/cpu.shares from inside the app-cli container:

$ cat /sys/fs/cgroup/cpu/cpu.shares
768

The app-cli container is allocated 768/2,048 CPU shares on the application node, or 37.5%. The numbers are identical. This is how OpenShift uses the Linux kernel to ensure that resource requests and limits are met for deployed containers.

Next, let’s look at how cgroups enforce memory limits and requests.

Verifying container memory limits

The memory limit for app-cli is controlled by the value in /sys/fs/cgroup/memory/ memory.limit_in_bytes in the app-cli container. This value is expressed in bytes. If you convert it to mebibytes, the value you created the memory limit in, you get 1,000 MiB. This is the defined memory limit for app-cli. To view the value for the upper memory limit for app-cli, use the cat command to echo the file contents from inside the app-cli container:

bash-4.2$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
1048576000

Resource limits for OpenShift containers are enforced with kernel cgroups. The only exception is the memory request value. There’s no cgroup to control the minimum amount of RAM available to a process; this value is primarily used to determine which node a pod is assigned to in your OpenShift cluster.

This chapter covered a lot of what’s required to create and maintain a healthy OpenShift cluster. We’ve gone far down into the Linux kernel to confirm how container resource limits are enforced. Although limits, requests, and quotas aren’t the most exciting things to work through, they’re absolutely critical to make OpenShift ready to handle production workloads effectively.

Your cluster is now connected with an authentication database, and the project you’ve been working with has effective resource limits and quotas. In the following chapters, you’ll keep building on that momentum.

9.7. Summary

• OpenShift can use a long list of user databases for user authentication.
• Special service accounts authenticate with SSL certificates that bypass the user database.
• OpenShift can bind roles that represent project or cluster permission sets to users and groups, to create a full role-based, access-control environment.
• Limit ranges provide minimum, maximum, and default compute resource limits for pods and containers at the project level.
• Limit ranges provide caps for the number of application components, such as services and persistent volume claims, that can be created in a single project.
• Quotas provide aggregate CPU and memory limits for projects. They have multiple scopes so different quotas can be applied to different types of project resources.
• Limits and requests are enforced in the Linux kernel by cgroups.
• Cgroups can be examined from the application node and used to troubleshoot issues when they arise.