Generating and Securing Identity

In providing a signed certificate by which a device may be identified, it is necessary to store the associated private key in a secure manner. This is not an easy task. Theft of the private key would enable an attacker to masquerade as a trusted device. This is the worst possible scenario for device authentication.

A simple yet insecure way to do this is to configure access rights to the key in such a way that only the most privileged user (root or administrator) can access it. This is the least desirable storage method, as an attacker who gains elevated access can exfiltrate the unprotected key.

Another way to do this is to encrypt the private key. This is better than relying on simple permissions, though it presents usability issues because a password (or other secret material) must be furnished in order to decrypt and use the key. This may not pose a problem for an end-user device, as the user can be prompted to enter the password, though this is usually not feasible for server deployments; human interaction is required for every software restart.

The best way by far to store device keys is through the use of secure cryptoprocessors. These devices, commonly referred to as a hardware security module (HSM) or a trusted platform module (TPM), provide a secure area in which cryptographic operations can be performed. They provide a limited API that can be used to generate asymmetric encryption keys, where the private key never leaves the security module. Since not even the operating system can directly access a private key stored by a security module, they are very difficult to steal.

Identity Security in Static and Dynamic Systems

In relatively static systems, it is common for an operator to be involved when new hosts are provisioned. This makes the injection story easy—the trusted human can directly cut the new keys on behalf of the hosts. Of course, as the infrastructure grows, this overhead will become problematic.

In automating the provisioning and signing process, there is an important decision to make: should a human be involved when signing new certificates? The answer to this largely depends on your sensitivity.

A signed device certificate carries quite a bit of power, and serves to identify anything with the private key as an authentic and trusted device. Just as we go through measures to protect their theft locally, we must also protect against their frivolous generation. If your installation is particularly sensitive, you might choose to involve a human every time a new certificate is signed.

If provisioning is automated, but still human-driven, it makes a lot of sense to allow the human driving that action to also authorize the associated signing request. Having a human involved every time is the best way to prevent unauthorized requests from being approved. Humans are not perfect though. They are susceptible to fatigue and other shortcomings. For this reason, it is recommended that they be responsible for approving only requests that they themselves have initiated.

It is possible to accomplish provisioning and signature authorization in a single step through the use of a temporal one-time password (TOTP). The TOTP can be provided along with the provisioning request and passed through to the signing service for verification, as shown in Figure 5-1. This simple yet strong mechanism allows for human control over the signing of new certificates while imposing only minimal administrative overhead. Since a TOTP can only be used once, a TOTP verification failure is an important security event.

Figure 5-1. A human providing a TOTP can safely authorize the signature of a certificate.

It goes without saying that none of this applies if you want to fully automate the provisioning of new hosts. Frequently referred to as “auto-scaling,” systems that can grow and shrink themselves are commonly found in large, highly automated installations.

Allowing a system to scale itself decreases the amount of care and feeding required, significantly reducing administrative overhead and cost. Signing a certificate is an operation that requires a great deal of trust; and just as with other zero trust components, this trust must be sourced from somewhere. There are three common choices:

• A human

• The resource manager

• The image or device

The human is an easy and secure choice for relatively static infrastructure or end user devices, but is an obvious nonstarter for automated infrastructure. In this case, you must choose the resource manager or the image...or both.

The resource manager is in a privileged position. It has the ability to both grow and shrink the infrastructure, and is likely able to influence its availability. It provides a good analog to a human in a more static system. It is in a position to assert, “Yes, I turned this new host on, and here is everything I know about it.” It can use this position to either directly or indirectly authorize the signing of a new certificate.

To make the job of resource manager easier, many cloud vendors such as Microsoft, Google, and others provide built-in support for identities that do not require credentials and can be used to authenticate against specific services in a well-defined manner. Learn more about Google’s service accounts at https://cloud.google.com/compute/docs/access/service-accounts#serviceaccount and Microsoft’s managed identities at https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources.

Depending on your needs, it might be desirable to not grant this ability wholly to the resource manager. In this case, credentials can be baked into an image. This is generally not advised as a primary mechanism, as it places too much responsibility on the image store; and protecting and rotating images can be fraught with peril. In a similar way, HSMs or TPMs can be leveraged to provide a device certificate that is tied to the hardware. This is better than baking material into the image, though requiring a TPM-backed device to sign a new certificate is still not ideal, especially when considering cloud-based deployments.

One good way to mitigate these concerns is to require both the resource manager and a trusted image/device. Generic authentication material baked into the image (or a registered TPM key) can be used to secure communication with the signing service and can serve as a component in a multifaceted authorization. The following are examples of components for authorization consideration:

• Registered TPM key or image key

• Correct IP address

• Valid TOTP (generated by resource manager)

• Expected certificate properties (i.e., expected common name)

By validating all of these points, the certificate signing service can be relatively certain that the request is legitimate. The resource manager alone cannot request a certificate, and since it does not have access to the hosts it provisions, the most an attacker could do is impact availability. Similarly, a stolen image alone cannot request a certificate, as it requires the resource manager to validate that it has provisioned the host and expects the request.

By splitting these responsibilities and requiring multiple systems to assert validity, we can safely (well, as safely as is possible) remove humans from the loop.

X.509

X.509 is perhaps the most important standard we have when it comes to device identity and authentication. It defines the format for public key certificates, revocation lists, and methods through which to validate certification chains. The framework it puts forth aids in the formation of identity used for secure device authentication in nearly every protocol we’ll discuss in this book.

One of the coolest things about X.509 is that the public/private key pairs it uses to prove identity can also be used to bootstrap encrypted communication. This is just one of many reasons that X.509 is so valuable for internet security.

Please refer to RFC5280, RFC4519, and the ITU-T X.509 documentation for more information on the X.509 certificate format and supported attributes.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 16210155439472130208 (0xe0f60a7cb39a38a0)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer:C=US, ST=TX, L=Houston, O=Contoso Corp                         
        Validity
            Not Before: Aug 18 22:54:43 2022 GMT
            Not After : Aug 18 22:54:43 2025 GMT
        Subject: C=US, ST=TX, L=Dallas, O=Contoso Corp, CN=mgmt011134.contoso.corp 
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:af:ff:04:2e:69:96:40:eb:62:20:a8:db:61:06:
                    ………………………………                       
                    3f:bd:b1:49:50:26:07:ac:72:c7:9b:81:5d:54:19:
                    88:8b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name: 
                DNS:mgmt011134.contoso.corp, 
                DNS:10.69.54.169,  
                DNS:fdc0:8a12:793b:adf7:8da6:12cd:e34c:daf6
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication,
                TLS Web Server Authentication
    Signature Algorithm: sha256WithRSAEncryption
         1e:e1:ed:8a:40:85:ac:fb:85:78:9c:88:ee:75:30:76:14:79:
         …………
         8d:9f:44:ea

TPMs

A trusted platform module (TPM) is a special chip that is embedded in a compute device called a cryptoprocessor, these chips are dedicated to performing cryptographic operations in a trusted and secure way. They include their own firmware and are often thought of as a computer on a chip.

This design enables a small and lean hardware API that is easily audited and analyzed for vulnerability. By providing facilities for cryptographic operations, and excluding interfaces for retrieving private keys, we get the security we need without ever exposing secret keys to the operating system. Instead, they are bound to the hardware.

This is a very important property and the reason that TPMs are so important for device authentication in zero trust networks. Great software frameworks for identity and authentication (like X.509) do a lot for device authentication. But without a way to bind the software key to the hardware device it is attempting to identify, we can‐ not really call it device identity. TPMs solve this problem, providing the necessary binding.

Encrypting data using a TPM

TPMs generate and store what is known as a storage root key, or an SRK. This key pair represents the trust root for the TPM device. Data encrypted using its public key can be decrypted by the originating TPM only.

The astute reader might question the usefulness of this function in the application of bulk data encryption. We know asymmetric cryptographic operations to be very expensive, and thus not suitable for the encryption of relatively large pieces of data. Thus, in order to leverage the TPM for bulk data encryption, we must reduce the amount of data that the SRK is responsible for securing.

An easy way to do this is to generate a random encryption key, encrypt the bulk data using known-performant symmetric encryption (i.e., AES), and then use the SRK to encrypt the resulting AES key. This strategy, shown in Figure 5-2, ensures that the encryption key cannot be recovered, unless in the presence of the TPM that originally protected it.

Figure 5-2. The data is encrypted with an AES key, which in turn is encrypted by the TPM

Most TPM libraries available for open consumption perform these steps for you, through the use of helper methods. It is recommended to inspect the internal operation of such methods before using them.

Intermediary keys and passphrases

Many TPM libraries (such as TrouSerS) create intermediary keys when encrypting data using the TPM. That is, they ask the TPM to create a new asymmetric key pair, use the public key to encrypt the AES key, and finally use the SRK to encrypt the private key. When decrypting the data, you must first decrypt the intermediate private key, use it to decrypt the AES key, then decrypt the original data.

This implementation seems strange, but there are some relatively sane reasons for it. One reason is that the additional level of indirection allows for more flexibility in the distribution of secured data. Both the SRK and intermediate keys support passphrases, so the use of an intermediary key enables the use of an additional, perhaps more widely known, passphrase.

This may or may not make sense for your particular deployment. For the purposes of “This key should only be decryptable on this device only,” it is OK (and more performant) to bypass the use of an intermediary key, if desired.

The most important application of TPM-backed secure storage is in protecting the device’s X.509 private key. This secret key serves to authoritatively prove device identity, and if stolen, so is the identity. Encrypting the private key using TPM means that while the key might still be taken from disk, it will not be recoverable without the original hardware.

Key Theft Is Still Possible

Encrypting the device’s private key and wrapping the key with the SRK does not solve all of the theft vectors. It protects the key from being directly read from disk, though an attacker with elevated privileges might still be able to read it from memory or simply ask the TPM to perform the operation for them. The following two sections provide additional information on how to further validate hardware identity (beyond X.509 identity).

PCR-00: A8 5A 84 B7 38 FC ...         # BIOS
PCR-01: 11 40 C1 7D 0D 25 ...         # BIOS Configuration
PCR-02: A3 82 9A 64 61 85 ...         # Option ROM
PCR-03: B2 A8 3B 0E BF 2F ...         # Option ROM Configuration
PCR-04: 78 93 CF 58 0E E1 ...         # MBR
PCR-05: 72 A7 A9 6C 96 39 ...         # MBR Configuration

Hardware-Based Zero Trust Supplicant?

The most common approach for supporting legacy devices in a zero trust network is to use an authentication proxy. The authentication proxy terminates the zero trust relationship and forwards the connection to the legacy host.

While it is possible to enforce policy between the authentication proxy and the legacy backend, this mode of operation is less than ideal and shares a handful of attack vectors with traditional perimeter networks. When dealing with legacy devices, it is desirable to push the zero trust termination point as close to the device as possible.

When possible, it is preferable to use a dedicated hardware device rather than an application proxy. This device can act as a zero trust supplicant, carrying a TPM chip, and plug directly into a legacy device’s Ethernet port. Pairing the two in your inventory management system can allow for seamless integration between legacy devices and a zero trust network.

There are many applications that would significantly benefit from such a device. SCADA and HVAC systems, for instance, come to mind.

Knowing What to Expect

One of the great powers of a zero trust network is that it knows what to expect. Trusted entities can push expectations into the system, allowing all levels of access to be denied by default—only expected actions/requests are permitted.

An inventory database is a major component in realizing this capability. A huge amount of information about what to expect can be generated from this data; things like which user or application should be running on it, what locations we might expect it to be in, or even the kind of operating system are all pieces of information that can be used to set expectations.

In the datacenter, these expectations can be very strong. For instance, when provisioning a new server, we often know what IP address it will be assigned and what purpose it will serve. We can use that information to drive network ACLs and/or host based firewalls, poking holes for that specific IP address only where necessary. In this way, we can have all traffic denied, allowing only the very specific flows we are expecting. The more properties that can be expected, the better.

This is not such an easy prospect for client-facing systems, however. Clients operate in new and unexpected ways all the time, and knowing exactly what to expect from them and when is very difficult. Servers in the datacenter often have relatively static and long-lived connections to a well-defined set of hosts or services. By contrast, clients tend to make many short-lived connections to a variety of services, the timing, frequency, and patterns of which can vary organically.

In order to address the wild nature of client-facing systems, we need a slightly different approach. One way to do this is to simply allow global access to the service and to protect it with mutually authenticated TLS, forcing the client to provide a device certificate before it can communicate with it. The device certificate can be used to look the device up in the inventory database and determine whether or not to authorize it. The advantage is that lots of systems support mutually authenticated TLS already, and specialized client software is not strictly required. One can provide reasonably strong security without too badly hindering accessibility or usability.

A significant drawback to this approach, however, is that the service is globally reachable. Requiring client certificates is a great way to mitigate this danger. However, we have seen from vulnerabilities like Heartbleed that the attack surface of a TLS server is relatively large. Additionally, the existence of the resources can be discovered by simply scanning for them, since we get to speak TCP to the resource before we authenticate with it.

How can we ensure that we don’t engage clients that are not trusted? There has to be some untrusted communication, after all. What comes before the authentication?

Secure Introduction

The very first connection from a new device is a precarious one. After all, these packets must be admitted somewhere, and if they are not strongly authenticated, then there is a risk. Therefore, the first system that a new device contacts needs a mechanism by which it can authenticate this initial contact.

This arrangement is commonly known as secure introduction. It is the process through which a new entity is introduced to an existing one in a way that trust is transferred to it. There are many ways in which this can be affected; the method through which an operator passes a TOTP code to a provisioner in order to authorize a certificate request is a form of secure introduction.

The best (and perhaps only) way to do secure introduction is by setting an expectation. Secure introduction practically always involves a trusted third party. This is a system that is already introduced, and it holds the ability to introduce new systems. This trusted third party is the system that then coordinates/validates the specifics of the system to be introduced and sets the appropriate expectations.

Local Measurement

Local measurement can be one of two types: hardware-backed or software-backed. Hardware-backed measurement is more secure and reliable, but limited in capability. Software-backed measurement is much less secure and reliable, but practically unlimited in its measurement capabilities.

One good option for hardware-backed local measurement is leveraging the TPM for remote attestation. Remote attestation uses a hardware device to provide a signed response outlining the hashes of the software currently running on that machine. The response is highly reliable and very difficult to reproduce. However, it generally only gives a picture of the low-level software or specifically targeted software. If an attacker has managed to get an unauthorized process running in user space, the TPM will not be very useful in its detection; thus, it has limited capability. See “Remote attestation” on page XX [78] for more information.

Software-backed local measurement involves some sort of agent installed on the endpoint which is used to report health and state measurements. This could be anything from a managed antivirus client to policy enforcement agents. These agents go to great lengths in order to attest and prove validity of the measurements they report, but even cursory thought quickly reaches the conclusion that these efforts are generally futile. Software-backed measurements lack the protection provided by hardware measurements, and an attacker with sufficient privilege can subvert systems like this.

Remote Measurement

Remote measurement is the best of the two options for one simple reason: it benefits from separation of duty. A compromised host can report whatever it wants to, possibly falsifying information in order to conceal the attacker. This is not possible with remote or passive measurement, since a completely different system is determining the health of the host in question.

Traditionally, remote measurement is performed as a simple vulnerability scan. The system in question will be periodically probed by a scanning device, which observes the response. The response gives some information away, like what operating system might be running on that device, what services might be active there, and maybe even what version of those services.

The scan results can be cross-referenced with known-bad signatures, like malicious software or vulnerable versions of legitimate software. Detection of known-bad signatures can then influence the trust of the device appropriately.

There are a number of open source and commercial options available in the vulnerability scanning arena, including OpenVAS, Nessus, and Metasploit. These projects are all fairly mature and relied on by many organizations.

Unfortunately, vulnerability scanning comes with the same fundamental problem as local measurement: it relies on interrogation of the endpoint. It’s the difference between asking someone if they robbed a bank, and watching them rob a bank. Sure, sometimes you can get the robber to admit that they did it, but a professional would never fall for that. Catching them in the act is much more effective. See “Network Communication Patterns” on page XX [92] for more about how to solve this dilemma.

    "iss": "https://idp.example.com/123456789/",
    "jti": "24c63fb56e5a2d77a6b512616ca9fa24",
    "iat": 1615305159,
    "aud": "https://sp.example.com/caep",
    "events": {
        "https://schemas.openid.net/secevent/caep/event-type/
        device-compliance-change": {
            "subject": {
                "device": {
                    "format": "iss_sub",
                    "iss": "https://idp.example.com/123456789/",
                    "sub": "e9297990-14d2-42ec-a4a9-4036db86509a"
                },
                "tenant": {
                    "format": "opaque",
                    "id": "123456789"
                }
            },
            "current_status": "not-compliant",
            "previous_status": "compliant",
            "initiating_entity": "policy",
            "reason_admin": {
                "en": "Location Policy Violation: C076E8A3"
            },
            "reason_user": {
                "en": "Device is no longer in a trusted location."
            },
            "event_timestamp": 1615304991643
        }
    }
}

CM-Based Inventory

We have mentioned several times the idea of using a configuration management database for inventory management purposes. This is a great first step toward a mature inventory management system and can provide a rich source of information about the various hosts and software running in your infrastructure.

We like to think that CM-based inventory management is a “freebie” in that configuration management is typically leveraged for the bevy of other benefits it brings.

Using it as an inventory database most often comes about out of convenience. Maintaining this view is important: configuration management systems aren’t designed to act as inventory management systems...they’re designed to act as configuration management systems! Using it as such will surely bring a few rough edges, and you will eventually outgrow it. This is not to say don’t do it. It is better to actually realize a zero trust network by leveraging as much existing technology as possible than it is to never get there due to the high barrier to entry. Once we accept this fact, we can begin to leverage the wealth of data provided to us by the CM agents.

Secure Source of Truth

One important thing to remember when using CM systems in the zero trust control plane is that the vast majority of the data available to CM systems is self-reported. This is critical to understand, since a compromised machine could potentially misrepresent itself. This can lead to complete compromise of the zero trust network if these facts are not considered during its design.

Thinking back to trust management, the trusted system in this case is the provisioner. Whether it be a human or some automated system, it is in the best position to assert the critical aspects of a device, which include the following:

• Device type

• Role

• IP address (in datacenter systems)

• Public key

These attributes are considered critical because they are often used in making authorization or authentication decisions. If an attacker can update the device role, for instance, perhaps they can coerce the network to expose protected services. For this reason, restricting write access to these attributes is important. Of course, you can still use self-reported attributes for making decisions, but they should not be considered fact under any circumstance. It’s useful to think of self-reported attributes as hints rather than truth.

Time Since Image

Over time, the likelihood that a device has been compromised increases dramatically. Endpoint security practices aim to decrease the risk associated with long-lived or long-running devices. Still, these practices are far from perfect.

Imaging a device ensures that the contents of the hard drive match a known good. While not effective against some lower-level attacks, it provides a reasonably strong assurance of trust. In the moments immediately following the image restore, a tremendous amount of trust exists in the device, as only the hardware or the restore system itself would be able to taint the process. Over time though, that trust wears off as the system goes through prolonged exposure.

Historical Access

Device authentication patterns, similar to user authentication patterns, are important in understanding risk and act as a nice proxy for behavioral filtering. Devices which have not been seen in a while are more suspicious than ones that come and go frequently. Maybe suspicious is the wrong word, but it’s certainly unusual.

The request in question can also be tied to a resource, and it is wise to consider the device and the resource together in this context. For instance, a months-old device requesting access to a new resource is more suspicious than a request to a resource it has been accessing weekly for some time. This stands to say that the “first few” access attempts to a particular resource will be viewed with more skepticism than subsequent attempts.

Similarly, frequency can be analyzed to understand if a resource is being suspiciously over-utilized. A request from a device that has made 100 requests in the last day, but only 104 over the last month, is certainly more suspicious than one with 0 in the last day and 4 in the last month.

Location

While network location, including attributes like geo-location, IP Address, etc, are typically something we aim to not make strong decisions on with regard to the zero trust model, they still provides reliable trust signaling in many cases.

One such case might be a sudden location change. Since we are talking about device authentication, we can set some reasonable expectations about the way that device moves around. For instance, a device authentication attempt from Europe might be pretty suspicious if we have authorized that same device in the US office just a couple hours prior.

It should be noted that this is a bit of a slippery slope when it comes to the zero trust model. Zero trust aims to eliminate positions of advantage within the network, so using network location to determine access right can be considered a little contradictory. The authors recognize this and acknowledge that location-related data can be valuable while making authorization decisions. That said, it is important that this consideration not be binary. One should look for patterns in locations, and never make an absolute decision based solely on location. For instance, a policy which dictates that an application can only be accessed from the office is a direct violation of the zero trust model.

Network Communication Patterns

For devices that are connected to networks owned by the operator, there is an opportunity to measure communication patterns to develop a norm. Sudden changes from this norm are suspicious and can affect how much the system trusts such a device. Network instrumentation and flow collection can quickly detect intrusions by observing them on the network. Making authorization decisions informed by this detection is very powerful. One example might be shutting down database access to a particular web server because that web server began making DNS queries for hosting providers on another continent.

The same applies to client devices. Consider a desktop that has never before initiated an SSH connection but is now frequently SSHing to internet hosts. It is fair to say that this change in behavior is suspicious and should result in the device being less trusted than it was previously.

Machine Learning

Machine learning assists in calculating trust score by considering the entire context of the access request, including the user, device, and resource requested, as well as historical activity to identify anomalous requests. Machine learning models are typically trained over time to distinguish between anomalous and legitimate access patterns. As a result, machine learning can assist in reducing any potential blind spots in identifying malicious requests. However, as with other aspects of zero trust, relying solely on machine learning is not recommended; rather, using it in conjunction with other trust signals yields the best results.

Use case: Bob wants to send a document for printing

Here is what we know about Bob’s request:

• Bob is requesting access to send a document to an organization’s printer.

• Bob is using his laptop with the device id “ABC.”

• Bob has used MFA and also used password as the first factor for authentication

• Bob is making the request during office hours.

Use case: Bob wants to delete an email

Here is what we know about Bob’s request:

• Bob wants to delete an email from his inbox.

• Bob is using his mobile phone with the device id “XYZ.”

• Bob has used MFA and also used his password as the first factor for authentication

• Bob is making the request during office hours.