7
NETWORK PROTOCOL SECURITY
Network protocols transfer information between participants in a network, and there’s a good chance that information is sensitive. Whether the information includes credit card details or top secret information from government systems, it’s important to provide security. Engineers consider many requirements for security when they initially design a protocol, but vulnerabilities often surface over time, especially when a protocol is used on public networks where anyone monitoring traffic can attack it.
All secure protocols should do the following:
• Maintain data confidentiality by protecting data from being read
• Maintain data integrity by protecting data from being modified
• Prevent an attacker from impersonating the server by implementing server authentication
• Prevent an attacker from impersonating the client by implementing client authentication
In this chapter, I’ll discuss ways in which these four requirements are met in common network protocols, address potential weaknesses to look out for when analyzing a protocol, and describe how these requirements are implemented in a real-world secure protocol. I’ll cover how to identify which protocol encryption is in use or what flaws to look for in subsequent chapters.
The field of cryptography includes two important techniques many network protocols use, both of which protect data or a protocol in some way: encryption provides data confidentiality, and signing provides data integrity and authentication.
Secure network protocols heavily use encryption and signing, but cryptography can be difficult to implement correctly: it’s common to find implementation and design mistakes that lead to vulnerabilities that can break a protocol’s security. When analyzing a protocol, you should have a solid understanding of the technologies and algorithms involved so you can spot and even exploit serious weaknesses. Let’s look at encryption first to see how mistakes in the implementation can compromise the security of an application.
Encryption Algorithms
The history of encryption goes back thousands of years, and as electronic communications have become easier to monitor, encryption has become considerably more important. Modern encryption algorithms often rely on very complex mathematical models. However, just because a protocol uses complex algorithms doesn’t mean it’s secure.
We usually refer to an encryption algorithm as a cipher or code depending on how it’s structured. When discussing the encrypting operation, the original, unencrypted message is referred to as plaintext. The output of the encryption algorithm is an encrypted message called cipher text. The majority of algorithms also need a key for encryption and decryption. The effort to break or weaken an encryption algorithm is called cryptanalysis.
Many algorithms that were once thought to be secure have shown numerous weaknesses and even backdoors. In part, this is due to the massive increase in computing performance since the invention of such algorithms (some of which date back to the 1970s), making feasible attacks that we once thought possible only in theory.
If you want to break secure network protocols, you need to understand some of the well-known cryptographic algorithms and where their weaknesses lie. Encryption doesn’t have to involve complex mathematics. Some algorithms are only used to obfuscate the structure of the protocol on the network, such as strings or numbers. Of course, if an algorithm is simple, its security is generally low. Once the mechanism of obfuscation is discovered, it provides no real security.
Here I’ll provide an overview some common encryption algorithms, but I won’t cover the construction of these ciphers in depth because in protocol analysis, we only need to understand the algorithm in use.
Substitution Ciphers
A substitution cipher is the simplest form of encryption. Substitution ciphers use an algorithm to encrypt a value based on a substitution table that contains one-to-one mapping between the plaintext and the corresponding cipher text value, as shown in Figure 7-1. To decrypt the cipher text, the process is reversed: the cipher value is looked up in a table (that has been reversed), and the original plaintext value is reproduced. Figure 7-1 shows an example substitution cipher.
Figure 7-1: Substitution cipher encryption
In Figure 7-1, the substitution table (meant as just a simple example) has six defined substitutions shown to the right. In a full substitution cipher, many more substitutions would typically be defined. During encryption, the first letter is chosen from the plaintext, and the plaintext letter’s substitution is then looked up in the substitution table. Here, H in HELLO is replaced with the letter X. This process continues until all the letters are encrypted.
Although substitution can provide adequate protection against casual attacks, it fails to withstand cryptanalysis. Frequency analysis is commonly used to crack substitution ciphers by correlating the frequency of symbols found in the cipher text with those typically found in plaintext data sets. For example, if the cipher protects a message written in English, frequency analysis might determine the frequency of certain common letters, punctuation, and numerals in a large body of written works. Because the letter E is the most common in the English language, in all probability the most frequent character in the enciphered message will represent E. By following this process to its logical conclusion, it’s possible to build the original substitution table and decipher the message.
XOR Encryption
The XOR encryption algorithm is a very simple technique for encrypting and decrypting data. It works by applying the bitwise XOR operation between a byte of plaintext and a byte of the key, which results in the cipher text. For example, given the byte 0x48 and the key byte 0x82, the result of XORing them would be 0xCA.
Because the XOR operation is symmetric, applying that same key byte to the cipher text returns the original plaintext. Figure 7-2 shows the XOR encryption operation with a single-byte key.
Figure 7-2: An XOR cipher operation with a single-byte key
Specifying a single-byte key makes the encryption algorithm very simple and not very secure. It wouldn’t be difficult for an attacker to try all 256 possible values for the key to decrypt the cipher text into plaintext, and increasing the size of the key wouldn’t help. As the XOR operation is symmetric, the cipher text can be XORed with the known plaintext to determine the key. Given enough known plaintext, the key could be calculated and applied to the rest of the cipher text to decrypt the entire message.
The only way to securely use XOR encryption is if the key is the same size as the message and the values in the key are chosen completely at random. This approach is called one-time pad encryption and is quite difficult to break. If an attacker knows even a small part of the plaintext, they won’t be able to determine the complete key. The only way to recover the key would be to know the entire plaintext of the message; in that case, obviously, the attacker wouldn’t need to recover the key.
Unfortunately, the one-time pad encryption algorithm has significant problems and is rarely used in practice. One problem is that when using a one-time pad, the size of the key material you send must be the same size as any message to the sender and recipient. The only way a one time pad can be secure is if every byte in the message is encrypted with a completely random value. Also, you can never reuse a one-time pad key for different messages, because if an attacker can decrypt your message one time, then they can recover the key, and then subsequent messages encrypted with the same key are compromised.
If XOR encryption is so inferior, why even mention it? Well, even though it isn’t “secure,” developers still use it out of laziness because it’s easy to implement. XOR encryption is also used as a primitive to build more secure encryption algorithms, so it’s important to understand how it works.
Random Number Generators
Cryptographic systems heavily rely on good quality random numbers. In this chapter, you’ll see them used as per-session keys, initialization vectors, and the large primes p and q for the RSA algorithm. However, getting truly random data is difficult because computers are by nature deterministic: any given program should produce the same output when given the same input and state.
One way to generate relatively unpredictable data is by sampling physical processes. For example, you could time a user’s key presses on the keyboard or sample a source of electrical noise, such as the thermal noise in a resistor. The trouble with these sorts of sources is they don’t provide much data—perhaps only a few hundred bytes every second at best, which isn’t enough for a general purpose cryptographic system. A simple 4096-bit RSA key requires at least two random 256-byte numbers, which would take several seconds to generate.
To make this sampled data go further, cryptographic libraries implement pseudorandom number generators (PRNGs), which use an initial seed value and generate a sequence of numbers that, in theory, shouldn’t be predictable without knowledge of the internal state of the generator. The quality of PRNGs varies wildly between libraries: the C library function rand(), for instance, is completely useless for cryptographically secure protocols. A common mistake is to use a weak algorithm to generate random numbers for cryptographic uses.
Symmetric Key Cryptography
The only secure way to encrypt a message is to send a completely random key that’s the same size as the message before the encryption can take place as a one-time pad. Of course, we don’t want to deal with such large keys. Fortunately, we can instead construct a symmetric key algorithm that uses mathematical constructs to make a secure cipher. Because the key size is considerably shorter than the message you want to send and doesn’t depend on how much needs to be encrypted, it’s easier to distribute.
If the algorithm used has no obvious weakness, the limiting factor for security is the key size. If the key is short, an attacker could brute-force the key until they find the correct one.
There are two main types of symmetric ciphers: block and stream ciphers. Each has its advantages and disadvantages, and choosing the wrong cipher to use in a protocol can seriously impact the security of network communications.
Block Ciphers
Many well-known symmetric key algorithms, such as the Advanced Encryption Standard (AES) and the Data Encryption Standard (DES), encrypt and decrypt a fixed number of bits (known as a block) every time the encryption algorithm is applied. To encrypt or decrypt a message, the algorithm requires a key. If the message is longer than the size of a block, it must be split into smaller blocks and the algorithm applied to each in turn. Each application of the algorithm uses the same key, as shown in Figure 7-3. Notice that the same key is used for encryption and decryption.
Figure 7-3: Block cipher encryption
When a symmetric key algorithm is used for encryption, the plaintext block is combined with the key as described by the algorithm, resulting in the generation of the cipher text. If we then apply the decryption algorithm combined with the key to the cipher text, we recover the original plaintext.
DES
Probably the oldest block cipher still used in modern applications is the DES, which was originally developed by IBM (under the name Lucifer) and was published as a Federal Information Processing Standard (FIPS) in 1979. The algorithm uses a Feistel network to implement the encryption process. A Feistel network, which is common in many block ciphers, operates by repeatedly applying a function to the input for a number of rounds. The function takes as input the value from the previous round (the original plaintext) as well as a specific subkey that is derived from the original key using a key-scheduling algorithm.
The DES algorithm uses a 64-bit block size and a 64-bit key. However, DES requires that 8 bits of the key be used for error checking, so the effective key is only 56 bits. The result is a very small key that is unsuitable for modern applications, as was proven in 1998 by the Electronic Frontier Foundation’s DES cracker—a hardware-key brute-force attacker that was able to discover an unknown DES key in about 56 hours. At the time, the custom hardware cost about $250,000; today’s cloud-based cracking tools can crack a key in less than a day far more cheaply.
Triple DES
Rather than throwing away DES completely, cryptographers developed a modified form that applies the algorithm three times. The algorithm in Triple DES (TDES or 3DES) uses three separate DES keys, providing an effective key size of 168 bits (although it can be proven that the security is actually lower than the size would suggest). As shown in Figure 7-4, in Triple DES, the DES encrypt function is first applied to the plaintext using the first key. Next, the output is decrypted using the second key. Then the output is encrypted again using the third key, resulting in the final cipher text. The operations are reversed to perform decryption.
Figure 7-4: The Triple DES encryption process
AES
A far more modern encryption algorithm is AES, which is based on the algorithm Rijndael. AES uses a fixed block size of 128 bits and can use three different key lengths: 128, 192, and 256 bits; they are sometimes referred to as AES128, AES192, and AES256, respectively. Rather than using a Feistel network, AES uses a substitution-permutation network, which consists of two main components: substitution boxes (S-Box) and permutation boxes (P-Box). The two components are chained together to form a single round of the algorithm. As with the Feistel network, this round can be applied multiple times with different values of the S-Box and P-Box to produce the encrypted output.
An S-Box is a basic mapping table not unlike a simple substitution cipher. The S-Box takes an input, looks it up in a table, and produces output. As an S-Box uses a large, distinct lookup table, it’s very helpful in identifying particular algorithms. The distinct lookup table provides a very large fingerprint, which can be discovered in application executables. I explained this in more depth in Chapter 6 when I discussed techniques to find unknown cryptographic algorithms by reverse engineering binaries.
Other Block Ciphers
DES and AES are the block ciphers that you’ll most commonly encounter, but there are others, such as those listed in Table 7-1 (and still others in commercial products).
Table 7-1: Common Block Cipher Algorithms
Cipher name |
Block size (bits) |
Key size (bits) |
Year introduced |
Data Encryption Standard (DES) |
64 |
56 |
1979 |
Blowfish |
64 |
32–448 |
1993 |
Triple Data Encryption Standard (TDES/3DES) |
64 |
56, 112, 168 |
1998 |
Serpent |
128 |
128, 192, 256 |
1998 |
Twofish |
128 |
128, 192, 256 |
1998 |
Camellia |
128 |
128, 192, 256 |
2000 |
Advanced Encryption Standard (AES) |
128 |
128, 192, 256 |
2001 |
The block and key size help you determine which cipher a protocol is using based on the way the key is specified or how the encrypted data is divided into blocks.
Block Cipher Modes
The algorithm of a block cipher defines how the cipher operates on blocks of data. Alone, a block-cipher algorithm has some weaknesses, as you’ll soon see. Therefore, in a real-world protocol, it is common to use the block cipher in combination with another algorithm called a mode of operation. The mode provides additional security properties, such as making the output of the encryption less predictable. Sometimes the mode also changes the operation of the cipher by, for example, converting a block cipher into a stream cipher (which I’ll explain in more detail in “Stream Ciphers” on page 158). Let’s take a look at some of the more common modes as well as their security properties and weaknesses.
Electronic Code Book
The simplest and default mode of operation for block ciphers is Electronic Code Book (ECB). In ECB, the encryption algorithm is applied to each fixed-size block from the plaintext to generate a series of cipher text blocks. The size of the block is defined by the algorithm in use. For example, if AES is the cipher, each block in ECB mode must be 16 bytes in size. The plaintext is divided into individual blocks, and the cipher algorithm applied. (Figure 7-3 showed the ECB mode at work.)
Because each plaintext block is encrypted independently in ECB, it will always encrypt to the same block of cipher text. As a consequence, ECB doesn’t always hide large-scale structures in the plaintext, as in the bitmap image shown in Figure 7-5. In addition, an attacker can corrupt or manipulate the decrypted data in independent-block encryption by shuffling around blocks of the cipher text before it is decrypted.
Figure 7-5: ECB encryption of a bitmap image
Cipher Block Chaining
Another common mode of operation is Cipher Block Chaining (CBC), which is more complex than ECB and avoids its pitfalls. In CBC, the encryption of a single plaintext block depends on the encrypted value of the previous block. The previous encrypted block is XORed with the current plaintext block, and then the encryption algorithm is applied to this combined result. Figure 7-6 shows an example of CBC applied to two blocks.
At the top of Figure 7-6 are the original plaintext blocks. At the bottom is the resulting cipher text generated by applying the block-cipher algorithm as well as the CBC mode algorithm. Before each plaintext block is encrypted, the plaintext is XORed with the previous encrypted block. After the blocks have been XORed together, the encryption algorithm is applied. This ensures that the output cipher text is dependent on the plaintext as well as the previous encrypted blocks.
Figure 7-6: The CBC mode of operation
Because the first block of plaintext has no previous cipher text block with which to perform the XOR operation, you combine it with a manually chosen or randomly generated block called an initialization vector (IV). If the IV is randomly generated, it must be sent with the encrypted data, or the receiver will not be able to decrypt the first block of the message. (Using a fixed IV is an issue if the same key is used for all communications, because if the same message is encrypted multiple times, it will always encrypt to the same cipher text.)
To decrypt CBC, the encryption operations are performed in reverse: decryption happens from the end of the message to the front, decrypting each cipher text block with the key and at each step XORing the decrypted block with the encrypted block that precedes it in the cipher text.
Alternative Modes
Other modes of operation for block ciphers are available, including those that can convert a block cipher into a stream cipher, and special modes, such as Galois Counter Mode (GCM), which provide data integrity and confidentiality. Table 7-2 lists several common modes of operation and indicates whether they generate a block or stream cipher (which I’ll discuss in the section “Stream Ciphers” on page 158). To describe each in detail would be outside the scope of this book, but this table provides a rough guide for further research.
Table 7-2: Common Block Cipher Modes of Operation
Mode name |
Abbreviation |
Mode type |
Electronic Code Book |
ECB |
Block |
Cipher Block Chaining |
CBC |
Block |
Output Feedback |
OFB |
Stream |
Cipher Feedback |
CFB |
Stream |
Counter |
CTR |
Stream |
Galois Counter Mode |
GCM |
Stream with data integrity |
Block Cipher Padding
Block ciphers operate on a fixed-size message unit: a block. But what if you want to encrypt a single byte of data and the block size is 16 bytes? This is where padding schemes come into play. Padding schemes determine how to handle the unused remainder of a block during encryption and decryption.
The simplest approach to padding is to pad the extra block space with a specific known value, such as a repeating-zero byte. But when you decrypt the block, how do you distinguish between padding bytes and meaningful data? Some network protocols specify an explicit-length field, which you can use to remove the padding, but you can’t always rely on this.
One padding scheme that solves this problem is defined in the Public Key Cryptography Standard #7 (PKCS#7). In this scheme, all the padded bytes are set to a value that represents how many padded bytes are present. For example, if three bytes of padding are present, each byte is set to the value 3, as shown in Figure 7-7.
Figure 7-7: Examples of PKCS#7 padding
What if you don’t need padding? For instance, what if the last block you’re encrypting is already the correct length? If you simply encrypt the last block and transmit it, the decryption algorithm will interpret legitimate data as part of a padded block. To remove this ambiguity, the encryption algorithm must send a final dummy block that only contains padding in order to signal to the decryption algorithm that the last block can be discarded.
When the padded block is decrypted, the decryption process can easily verify the number of padding bytes present. The decryption process reads the last byte in the block to determine the expected number of padding bytes. For example, if the decryption process reads a value of 3, it knows that three bytes of padding should be present. The decryption process then reads the other two bytes of expected padding, verifying that each byte also has a value of 3. If padding is incorrect, either because all the expected padding bytes are not the same value or the padding value is out of range (the value must be less than or equal to the size of a block and greater than 0), an error occurs that could cause the decryption process to fail. The manner of failure is a security consideration in itself.
Padding Oracle Attack
A serious security hole, known as the padding oracle attack, occurs when the CBC mode of operation is combined with the PKCS#7 padding scheme. The attack allows an attacker to decrypt data and in some cases encrypt their own data (such as a session token) when sent via this protocol, even if they don’t know the key. If an attacker can decrypt a session token, they might recover sensitive information. But if they can encrypt the token, they might be able to do something like circumvent access controls on a website.
For example, consider Listing 7-1, which decrypts data from the network using a private DES key.
def decrypt_session_token(byte key[])
{
➊ byte iv[] = read_bytes(8);
byte token[] = read_to_end();
➋ bool error = des_cbc_decrypt(key, iv, token);
if(error) {
➌ write_string("ERROR");
} else {
➍ write_string("SUCCESS");
}
}
Listing 7-1: A simple DES decryption from the network
The code reads the IV and the encrypted data from the network ➊ and passes it to a DES CBC decryption routine using an internal application key ➋. In this case, it decrypts a client session token. This use case is common in web application frameworks, where the client is effectively stateless and must send a token with each request to verify its identity.
The decryption function returns an error condition that signals whether the decryption failed. If so, it sends the string ERROR to the client ➌; otherwise, it sends the string SUCCESS ➍. Consequently, this code provides an attacker with information about the success or failure of decrypting an arbitrary encrypted block from a client. In addition, if the code uses PKCS#7 for padding and an error occurs (because the padding doesn’t match the correct pattern in the last decrypted block), an attacker could use this information to perform the padding oracle attack and then decrypt the block of data the attacker sent to a vulnerable service.
This is the essence of the padding oracle attack: by paying attention to whether the network service successfully decrypted the CBC-encrypted block, the attacker can infer the block’s underlying unencrypted value. (The term oracle refers to the fact that the attacker can ask the service a question and receive a true or false answer. Specifically, in this case, the attacker can ask whether the padding for the encrypted block they sent to the service is valid.)
To better understand how the padding oracle attack works, let’s return to how CBC decrypts a single block. Figure 7-8 shows the decryption of a block of CBC-encrypted data. In this example, the plaintext is the string Hello with three bytes of PKCS#7 padding after it.
By querying the web service, the attacker has direct control over the original cipher text and the IV. Because each plaintext byte is XORed with an IV byte during the final decryption step, the attacker can directly control the plaintext output by changing the corresponding byte in the IV. In the example shown in Figure 7-8, the last byte of the decrypted block is 0x2B, which gets XORed with the IV byte 0x28 and outputs 0x03, a padding byte. But if you change the last IV byte to 0xFF, the last byte of the cipher text decrypts to 0xD4, which is no longer a valid padding byte, and the decryption service returns an error.
Figure 7-8: CBC decryption with IV
Now the attacker has everything they need to figure out the padding value. They query the web service with dummy cipher texts, trying all possible values for the last byte in the IV. Whenever the resulting decrypted value is not equal to 0x01 (or by chance another valid padding arrangement), the decryption returns an error. But once padding is valid, the decryption will return success.
With this information, the attacker can determine the value of that byte in the decrypted block, even though they don’t have the key. For example, say the attacker sends the last IV byte as 0x2A. The decryption returns success, which means the decrypted byte XORed with 0x2A should equal 0x01. Now the attacker can calculate the decrypted value by XORing 0x2A with 0x01, yielding 0x2B; if the attacker XORs this value with the original IV byte (0x28), the result is 0x03, the original padding value, as expected.
The next step in the attack is to use the IV to generate a value of 0x02 in the lowest two bytes of the plaintext. In the same manner that the attacker used brute force on the lowest byte earlier, now they can brute force the second-to-lowest byte. Next, because the attacker knows the value of the lowest byte, it’s possible to set it to 0x02 with the appropriate IV value. Then, they can perform brute force on the second-to-lowest byte until the decryption is successful, which means the second byte now equals 0x02 when decrypted. By repeating this process until all bytes have been calculated, an attacker could use this technique to decrypt any block.
Stream Ciphers
Unlike block ciphers, which encrypt blocks of a message, stream ciphers work at the individual bit level. The most common algorithm used for stream ciphers generates a pseudorandom stream of bits, called the key stream, from an initial key. This key stream is then arithmetically applied to the message, typically using the XOR operation, to produce the cipher text, as shown in Figure 7-9.
Figure 7-9: A stream cipher operation
As long as the arithmetic operation is reversible, all it takes to decrypt the message is to generate the same key stream used for encryption and perform the reverse arithmetic operation on the cipher text. (In the case of XOR, the reverse operation is actually XOR.) The key stream can be generated using a completely custom algorithm, such as in RC4, or by using a block cipher and an accompanying mode of operation.
Table 7-3 lists some common algorithms that you might find in real-world applications.
Table 7-3: Common Stream Ciphers
Cipher name |
Key size (bits) |
Year introduced |
A5/1 and A5/2 (used in GSM voice encryption) |
54 or 64 |
1989 |
RC4 |
Up to 2048 |
1993 |
Counter mode (CTR) |
Dependent on block cipher |
N/A |
Output Feedback mode (OFB) |
Dependent on block cipher |
N/A |
Cipher Feedback mode (CFB) |
Dependent on block cipher |
N/A |
Asymmetric Key Cryptography
Symmetric key cryptography strikes a good balance between security and convenience, but it has a significant problem: participants in the network need to physically exchange secret keys. This is tough to do when the network spans multiple geographical regions. Fortunately, asymmetric key cryptography (commonly called public key encryption) can mitigate this issue.
An asymmetric algorithm requires two types of keys: public and private. The public key encrypts a message, and the private key decrypts it. Because the public key cannot decrypt a message, it can be given to anyone, even over a public network, without fear of its being captured by an attacker and used to decrypt traffic, as shown in Figure 7-10.
Figure 7-10: Asymmetric key encryption and decryption
Although the public and private keys are related mathematically, asymmetric key algorithms are designed to make retrieving a private key from a public key very time consuming; they’re built upon mathematical primitives known as trapdoor functions. (The name is derived from the concept that it’s easy to go through a trapdoor, but if it shuts behind you, it’s difficult to go back.) These algorithms rely on the assumption that there is no workaround for the time-intensive nature of the underlying mathematics. However, future advances in mathematics or computing power might disprove such assumptions.
RSA Algorithm
Surprisingly, not many unique asymmetric key algorithms are in common use, especially compared to symmetric ones. The RSA algorithm is currently the most widely used to secure network traffic and will be for the foreseeable future. Although newer algorithms are based on mathematical constructs called elliptic curves, they share many general principles with RSA.
The RSA algorithm, first published in 1977, is named after its original developers—Ron Rivest, Adi Shamir, and Leonard Adleman. Its security relies on the assumption that it’s difficult to factor large integers that are the product of two prime numbers.
Figure 7-11 shows the RSA encryption and decryption process. To generate a new key pair using RSA, you generate two large, random prime numbers, p and q, and then choose a public exponent (e). (It’s common to use the value 65537, because it has mathematical properties that help ensure the security of the algorithm.) You must also calculate two other numbers: the modulus (n), which is the product of p and q, and a private exponent (d), which is used for decryption. (The process to generate d is rather complicated and beyond the scope of this book.) The public exponent combined with the modulus constitutes the public key, and the private exponent and modulus form the private key.
For the private key to remain private, the private exponent must be kept secret. And because the private exponent is generated from the original primes, p and q, these two numbers must also be kept secret.
Figure 7-11: A simple example of RSA encryption and decryption
The first step in the encryption process is to convert the message to an integer, typically by assuming the bytes of the message actually represent a variable-length integer. This integer, m, is raised to the power of the public exponent. The modulo operation, using the value of the public modulus n, is then applied to the raised integer me. The resulting cipher text is now a value between zero and n. (So if you have a 1024-bit key, you can only ever encrypt a maximum of 1024 bits in a message.) To decrypt the message, you apply the same process, substituting the public exponent for the private one.
RSA is very computationally expensive to perform, especially relative to symmetric ciphers like AES. To mitigate this expense, very few applications use RSA directly to encrypt a message. Instead, they generate a random session key and use this key to encrypt the message with a symmetric cipher, such as AES. Then, when the application wants to send a message to another participant on the network, it encrypts only the session key using RSA and sends the RSA-encrypted key along with the AES-encrypted message. The recipient decrypts the message first by decrypting the session key, and then uses the session key to decrypt the actual message. Combining RSA with a symmetric cipher like AES provides the best of both worlds: fast encryption with public key security.
RSA Padding
One weakness of this basic RSA algorithm is that it is deterministic: if you encrypt the same message multiple times using the same public key, RSA will always produce the same encrypted result. This allows an attacker to mount what is known as a chosen plaintext attack in which the attacker has access to the public key and can therefore encrypt any message. In the most basic version of this attack, the attacker simply guesses the plaintext of an encrypted message. They continue encrypting their guesses using the public key, and if any of the encrypted guesses match the value of the original encrypted message, they know they’ve successfully guessed the target plaintext, meaning they’ve effectively decrypted the message without private key access.
To counter chosen plaintext attacks, RSA uses a form of padding during the encryption process that ensures the encrypted output is nondeterministic. (This “padding” is different from the block cipher padding discussed earlier. There, padding fills the plaintext to the next block boundary so the encryption algorithm has a full block to work with.) Two padding schemes are commonly used with RSA: one is specified in the Public Key Cryptography Standard #1.5; the other is called Optimal Asymmetric Encryption Padding (OAEP). OAEP is recommended for all new applications, but both schemes provide enough security for typical use cases. Be aware that not using padding with RSA is a serious security vulnerability.
Diffie–Hellman Key Exchange
RSA isn’t the only technique used to exchange keys between network participants. Several algorithms are dedicated to that purpose; foremost among them is the Diffie–Hellman Key Exchange (DH) algorithm.
The DH algorithm was developed by Whitfield Diffie and Martin Hellman in 1976 and, like RSA, is built upon the mathematical primitives of exponentiation and modular arithmetic. DH allows two participants in a network to exchange keys and prevents anyone monitoring the network from being able to determine what that key is. Figure 7-12 shows the operation of the algorithm.
Figure 7-12: The Diffie–Hellman Key Exchange algorithm
The participant initiating the exchange determines a parameter, which is a large prime number, and sends it to the other participant: the chosen value is not a secret and can be sent in the clear. Then each participant generates their own private key value—usually using a cryptographically secure random number generator—and computes a public key using this private key and a selected group parameter that is requested by the client. The public keys can safely be sent between the participants without the risk of revealing the private keys. Finally, each participant calculates a shared key by combining the other’s public key with their own private key. Both participants now have the shared key without ever having directly exchanged it.
DH isn’t perfect. For example, this basic version of the algorithm can’t handle an attacker performing a man-in-the-middle attack against the key-exchange. The attacker can impersonate the server on the network and exchange one key with the client. Next, the attacker exchanges a different key with the server, resulting in the attacker now having two separate keys for the connection. Then the attacker can decrypt data from the client and forward it on to the server, and vice versa.
Signature Algorithms
Encrypting a message prevents attackers from viewing the information being sent over the network, but it doesn’t identify who sent it. Just because someone has the encryption key doesn’t mean they are who they say they are. With asymmetric encryption, you don’t even need to manually exchange the key ahead of time, so anyone can encrypt data with your public key and send it to you.
Signature algorithms solve this problem by generating a unique signature for a message. The message recipient can use the same algorithm used to generate the signature to prove the message came from the signer. As an added advantage, adding a signature to a message protects it against tampering if it’s being transmitted over an untrusted network. This is important, because encrypting data does not provide any guarantee of data integrity; that is, an encrypted message can still be modified by an attacker with knowledge of the underlying network protocol.
All signature algorithms are built upon cryptographic hashing algorithms. First, I’ll describe hashing in more detail, and then I’ll explain some of the most common signature algorithms.
Cryptographic Hashing Algorithms
Cryptographic hashing algorithms are functions that are applied to a message to generate a fixed-length summary of that message, which is usually much shorter than the original message. These algorithms are also called message digest algorithms. The purpose of hashing in signature algorithms is to generate a relatively unique value to verify the integrity of a message and to reduce the amount of data that needs to be signed and verified.
For a hashing algorithm to be suitable for cryptographic purposes, it has to fulfill three requirements:
Pre-image resistance Given a hash value, it should be difficult (such as by requiring a massive amount of computing power) to recover a message.
Collision resistance It should be difficult to find two different messages that hash to the same value.
Nonlinearity It should be difficult to create a message that hashes to any given value.
A number of hashing algorithms are available, but the most common are members of either the Message Digest (MD) or Secure Hashing Algorithm (SHA) families. The Message Digest family includes the MD4 and MD5 algorithms, which were developed by Ron Rivest. The SHA family, which contains the SHA-1 and SHA-2 algorithms, among others, is published by NIST.
Other simple hashing algorithms, such as checksums and cyclic redundancy checks (CRC), are useful for detecting changes in a set of data; however, they are not very useful for secure protocols. An attacker can easily change the checksum, as the linear behavior of these algorithms makes it trivial to determine how the checksum changes, and this modification of the data is protected so the target has no knowledge of the change.
Asymmetric Signature Algorithms
Asymmetric signature algorithms use the properties of asymmetric cryptography to generate a message signature. Some algorithms, such as RSA, can be used to provide the signature and the encryption, whereas others, such as the Digital Signature Algorithm (DSA), are designed for signatures only. In both cases, the message to be signed is hashed, and a signature is generated from that hash.
Earlier you saw how RSA can be used for encryption, but how can it be used to sign a message? The RSA signature algorithm relies on the fact that it’s possible to encrypt a message using the private key and decrypt it with the public one. Although this “encryption” is no longer secure (the key to decrypt the message is now public), it can be used to sign a message.
For example, the signer hashes the message and applies the RSA decryption process to the hash using their private key; this encrypted hash is the signature. The recipient of the message can convert the signature using the signer’s public key to get the original hash value and compare it against their own hash of the message. If the two hashes match, the sender must have used the correct private key to encrypt the hash; if the recipient trusts that the only person with the private key is the signer, the signature is verified. Figure 7-13 shows this process.
Figure 7-13: RSA signature processing
Message Authentication Codes
Unlike RSA, which is an asymmetric algorithm, Message Authentication Codes (MACs) are symmetric signature algorithms. As with symmetric encryption, symmetric signature algorithms rely on sharing a key between the sender and recipient.
For example, say you want to send me a signed message and we both have access to a shared key. First, you’d combine the message with the key in some way. (I’ll discuss how to do this in more detail in a moment.) Then you’d hash the combination to produce a value that couldn’t easily be reproduced without the original message and the shared key. When you sent me the message, you’d also send this hash as the signature. I could verify that the signature is valid by performing the same algorithm as you did: I’d combine the key and message, hash the combination, and compare the resulting value against the signature you sent. If the two values were the same, I could be sure you’re the one who sent the message.
How would you combine the key and the message? You might be tempted to try something simple, such as just prefixing the message with the key and hashing to the combined result, as in Figure 7-14.
Figure 7-14: A simple MAC implementation
But with many common hashing algorithms (including MD5 and SHA-1), this would be a serious security mistake, because it opens a vulnerability known as the length-extension attack. To understand why, you need to know a bit about the construction of hashing algorithms.
Length-Extension and Collision Attacks
Many common hashing algorithms, including MD5 and SHA-1, consist of a block structure. When hashing a message, the algorithm must first split the message into equal-sized blocks to process. (MD5, for example, uses a block size of 64 bytes.)
As the hashing algorithm proceeds, the only state it maintains between each block is the hash value of the previous block. For the first block, the previous hash value is a set of well-chosen constants. The well-chosen constants are specified as part of the algorithm and are generally important for the secure operation. Figure 7-15 shows an example of how this works in MD5.
Figure 7-15: The block structure of MD5
It’s important to note that the final output from the block-hashing process depends only on the previous block hash and the current block of the message. No permutation is applied to the final hash value. Therefore, it’s possible to extend the hash value by starting the algorithm at the last hash instead of the predefined constants and then running through blocks of data you want to add to the final hash.
In the case of a MAC in which the key has been prefixed at the start of the message, this structure might allow an attacker to alter the message in some way, such as by appending extra data to the end of an uploaded file. If the attacker can append more blocks to the end of the message, they can calculate the corresponding value of the MAC without knowing the key because the key has already been hashed into the state of the algorithm by the time the attacker has control.
What if you move the key to the end of the message rather than attaching it to the front? Such an approach certainly prevents the length-extension attack, but there’s still a problem. Instead of an extension, the attacker needs to find a hash collision—that is, a message with the same hash value as the real message being sent. Because many hashing algorithms (including MD5) are not collision resistant, the MAC may be open to this kind of collision attack. (One hashing algorithm that’s not vulnerable to this attack is SHA-3.)
Hashed Message Authentication Codes
You can use a Hashed Message Authentication Code (HMAC) to counter the attacks described in the previous section. Instead of directly appending the key to the message and using the hashed output to produce a signature, an HMAC splits the process into two parts.
First, the key is XORed with a padding block equal to the block size of the hashing algorithm. This first padding block is filled with a repeating value, typically the byte 0x36. The combined result is the first key, sometimes called the inner padding block. This is prefixed to the message, and the hashing algorithm is applied. The second step takes the hash value from the first step, prefixes the hash with a new key (called the outer padding block, which typically uses the constant 0x5C), and applies the hash algorithm again. The result is the final HMAC value. Figure 7-16 diagrams this process.
Figure 7-16: HMAC construction
This construction is resistant to length-extension and collision attacks because the attacker can’t easily predict the final hash value without the key.
Public Key Infrastructure
How do you verify the identity of the owner of a public key in public key encryption? Simply because a key is published with an associated identity—say, Bob Smith from London—doesn’t mean it really comes from Bob Smith from London. For example, if I’ve managed to make you trust my public key as coming from Bob, anything you encrypt to him will be readable only by me, because I own the private key.
To mitigate this threat, you implement a Public Key Infrastructure (PKI), which refers to the combined set of protocols, encryption key formats, user roles, and policies used to manage asymmetric public key information across a network. One model of PKI, the web of trust (WOT), is used by such applications as Pretty Good Privacy (PGP). In the WOT model, the identity of a public key is attested to by someone you trust, perhaps someone you’ve met in person. Unfortunately, although the WOT works well for email, where you’re likely to know who you’re communicating with, it doesn’t work as well for automated network applications and business processes.
X.509 Certificates
When a WOT won’t do, it’s common to use a more centralized trust model, such as X.509 certificates, which generate a strict hierarchy of trust rather than rely on directly trusting peers. X.509 certificates are used to verify web servers, sign executable programs, or authenticate to a network service. Trust is provided through a hierarchy of certificates using asymmetric signature algorithms, such as RSA and DSA.
To complete this hierarchy, valid certificates must contain at least four pieces of information:
• The subject, which specifies the identity for the certificate
• The subject’s public key
• The issuer, which identifies the signing certificate
• A valid signature applied over the certificate and authenticated by the issuer’s private key
These requirements create a hierarchy called a chain of trust between certificates, as shown in Figure 7-17. One advantage to this model is that because only public key information is ever distributed, it’s possible to provide component certificates to users via public networks.
Figure 7-17: The X.509 certificate chain of trust
Note that there is usually more than one level in the hierarchy, because it would be unusual for the root certificate issuer to directly sign certificates used by an application. The root certificate is issued by an entity called a certificate authority (CA), which might be a public organization or company (such as Verisign) or a private entity that issues certificates for use on internal networks. The CA’s job is to verify the identity of anyone it issues certificates to.
Unfortunately, the amount of actual checking that occurs is not always clear; often, CAs are more interested in selling signed certificates than in doing their jobs, and some CAs do little more than check whether they’re issuing a certificate to a registered business address. Most diligent CAs should at least refuse to generate certificates for known companies, such as Microsoft or Google, when the certificate request doesn’t come from the company in question. By definition, the root certificate can’t be signed by another certificate. Instead, the root certificate is a self-signed certificate where the private key associated with the certificate’s public key is used to sign itself.
Verifying a Certificate Chain
To verify a certificate, you follow the issuance chain back to the root certificate, ensuring at each step that every certificate has a valid signature that hasn’t expired. At this point, you decide whether you trust the root certificate—and, by extension, the identity of the certificate at the end of the chain. Most applications that handle certificates, like web browsers and operating systems, have a trusted root certificate database.
What’s to stop someone who gets a web server certificate from signing their own fraudulent certificate using the web server’s private key? In practice, they can do just that. From a cryptography perspective, one private key is the same as any other. If you based the trust of a certificate on the chain of keys, the fraudulent certificate would chain back to a trusted root and appear to be valid.
To protect against this attack, the X.509 specification defines the basic constraints parameter, which can be optionally added to a certificate. This parameter is a flag that indicates the certificate can be used to sign another certificate and thus act as a CA. If a certificate’s CA flag is set to false (or if the basic constraints parameter is missing), the verification of the chain should fail if that certificate is ever used to sign another certificate. Figure 7-18 shows this basic constraint parameter in a real certificate that says this certificate should be valid to act as a certificate authority.
But what if a certificate issued for verifying a web server is used instead to sign application code? In this situation, the X.509 certificate can specify a key usage parameter, which indicates what uses the certificate was generated for. If the certificate is ever used for something it was not designed to certify, the verification chain should fail.
Finally, what happens if the private key associated with a given certificate is stolen or a CA accidentally issues a fraudulent certificate (as has happened a few times)? Even though each certificate has an expiration date, this date might be many years in the future. Therefore, if a certificate needs to be revoked, the CA can publish a certificate revocation list (CRL). If any certificate in the chain is on the revocation list, the verification process should fail.
Figure 7-18: X.509 certificate basic constraints
As you can see, the certificate chain verification could potentially fail in a number of places.
Case Study: Transport Layer Security
Let’s apply some of the theory behind protocol security and cryptography to a real-world protocol. Transport Layer Security (TLS), formerly called Secure Sockets Layer (SSL), is the most common security protocol in use on the internet. TLS was originally developed as SSL by Netscape in the mid-1990s for securing HTTP connections. The protocol has gone through multiple revisions: SSL versions 1.0 through 3.0 and TLS versions 1.0 through 1.2. Although it was originally designed for HTTP, you can use TLS for any TCP protocol. There’s even a variant, the Datagram Transport Layer Security (DTLS) protocol, to use with unreliable protocols, such as UDP.
TLS uses many of the constructs described in this chapter, including symmetric and asymmetric encryption, MACs, secure key exchange, and PKI. I’ll discuss the role each of these cryptographic tools plays in the security of a TLS connection and touch on some attacks against the protocol. (I’ll only discuss TLS version 1.0, because it’s the most commonly supported version, but be aware that versions 1.1 and 1.2 are slowly becoming more common due to a number of security issues with version 1.0.)
The TLS Handshake
The most important part of establishing a new TLS connection is the handshake, where the client and server negotiate the type of encryption they’ll use, exchange a unique key for the connection, and verify each other’s identity. All communication uses a TLS Record protocol—a predefined tag-length-value structure that allows the protocol parser to extract individual records from the stream of bytes. All handshake packets are assigned a tag value of 22 to distinguish them from other packets. Figure 7-19 shows the flow of these handshake packets in a simplified form. (Some packets are optional, as indicated in the figure.)
As you can see from all the data being sent back and forth, the handshake process can be time-intensive: sometimes it can be truncated or bypassed entirely by caching a previously negotiated session key or by the client’s asking the server to resume a previous session by providing a unique session identifier. This isn’t a security issue because, although a malicious client could request the resumption of a session, the client still won’t know the private negotiated session key.
Figure 7-19: The TLS handshake process
Initial Negotiation
As the first step in the handshake, the client and server negotiate the security parameters they want to use for the TLS connection using a HELLO message. One of the pieces of information in a HELLO message is the client random, a random value that ensures the connection process cannot be easily replayed. The HELLO message also indicates what types of ciphers the client supports. Although TLS is designed to be flexible with regard to what encryption algorithms it uses, it only supports symmetric ciphers, such as RC4 or AES, because using public key encryption would be too expensive from a computational perspective.
The server responds with its own HELLO message that indicates what cipher it has chosen from the available list provided by the client. (The connection ends if the pair cannot negotiate a common cipher.) The server HELLO message also contains the server random, another random value that adds additional replay protection to the connection. Next, the server sends its X.509 certificate, as well as any necessary intermediate CA certificates, so the client can make an informed decision about the identity of the server. Then the server sends a HELLO Done packet to inform the client it can proceed to authenticate the connection.
Endpoint Authentication
The client must verify that the server certificates are legitimate and that they meet the client’s own security requirements. First, the client must verify the identity in the certificate by matching the certificate’s Subject field to the server’s domain name. For example, Figure 7-20 shows a certificate for the domain www.domain.com. The Subject contains a Common Name (CN) ➊ field that matches this domain.
Figure 7-20: The Certificate Subject for www.domain.com
A certificate’s Subject and Issuer fields are not simple strings but X.500 names, which contain other fields, such as the Organization (typically the name of the company that owns the certificate) and Email (an arbitrary email address). However, only the CN is ever checked during the handshake to verify an identity, so don’t be confused by the extra data. It’s also possible to have wildcards in the CN field, which is useful for sharing certificates with multiple servers running on a subdomain name. For example, a CN set to *.domain.com would match both www.domain.com and blog.domain.com.
After the client has checked the identity of the endpoint (that is, the server at the other end of the connection), it must ensure that the certificate is trusted. It does so by building the chain of trust for the certificate and any intermediate CA certificates, checking to make sure none of the certificates appear on any certificate revocation lists. If the root of the chain is not trusted by the client, it can assume the certificate is suspect and drop the connection to the server. Figure 7-21 shows a simple chain with an intermediate CA for www.domain.com.
Figure 7-21: The chain of trust for www.domain.com
TLS also supports an optional client certificate that allows the server to authenticate the client. If the server requests a client certificate, it sends a list of acceptable root certificates to the client during its HELLO phase. The client can then search its available certificates and choose the most appropriate one to send back to the server. It sends the certificate—along with a verification message containing a hash of all the handshake messages sent and received up to this point—signed with the certificate’s private key. The server can verify that the signature matches the key in the certificate and grant the client access; however, if the match fails, the server can close the connection. The signature proves to the server that the client possesses the private key associated with the certificate.
Establishing Encryption
When the endpoint has been authenticated, the client and server can finally establish an encrypted connection. To do so, the client sends a randomly generated pre-master secret to the server encrypted with the server’s certificate public key. Next, both client and server combine the pre-master secret with the client and server randoms, and they use this combined value to seed a random number generator that generates a 48-byte master secret, which will be the session key for the encrypted connection. (The fact that both the server and the client generate the master key provides replay protection for the connection, because if either endpoint sends a different random during negotiation, the endpoints will generate different master secrets.)
When both endpoints have the master secret, or session key, an encrypted connection is possible. The client issues a change cipher spec packet to tell the server it will only send encrypted messages from here on. However, the client needs to send one final message to the server before normal traffic can be transmitted: the finished packet. This packet is encrypted with the session key and contains a hash of all the handshake messages sent and received during the handshake process. This is a crucial step in protecting against a downgrade attack, in which an attacker modifies the handshake process to try to reduce the security of the connection by selecting weak encryption algorithms. Once the server receives the finished message, it can validate that the negotiated session key is correct (otherwise, the packet wouldn’t decrypt) and check that the hash is correct. If not, it can close the connection. But if all is correct, the server will send its own change cipher spec message to the client, and encrypted communications can begin.
Each encrypted packet is also verified using an HMAC, which provides data authentication and ensures data integrity. This verification is particularly important if a stream cipher, such as RC4, has been negotiated; otherwise, the encrypted blocks could be trivially modified.
Meeting Security Requirements
The TLS protocol successfully meets the four security requirements listed at the beginning of this chapter and summarized in Table 7-4.
Table 7-4: How TLS Meets Security Requirements
Security requirement |
How it’s met |
Data confidentiality |
Selectable strong cipher suites |
Data integrity |
Encrypted data is protected by an HMAC |
Server authentication |
The client can choose to verify the server endpoint using the PKI and the issued certificate |
Client authentication |
Optional certificate-based client authentication |
But there are problems with TLS. The most significant one, which as of this writing has not been corrected in the latest versions of the protocol, is its reliance on certificate-based PKI. The protocol depends entirely on trust that certificates are issued to the correct people and organizations. If the certificate for a network connection indicates the application is communicating to a Google server, you assume that only Google would be able to purchase the required certificate. Unfortunately, this isn’t always the case. Situations in which corporations and governments have subverted the CA process to generate certificates have been documented. In addition, mistakes have been made when CAs didn’t perform their due diligence and issued bad certificates, such as the Google certificate shown in Figure 7-22 that eventually had to be revoked.
Figure 7-22: A certificate for Google “wrongly” issued by CA TÜRKTRUST
One partial fix to the certificate model is a process called certificate pinning. Pinning means that an application restricts acceptable certificates and CA issuers for certain domains. As a result, if someone manages to fraudulently obtain a valid certificate for www.google.com, the application will notice that the certificate doesn’t meet the CA restrictions and will fail the connection.
Of course, certificate pinning has its downsides and so is not applicable to every scenario. The most prevalent issue is the management of the pinning list; specifically, building an initial list might not be too challenging a task, but updating the list adds additional burdens. Another issue is that a developer cannot easily migrate the certificates to another CA or easily change certificates without also having to issue updates to all clients.
Another problem with TLS, at least when it comes to network surveillance, is that a TLS connection can be captured from the network and stored by an attacker until it’s needed. If that attacker ever obtains the server’s private key, all historical traffic could be decrypted. For this reason, a number of network applications are moving toward exchanging keys using the DH algorithm in addition to using certificates for identity verification. This allows for perfect forward secrecy—even if the private key is compromised, it shouldn’t be easy to also calculate the DH-generated key.
Final Words
This chapter focused on the basics of protocol security. Protocol security has many aspects and is a very complex topic. Therefore, it’s important to understand what could go wrong and identify the problem during any protocol analysis.
Encryption and signatures make it difficult for an attacker to capture sensitive information being transmitted over a network. The process of encryption converts plaintext (the data you want to hide) into cipher text (the encrypted data). Signatures are used to verify that the data being transmitted across a network hasn’t been compromised. An appropriate signature can also be used to verify the identity of the sender. The ability to verify the sender is very useful for authenticating users and computers over an untrusted network.
Also described in this chapter are some possible attacks against cryptography as used in protocol security, including the well-known padding oracle attack, which could allow an attack to decrypt traffic being sent to and from a server. In later chapters, I’ll explain in more detail how to analyze a protocol for its security configuration, including the encryption algorithms used to protect sensitive data.