At some point, most of us have accessed a web page to download information. But just what's involved when retrieving a web page? In this chapter, we'll take a closer look at the HyperText Transfer Protocol (HTTP). We'll start with an overview and review some of the objects and elements that we can obtain when requesting content. We'll then compare the available HTTP versions, 1.0, 1.1, and 2.0, along with the methods used, such as GET, POST, and HEAD. HTTP has three versions, each with a default method to establish and maintain a connection. So that you understand the mechanics of the different methods, we'll compare the differences between a non-persistent and persistent connection.
HTTP is a stateless protocol. You'll learn how HTTP uses cookies to maintain state by keeping track of the details of each transaction. To help you troubleshoot a web connection, we'll review what takes place during a transaction and examine the general format of request and response messages. We'll dissect the HTTP header and fields for both the client and the server. We'll then summarize this by following an HTTP stream, and then break down each of the elements when examining a transaction.
This chapter will address all of these by covering the following topics:
HTTP is an application-layer protocol responsible for exchanging data between a client browser and a web server. HTTP allows us to gather, distribute, collaborate, and disseminate a wide range of data. Although a stateless protocol, HTTP has several request methods, error codes, and headers that allow us to access resources from many different types of applications.
Note
HTTP is used to transport data across the network. Two other protocols that transport data include File Transfer Protocol (FTP) and Simple Mail Transfer Protocol (SMTP). All are essential in providing an efficient way to exchange data.
In this section, we'll begin by outlining the definition of a web page and describe HTTP's function in retrieving objects from a website. We'll compare the role of client and server, outline the versions in use today, and then summarize with a review of the different types of HTTP methods.
Let's start by reviewing the elements of a web page.
Using a browser, such as Firefox or Chrome, a client requests a page by clicking on a hyperlink. The link directs the client to a website, which is a set of linked web pages, and the server sends the requested data to the client.
The first page we hit is called index or default, and that's where most of us start when searching for content. Once at the website, each page contains objects, such as a HyperText Markup Language (HTML) file, Joint Photographic Experts Group (JPEG) images, text, applications, and/or JavaScript. Most websites have many linked pages that are hyperlinked to other content within the site, along with a variety of information and objects that can be extracted.
At a high level, HTTP will retrieve a web page that can contain elements, as shown in the following graphic:
Prior to downloading any objects, the client must first access the web page. Next, let's review how resources are located when requesting data from a website.
When a client requests a web page, the user will click on a link to request an object using a Uniform Resource Identifier (URI), which is used to identify an object (or resource) on the internet. Two subsets of a URI are a Uniform Resource Locator (URL) and a Uniform Resource Name (URN).
In some cases, the client will request a specific page from the server. For example, if requesting https://www.nist.gov/blogs/cybersecurity-insights, the URL will break down in the following manner:
Once the request reaches the target, the next step is to interact with the server using either a standard or encrypted connection.
When a client initiates a connection with a web server, they will use either Transmission Control Protocol (TCP) port 80 or port 443, depending on the type of connection, as described here:
Note
In some cases, the server may be reached using an alternate port, such as http-alt 8080 or http-alt 8008.
Using a secure connection is common today, as most websites use encryption to protect data transactions. When accessing a secure site, the URL will be identified using the HTTP Secure (HTTPS) preface. HTTPS uses Transport Layer Security (TLS) to secure all transactions between the client and the server. Even if someone were able to obtain the data stream, they would not be able to read the contents without the appropriate key to decrypt the data.
Once the target is located, the client will make a request to the server. Each party has a specific role in a web transaction, as discussed next.
HTTP is a client-server model, whereby the client makes a request to the server, and the server responds to the client, as shown in the following illustration:
A client is a host that initiates each session using a browser to interface with a server and retrieve objects.
A server is an always-on host with a fixed Internet Protocol (IP) address that uses dedicated web server software. Although there are several web server applications, the top three are listed as follows:
Whenever requesting a web page, the version will be either HTTP version 1.0, 1.1, or 2.0, as discussed next.
HTTP has three versions in use today. Each has an associated Request for Comments (RFC), as follows:
All three versions are in use; however, you will most likely see versions 1.1 and 2.0 for most of today's web traffic, as these versions offer improved performance.
When doing an analysis, you will see a method listed in the HTTP header that will indicate the type of action that is requested.
HTTP is primarily used to obtain objects and interact with a web server using various methods to exchange information. Some of the most common methods are described in the following table:
While there are several HTTP request methods, the two most commonly used are GET and POST.
In the next section, we'll compare connection methods and cover how cookies are used to keep track of transactions.
Every version of HTTP has evolved in methods to transport and process data, with techniques such as using persistent connections with a pipelining goal to optimize the connection. In addition, because HTTP is a stateless protocol, cookies are used to maintain client information about the connection, such as shopping cart elements and pages the client has visited.
In this segment, we'll compare the different methods of making and maintaining a connection, along with how cookies are used to preserve state information. Let's start by comparing connection types.
When HTTP version 1.0 was developed, it had minimal functionality and used a non-persistent connection. With the introduction of version 1.1, several enhancements were added. One of the improvements was the ability to keep the data moving by using a persistent connection.
Let's start by understanding the mechanics of a non-persistent connection.
A non-persistent connection is one of the original ways to retrieve content from a web page. This method works in the following manner:
If the client wants to obtain any more content from the web server, a new connection must be made, resulting in one round trip per request and response. As you can imagine, all of the round trips and connections can add to latency.
There are two approaches in a non-persistent connection, as outlined here:
If possible, a parallel connection is used, as this will improve the overall efficiency of obtaining objects.
When using HTTP version 1.0, the default is to use a non-persistent connection. HTTP version 1.1 improved the method to connect by offering a persistent connection, as outlined next.
To optimize a connection, HTTP version 1.1 introduced the idea of using a persistent connection to keep a session alive until all objects are retrieved. This type of connection is also referred to as an HTTP keep-alive, as it prevents having to reestablish a new connection every time a request is made.
Note
Although the idea of a keep-alive was made official in version 1.1, version 1.0 adopted an unofficial method to keep the session alive as well. That is why you might see HTTP 1.0 traffic using a keep-alive during a transaction.
When used, the HTTP header can set a timeout and define the maximum number of requests the client can make.
Keep-alive packets are sent between the client and the server to keep the session active and to verify that both sides are still responding. A keep-alive packet doesn't have any data; it has the acknowledgment (ACK) flag set and the sequence number is set to one less than the current sequence number.
HTTP version 2.0 improves this option and permits multiple concurrent transactions to be combined in a single connection.
HTTP 1.1 made persistent connections the default mode. Next, let's see how a persistent connection can benefit from using a concept called pipelining.
To optimize a data transfer using a persistent connection, HTTP can employ pipelining, which maintains the connection to the server until all objects are obtained. Pipelining improves the efficiency of a data transaction, as the client can continue to request objects immediately without waiting for the server to send previous requests.
Instead of closing each connection after a request-response transaction, the server will keep the connection open and wait for further requests. A persistent connection using pipelining is the default connection method in HTTP 1.1. This type of connection requires a bit more overhead in terms of setting up the connection. In addition, because the server is waiting for multiple requests, it might be idle for short segments of time. However, this type of connection is preferred as it improves overall efficiency when obtaining objects from a web server.
Because HTTP is a stateless protocol, cookies are used to keep track of the details of each transaction.
When using HTTP, neither the server nor the client natively maintains information on the state of the transaction. To overcome this, HTTP uses the concept of a cookie.
A cookie is a string of characters created by the server that can be placed on a user's system and then managed by the user's browser to interact with the server. Once the user accepts a cookie, the information contained in the cookie can be used for the following reasons:
For a closer look at a cookie being used in an HTTP conversation, go to http://tcpreplay.appneta.com/wiki/captures.html#bigflows-pcap. Once there, download bigFlows.pcap so that you can follow along.
Go to Frame 912 and then expand the HTTP header. Once there, you will see the cookie and cookie pair, as shown in the following screenshot:
In most cases, cookies are designed to be helpful; however, they can pose a privacy risk as they can gather marketing statistics and personal information. That is why, in most cases, a user will have the ability to opt out of allowing cookies on their system.
In some cases, a cookie is sent to the server to be maintained and retrieved for future visits by the client. However, if a cookie is not sent to the server, it is said to be non-persistent, and one of the following scenarios will take place:
During an HTTP transaction, there are a series of requests and responses between the client and the server. In the next section, we'll examine the general format of request and response messages.
When interacting with a web page, a client will request objects from a server. In most cases, a standard transaction involves a request from the client and then a response from the server. Because the client and the server both convey a different message, the headers are slightly different. In this segment, we'll take a high-level view of the HTTP header and fields for both the client and the server.
Let's start by examining the elements of an HTTP request.
When viewing a client request, you will most likely see a request line followed by header lines, as shown in the following diagram:
Each request message will indicate the method (such as GET or POST) and the URL, along with additional parameters. It's also common to see the following:
Both symbols in the header provide formatting guidance that represents the end of the content for that particular line. To see an example, return to bigFlows.pcap and select Frame 183. Then, expand the HTTP header, as shown here:
An HTTP request is used to retrieve objects. Most web servers provide rich, interactive content. As a result, the headers must indicate the types of content that the client can support. Within an HTTP request, you will see a reference to the Multipurpose Internet Mail Extensions (MIME) header. Let's explore this next.
MIME is designed to present data in a variety of formats that include text, images, audio, applications, and video. Originally developed to support email, MIME headers became a part of the evolution of HTTP.
When the server receives the HTTP request, the request line will list what the client wants, along with the HTTP version that the browser supports. After that, the MIME header contains fields that relate to the type of content that the client can support. An example is shown here:
The MIME header holds values such as the browser version, along with character encoding and objects that the client will accept.
Once the server receives the request, it will send the response to the client. The header format is much the same as the HTTP request; however, this contains details that are specific to the server response.
The HTTP response from the server begins with the status line, indicating the version, status code, and reason. After the status line, the structure is similar to the HTTP request, as we find header lines, as shown in the following diagram:
Following the status code and header lines, you will see the message body, which can contain client-requested items, along with details of the transaction.
One of the main elements of an HTTP response is the status of the message. HTTP indicates the status by using a specific code, as outlined next.
A status code is shown in the first line of a web server to the client response message. Status codes are grouped by category, as follows:
While many status codes are available, some of the most common codes include the following:
After the status line, you will see various elements within the MIME header related to the response, as outlined next.
Whenever you retrieve data from a web page, your browser will properly format the request. In turn, the server will respond to the request and deliver the data. The browser will then present the results to the client.
When a server returns objects to a client, there are various elements within the MIME header, as shown here:
The field values define details such as the type of content, where it originated, and length.
In the next section, let's explore the process from making a request, to displaying an object, to closing a connection.
To get a solid understanding of what happens when requesting and receiving a web page, we'll step through the process by following an HTTP stream. We'll then further break down each of the elements when examining an HTTP transaction.
Note
Keep in mind that each HTTP session is different. This example will provide a sampling of what you can expect when viewing an HTTP conversation. In a true analysis exercise, you will most likely have to research the meaning of some of the various field values.
For this example, we'll use HTTP.pcap, as it is a complete conversation. To obtain a copy, go to https://www.cloudshark.org/captures/0012f52602a3, then download the file and open it in Wireshark. Once open, expand Frame 1 under the TCP header, where you will see the following:
[Conversation completeness: Complete, WITH_DATA (31)]
This is a small capture with only 40 packets, so it isn't difficult to see all elements of the complete conversation. However, one way to view the activity between hosts is by running a flow graph. To view the entire conversation, go to the Statistics menu choice and then select Flow Graph. Wireshark will then run the graph, as shown in the following screenshot:
Once in the Flow Graph setting, you can select Limit to display filter, as shown on the left-hand side of the graph. However, because this capture only contains one conversation, this isn't necessary.
After viewing the flow graph, exit the window so that we can focus on the capture. To see the dialog between the client and the server, place your cursor on Frame 4, right-click, and select Follow | TCP Stream, as shown in the following screenshot:
Once selected, Wireshark will present a window with the following conversation:
Within the window, you will see the following:
Now that we have seen the entire conversation, let's go through the process. The first step is for the client to establish contact with the server by using a three-way handshake.
Before any data is exchanged, the process begins with a three-way handshake, which we see in Frames 1, 2, and 3 of HTTP.pcap, as shown in the following screenshot:
A three-way handshake is an exchange of synchronization (SYN) and ACK packets between the client (192.168.1.140) and the server (174-143.213.184), as described in more detail here:
No data is exchanged as the handshake simply establishes the connection. Once established, the next step is for the client to make a request to the server.
Frame 4 begins the conversation as the client makes a request to the server for an image. The HTTP header is shown in the following screenshot:
When examining the client request, we see several values in the HTTP header. The first line is the GET method requesting data from the server. Here are the parameters of this request:
Following the GET method, you will see these values:
Following the field values, we see three Wireshark-generated details, as outlined next:
After the client request is sent to the server, data is then sent to the client. From Frame 5 to Frame 35, you will see a series of ACK packets, from both the client and the server as they acknowledge the data is being transferred and received.
Once complete, the server will respond to the client and indicate that all data has been sent.
In Frame 36, we see the server returning a Status Code: 200 OK response, meaning the data has successfully been delivered.
Unlike the request packet, we see more information contained in the HTTP response, as shown in the following screenshot:
When examining the server response, we see several values in the HTTP header. The first line of the data transfer is a Wireshark-generated value that indicates the status of the response, as shown here:
[Expert Info (Chat/Sequence): HTTP/1.1 200 OK ]
This line defines that HTTP/1.1 200 (PNG) falls under the Chat severity in the Expert Information console. To view this, select the cyan Expert Info icon in the lower left-hand corner of the Wireshark interface. Once open, expand the second Chat entry, which will display the following information:
After the Wireshark-generated values, you will see the following:
After the details of the transaction, you will see the following values:
On a network, the concept of cache is important as, in most cases, the requesting host wants to receive the freshest copy possible. Cache values are used to identify the age of the page.
Most objects on the network typically have a timeout value. The next few values provide a method to monitor the age of a page. Entries include the following:
Note
On assets that rarely change, it's customary to set the Cache-Control value to max-age=31536000, which represents the number of seconds in 1 year.
At the end of the HTTP field values, we see the Portable Graphics Format header, which indicates that the object is a PNG file. Next, let's investigate the image.
If there are unencrypted objects in the capture, you can export them in Wireshark. To see what's in the HTTP.pcap file, go to the File menu choice and then Export Objects | HTTP…, which will bring up the following window:
If you would like to view the contents, click on the image and select Save, which will bring up a dialog box giving you a choice as to where you would like to save the file. Once saved, you can open the image, as shown here:
Once the data is transferred and the client is no longer requesting objects, the next step is to end the conversation.
At the end of the transaction, the client and server exchange a series of finish (FIN) and ACK packets, as shown here:
While this was a simple example of a data transaction, it provides a solid way to understand the mechanics of a client interacting with a web server.
HTTP is a rich protocol, and during a transaction, data will travel across several networks, along with encountering different clients and servers. As a result, there are numerous rules and variables. For this discussion, we focused on the key elements of an HTTP request-response session between a client and server.
We began by stepping through the key elements of a web page, and the role of the client and the server when retrieving data and objects. We reviewed the different HTTP versions, along with briefly touching on the available HTTP methods. We then moved on to learn about the different types of connections and how cookies help maintain state during a transaction. Finally, we summarized with a simple example of a complete HTTP conversation.
In the next chapter, we'll review Address Resolution Protocol (ARP) and begin with an overview of the role and purpose of this essential protocol. So that you understand how ARP works, we will cover an ARP transaction and take a closer look at the header and field values. We will see the importance of a gratuitous ARP and briefly mention some other types of ARP traffic that you might encounter. Finally, we will look at ARP attacks and how to identify and defend against these types of threats.
Now, it's time to check your knowledge. Select the best response to the following questions and then check your answers, which can be found in the Assessment appendix:
Please refer to the following links for more information: