The traditional focus of digital forensics has been on locating evidence on a potentially compromised endpoint. More specifically, computer forensics is largely focused on a system’s storage. Law enforcement officers interested in criminal activity such as fraud or child exploitation can find the evidence required for prosecution on a single hard drive. In the realm of incident response, however, it is critical that the focus extends far beyond a suspected compromised system. For example, there is a wealth of information that can be obtained within the hardware and software in question, along with the flow of traffic from a compromised host to an external Command-and-Control (C2) server.
This chapter focuses on the preparation, identification, and collection of evidence that is commonly found among network devices and along traffic routes within an internal network. This collection is critical during incidents where an external threat source is in the process of commanding internal systems or stealing data from the network. Network-based evidence is also useful when examining host evidence, as it provides a second source of event corroboration, which is extremely useful in determining the root cause of an incident.
We will cover the following topics in this chapter:
There are network log sources that can provide CSIRT personnel and incident responders with good information. Each network device provides different evidence based on its manufacturer and model. As a preparation task, CSIRT personnel should become familiar with how to access these devices to obtain the necessary evidence or have existing communication structures in place to engage IT personnel to assist with the proper response techniques during an incident.
Network devices such as switches, routers, and firewalls also have their own internal logs that maintain data on who accessed the device and made changes. Incident responders should become familiar with the types of network devices on their organization’s network and be able to access these logs in the event of an incident:
As covered in the previous section, there is a wide range of network devices, each with its own method of aggregating and reporting relevant data. The ability to acquire network-based evidence is largely dependent on the preparations that are undertaken by an organization prior to an incident. Without some of the critical components of a proper infrastructure security program, key pieces of evidence will not be readily available to incident responders. The result is that evidence may be lost while CSIRT members hunt down critical pieces of information. In terms of preparation, organizations can aid the CSIRT by having proper network documentation, up-to-date configurations of network devices, and a central log management solution, such as a SIEM, in place.
Another consideration in terms of preparation is to incorporate the specific tasks that need to be performed for proper network evidence collection into the incident response playbooks. For example, firewall logs that are not sent to a central log management solution, such as a SIEM, are stored on the device itself. Playbooks that involve acquiring network evidence should be verbose enough to walk through the process of gathering these logs.
Aside from the technical preparation for network evidence collection, CSIRT personnel needs to be aware of any legal or regulatory issues with regard to collecting network evidence. Additionally, CSIRT personnel needs to be aware that capturing network traffic can be considered an invasion of privacy if there is no policy clearly stating that network monitoring may take place. Therefore, the legal representative of the CSIRT should ensure that all employees of the organization understand that their use of the information system will be monitored. This should be expressly stated in policy prior to any evidence collection taking place.
To identify potential sources of evidence, incident responders need to have a solid understanding of what the internal network infrastructure looks like. One method that can be employed by organizations is to create and maintain an up-to-date network diagram. This diagram should be detailed enough that incident responders can identify individual network components such as switches, routers, or wireless access points. This diagram should also contain internal IP addresses so that incident responders can immediately access those systems through remote methods. For instance, examine the following simple network diagram:
Figure 5.1 – A sample network diagram
This diagram allows for the quick identification of potential evidence sources. For example, suppose that the laptop connected to the switch at 192.168.2.1 is identified as communicating with a known malware C2 server. A CSIRT analyst could examine the network diagram and ascertain that the C2 traffic would have to traverse several network hardware components on its way out of the internal network. For example, there would be traffic traversing the switch at 192.168.10.1, through the firewall at 192.168.0.1, and, finally, from the router out to the internet.
Determining whether an attacker has made modifications to a network device such as a switch or a router can be made easier if the CSIRT has a standard configuration immediately available. Organizations should already have configurations for network devices stored for disaster recovery purposes but they should also have these available to CSIRT members in the event that there is an incident.
Two main sources of evidence available while investigating an incident are ingress/egress points into the network from the internet. Modern malware and other exploits will often require the ability to reach internet-based resources. This may be for the purpose of downloading additional malware or to exploit code. Other attacks that involve data exfiltration will require access to the internet. Finally, adversaries will often have to establish C2 over compromised systems. In these cases, traffic from various protocols will traverse the perimeter of the victim network. Depending on the victim, this traffic will have to traverse a firewall, internet proxy, or both. As a result, both technologies provide incident response personnel with a major source of evidence.
Firewalls have evolved from a simplified routing and blocking technology into platforms that provide a significant insight into the traffic coming into and leaving the network. Next-generation firewalls often combine the deny/allow ruleset with IDSs or IPSs, as well as controlling network access to applications. This creates a significant source of evidence that can be leveraged during an incident.
Acquiring evidence from firewalls is largely dependent on the manufacturer and the specific model that is used. Incident responders should thoroughly understand the feature set and specific data that can be obtained as part of their preparation. Although features differ between vendors and models, there are some key evidence points that are near-universal:
A special type of firewall is the Web Application Firewall (WAF). This device or software sits between the public internet and a web application. This allows an organization to protect the application from the public. WAF policies can be changed very quickly to respond to attacks such as Denial-of-Service (DoS) or adversaries attempting to compromise the web application infrastructure. From an evidentiary standpoint, WAFs provide analysts with log files of connections made from the internet, along with other data points such as the type of HTTP requests made and information on the systems that access resources. This is a good source of evidence of attempted or successful exploitation of web servers and web applications.
Adversaries often make use of scripting such as Microsoft Visual Basic or PowerShell to download secondary exploit packages or malware. These scripts will often contain a URL that points to the exploit or malware. Adversaries make use of URLs as opposed to IP addresses, as the IP addresses can be easily changed via domain name registration, allowing them to change their infrastructure without having to change their scripts.
Organizations that make use of web proxy servers for HTTP and HTTPS requests will have a record of any system on the internal network that reached out to an external site. From here, they may be able to identify the location and, potentially, the malware or exploit that has been downloaded. Additional insight may be gained from C2 traffic that makes use of similar tactics to malware.
As detecting attacks often takes months, it is imperative that incident responders can view the history of an activity that has happened over the span of weeks or even months. Given the relatively small size of proxy requests, even just the date, time, requesting system, and the URL that was visited can provide a significant piece of evidence that might not be available otherwise.
First designed by Cisco Systems in 1996, NetFlow is a feature found in network devices such as switches and routers that allows network administrators to monitor traffic within the network. NetFlow is not strictly a security tool but it does provide a good deal of data to incident responders in the event of an incident. NetFlow is sent by network devices via the UDP protocol to a central collection point, often called the NetFlow Collector.
In a security context, NetFlow provides deep insights into the internal traffic of systems as they communicate with each other. This is often referred to as east-west traffic, as opposed to north-south traffic, which is used to describe internal systems communicating with external systems through the perimeter firewall. For example, the following diagram shows a simple network. In a real-world scenario, an attacker may compromise a system on the 10.10.2.0/24 subnet. From there, they may attempt to pivot to a file server on the 10.10.1.0/24 subnet. Once there, they can acquire confidential data and move it back to the compromised system for exfiltration. The switches forward the NetFlow data to the collector, which includes the IP addresses, protocols, and data size. This data is critical for providing incident response analysts with details that they may not normally otherwise acquire:
Figure 5.2 – A NetFlow diagram
NetFlow data is limited and depends on the types of devices that are configured to send data to the NetFlow server. Generally, NetFlow contains the source and destination IP addresses, the source and destination ports, the protocol, and finally, the amount of traffic sent. Even with this limited information, analysts gain insight into adversary activities, such as potential lateral movement or data exfiltration.
Configuring NetFlow is dependent on the type and manufacturer of the network components. Moreover, there is a wide range of collectors and analysis tools that can be leveraged depending on the budget and other resources. One of the advantages of including NetFlow analysis in the overall network operations is that it not only provides data to the incident response team, but it is also highly useful in day-to-day network operations in terms of hunting down latency or other communication issues. This dual purpose makes including it as part of the overall network operations easier to justify.
Capturing network traffic is critical to having a full understanding of an incident. Being able to identify potential C2 IP address traffic may provide further information about the type of malware that has infected a host. In other types of incidents, CSIRT members may be able to identify potential exfiltration methods that an external threat actor is utilizing.
One method is to set up what is referred to as a network tap. A network tap is a system that is in line with the compromised host and the switch. For example, in the network diagram, if the compromised host is on the 192.168.1.0/24 subnet, the tap should be placed in between the host and the switch. This often involves placing a system in between the host and the switch.
Another option is to configure a Switched Port Analyzer (SPAN) port. In this configuration, the switch closest to the compromised host will have port mirroring enabled. This then sends the traffic from the entire segment the switch is on to the system that is on the mirrored port.
Finally, some network devices have built-in applications, such as tcpdump, that can be utilized to capture traffic for further analysis. This may be the quickest option, as it does not require physical access to the network or the switch and can be set up remotely. The drawback to this method is that storage on the switch may not support a large capture file and the added strain may increase the chances of some packets not being captured.
tcpdump is a command-line tool specifically designed for packet capture. tcpdump is often included with Linux distributions and is found on many network devices. For many of these devices, tcpdump has to be run as a root user or with root privileges, as it will be monitoring network traffic. The relevant documentation is available at http://www.tcpdump.org/. To perform a packet capture with tcpdump, the following process can be used:
arkime@arkime: :~$ tcpdump -h
The command will produce the following help menu:
Figure 5.3 – The tcpdump help menu
The default tcpdump setting is to capture traffic on all available interfaces. Running the following command produces a list of all the interfaces that tcpdump can capture traffic on:
arkime@arkime::~$ tcpdump -D
The following screenshot shows that in this case, the ens160 (Ethernet) interface and the lo (loopback) interface are available for capturing traffic:
Figure 5.4 – tcpdump capture interfaces
arkime@arkime:~$ sudo tcpdump -i ens160 -v
The -i switch tells tcpdump which interface to perform the packet capture on. In this case, it is on the following Ethernet interface: ens160. The -v switch sets the verbosity of the packet capture. In this case, the verbosity is set rather low. For additional data, the switch can be set to -vvv for a more detailed look at the packets. The following screenshot shows what information is displayed by the command:
Figure 5.5 – The tcpdump command output
While this method determines whether traffic is traversing that interface or not, the individual packet information is useless to an analyst due to the speed with which the individual packets appear on the screen. For the packet capture to be of any use, it is recommended that you output the file so that a later examination can be performed with a packet analysis tool such as Wireshark. Wireshark will be reviewed later in this chapter and in greater detail in Chapter 9.
arkime@arkime:~$ sudo tcpdump -i ens160 -vvv -w ping_capture
PING command
PING (short for Packet Internet Groper) is the utility that uses the ICMP packets being sent in this section. PING is used to determine whether there is network connectivity to various systems. In this case, a check of connectivity to the Google DNS at 8.8.8.8 is being performed.
The command tells tcpdump to capture network traffic and write the file out to capture. Unlike the previous capture, there is no traffic indicated on the screen.
Figure 5.6 – The tcpdump output
Figure 5.7 – Wireshark packet capture analysis
tcpdump can also be configured to focus the capture on specific sources or destination IP addresses and ports. For example, if an incident response analyst needs to collect packets leaving a specific host at the 192.168.10.54 IP address, the following tcpdump command will produce the desired results:
arkime@arkime:~$ sudo tcpdump -i ens33 src host 192.168.10.54
Packets going to a destination such as a known C2 server at the IP address can also be separated from the background network traffic with the following command:
arkime@arkime:~$ sudo tcpdump -i ens33 dst host 162.4.5.23
tcpdump is a powerful tool and has plenty of options. Incident response analysts are advised to examine and incorporate its various features into their toolkit.
During an incident, it may become necessary to obtain a packet capture from a Windows system. In incidents such as a compromised web server or application server, a Windows system will not have a native application with which to conduct a packet capture. There are several packet capture tools available on Windows systems. The first tool that can be utilized is WinPcap. This tool is generally recognized as the standard for packet capture on Windows systems and is available for free download at https://www.winpcap.org/. The drawback to this tool from a forensics perspective is that it must be installed on the system. This can complicate a forensic analysis, as any changes to the system have to be thoroughly documented. For this reason, it is a good preparatory step to ensure that high-risk systems such as web servers, file servers, and application servers already have WinPcap installed.
Another option available to incident response analysts is the use of tools such as RawCap. RawCap has the same basic capability as WinPcap without the need to install it on the local system. RawCap can be easily run from a USB device attached to the system. To perform a packet capture with RawCap, the following process is used:
D:>RawCap.exe -help
The command will produce the following output:
Figure 5.8 – The Rawcap.exe menu
The output produces a list of interfaces. One of the advantages of RawCap is that, even from a USB device, the incident response analyst can perform a packet capture on each of the interfaces. In this example, the capture will be performed on the Ethernet interface indicated by the number 0.
C:ProgramDatachocolateyinRawCap.exe 0 RawCap.pcap
The command produces the following output:
Figure 5.9 – The output of a RawCap packet capture
Figure 5.10 – Analysis of the RawCap file in Wireshark
Now that we have introduced Wireshark, we will examine how the tool can also be used to capture network traffic.
Wireshark is a Unix or Windows packet capture and analysis tool. Unlike tcpdump or tools such as RawCap, Wireshark is a GUI-based tool and includes not only packet capture but also analysis features. As a result, Wireshark may be difficult to deploy rapidly during an incident, as the program has to be installed. Furthermore, the tool is only supported on Windows or macOS. Installing Wireshark on a Linux system requires a bit more effort. The one distinct advantage that Wireshark has over command-line options is that incident response analysts can perform a detailed inspection of the traffic as it is being captured. Wireshark can be run on the system itself or on a USB drive. Once installed, it must be run as an administrator. To perform a packet capture with Wireshark, the following process is used:
Figure 5.11 – Wireshark Capture interfaces
In the screenshot, there is only one interface that appears to be handling traffic. The capture will be performed on the Ethernet0 interface.
Figure 5.12 – A Wireshark capture view
Another tool that is included with Wireshark and is useful during evidence acquisition is mergecap. mergecap is a command-line tool that allows incident response analysts to combine multiple packet capture files from Wireshark, tcpdump, or RawCap. This is extremely useful in situations where incident response analysts obtain packet captures from several sources but want to check for traffic to a specific host. To access the menu for mergecap, type the following into Command Prompt:
arkimie@arkime:~$mergecap -help
That command produces the following help information:
Figure 5.13 – The mergecap help menu
To merge several packet capture files, the following command is used:
arkime@arkime:~$mergecap -w switches.pcap switch1.pcap switch2.pcap switch3.pcap
By combining the output of three packet captures into one file, the incident response analyst can examine a wider range of activities across multiple network paths. If, for example, the analyst is searching for traffic coming from an unknown host to an external C2 server, they would be able to combine captures over the entire span of the network and then search for that IP, rather than individually picking through each packet capture.
In order to conduct a proper examination of log files and other network data such as packet captures, they often have to be moved from the log source and examined offline. As with any source of evidence, log files or packet captures have to be handled with due care to ensure that they are not corrupted or modified during the transfer. One simple solution is to transfer the evidence immediately to a USB drive or similar removable medium. From there, a hash can be created for the evidence prior to any examination.
The acquisition of network evidence such as a packet capture or a log file should be thoroughly documented. Incident response personnel may be acquiring log files and packet captures from several sources over the entire network. As a result, they should ensure that they can trace back every separate piece of evidence to its source, as well as the date and time that the evidence was collected. This can be recorded on a network evidence log sheet and entries can be completed for each piece of evidence. For example, see the following entry:
Figure 5.14 – A network evidence collection entry
The log entry captures the following necessary information:
arkime@arkime:~$md5sum --help
That produces the following help menu:
Figure 5.15 – The md5sum help menu
The MD5 hash can be calculated for the packet capture from the switch by simply entering the following command:
arkime@arkime:~$md5sum ping_capture
This produces the following output:
Figure 5.16 – A md5sum file calculation
Setting the time standard
Prior to an incident, it is important to identify what time zone will be in use. From an evidentiary standpoint, the time zone does not really matter as long as it is consistent throughout the entire incident investigation.
Once the collection is complete, a chain of custody form should also be filled out for the external medium that contains the evidence files. From here, the files can be analyzed.
Evidence that is pertinent to incident responders is not just located on the hard drive of a compromised host. There is a wealth of information available from network devices spread throughout the environment. With proper preparation, a CSIRT may be able to leverage the evidence provided by these devices using solutions such as a SIEM. CSIRT personnel also has the ability to capture network traffic for later analysis through a variety of methods and tools. However, all these techniques are influenced by the legal and policy implications that CSIRT personnel and the organization at large need to navigate. By preparing for the legal and technical challenges of network evidence collection, CSIRT members can leverage this evidence and move closer to the goal of determining the root cause of an incident and bringing the organization back to its full operation.
This chapter discussed several sources of evidence available to incident response analysts. Logs from network devices, whether they report to a SIEM or through other methods, can give you an insight into what has transpired in the network. Packet captures provide details about the exact nature of network traffic. Finally, analysts must be prepared to acquire these sources of evidence in a forensically sound manner.
In the next chapter, the focus will shift from network evidence acquisition to acquiring volatile data from host-based systems.