Chapter 5. Statistical Flow Analysis

“Often think of the rapidity with which things pass by and disappear. . . . For substance is like a river in a continual flow, and the activities of things are in constant change, and the causes work in infinite varieties; and there is hardly anything which stands still.”

—The Meditations, by Marcus Aurelius¹

1. Marcus Aurelius, The Meditations (The Internet Classics Archive, 2000), http://classics.mit.edu/Antoninus/meditations.mb.txt.

If you were capable of watching all of the automobiles driving on the roads within a geographic area, mapping sources and destinations, volumes of people and objects moved, and directions of transfer, you would be able to tell quite a lot about the activities of the people in the area. Not only could you make broad statements about the general habits of all the people, you could also identify areas of congestion on the roads and unusual volumes or types of activity. If you had very granular data, you could track vehicles from specific home addresses, to offices, stores, and back. You could infer the normal work times of individuals, general interests, and hobbies. You could look back in time after a crime had been committed—let’s say a robbery at a store—and make a list of all the people who were likely present during that time frame.

In many ways, network flow record analysis is analagous to analysis of traffic patterns on the roads in real life, except that it is far easier to gather granular information on computer networks.

Every packet that traverses a network can be logged and recorded. Since the early days of networking, it has been common to record basic information about flows,² including source and destination IP address, source and destination port (where applicable), protocol, date, time, and the size of the data transmitted in each packet. Network flow records were originally generated, captured, and analyzed to improve network performance, but gradually it has become clear that the data is extremely valuable for other reasons as well. This information can provide network engineers and investigators with a wealth of information about activity on the network.

2. N. Brownlee, “RFC 2722—Traffic Flow Measurement: Architecture,” IETF, October 1999, http://www.rfc-editor.org/rfc/rfc2722.txt.

Forensic analysts investigating a crime may find that flow records can reveal a detailed potrait of network activity. In network forensic investigations, statistical analysis of flow data can serve a variety of purposes, including:

• Identifying Compromised Hosts Often, compromised hosts send out more traffic than normal, transmit or receive traffic on unusual ports, or communicate with other systems that are known to be malicious.

• Confirming or Disproving Data Leakage Network investigators often receive flow records with requests to determine whether an unknown attacker exported sensitive information across the network perimeter. If the time frame is bounded, and flow records are comprehensive, it is possible to analyze the volume of exported data and determine whether a leak may have occurred.

• Individual Profiling Flow data can provide a surprisingly rich picture of an individual’s activity. Flow data from a user’s workstation can reveal normal working hours, periods of inactivity, lunch times, break times, sources of entertainment, inappropriate activity, and more. By correlating activity between endpoints, you may determine which users communicate and exchange files or URLs between each other. Profiling can be especially helpful in cases that involve human resources issues.

Statistical flow analysis is becoming increasingly important in the modern world due to the sheer volume of traffic that organizations produce, manage, and inspect. It is easily possible for forensic investigators to be so inundated with data that we lack the processing power to crunch through it all. Consequently, we must be very judicious about which bits of information we collect. While it is possible to record full contents of all packets, this would have a high impact on performance, take up a large amount of storage space, and is generally not necessary. Statistical flow analysis can help investigators identify specific targets for content analysis and further investigation.

Every bit of data that is not collected is lost forever. Yet too much useless data can make analysis difficult or impossible.

5.1 Process Overview

In order to gain intelligence about what is flowing on a network, flow records must be generated, collected, and analyzed. A “flow record” is simply a summary of information about a flow, which typically includes the source and destination IP address, ports, date and time, and the amount of data transmitted. Other information may be included in a flow record as well, depending on the equipment and software generating the flow record, as well as the local configuration.

A “sensor” is what we commonly call the device that is used to monitor the flows of traffic on any given segment and extract critical bits of information in a flow record. Network equipment rarely includes much disk space for storing logs of any type, including network flow data. As a result, sensors are designed to immediately export flow records across the network, once the flow is complete.³ Flow record data is exported from a sensor to a “collector,” which is a server configured to listen on the network for flow record data and store it to a hard drive. An organization may have multiple collectors to capture flow record data from different network segments, or to provide redundancy. Once the flow record data has been exported and stored on a collector or multiple collectors, it can be aggregated and analyzed using a wide variety of commercial, open-source, and homegrown tools.

3. R. Stewart, “RFC 4960—Stream Control Transmission Protocol,” IETF, September 2007, http://www.rfc-editor.org/rfc/rfc4960.txt.

Forensic investigators interested in analyzing flow record data should start by evaluating whether the flow record data of interest is already being collected. If not, you will need to decide whether it is appropriate and/or possible for you to initiate collection and analysis of flow data, and what strategy will work best for the environment under investigation.

5.2 Sensors

Every organization’s network environment is unique. Some organizations may already have flow sensors installed before an investigation commences, or at least their existing network equipment may be capable of producing flow export data if configured properly. Other organizations may have no such equipment, or the existing sensors may not be suitably placed for generating flow records necessary for the investigation. The architecture of the local network environment and the types of equipment available for use as sensors will strongly influence whether flow record data can be produced within the resources available for the investigation, and will also affect the resulting output format of the flow export data and subsequent analysis techniques.

5.2.1 Sensor Types

Sensors can be deployed as standalone appliances, or as software processes running on network equipment that serves other purposes. Many common types of network equipment are capable of producing flow record data, although network administrators do not always take advantage of these features. Specific types of network equipment typically support only limited output formats for flow record exportation, and this is one of the primary reasons that setting up a standalone appliance may be preferable.

5.2.1.1 Network Equipment

Some switches support flow record generation and exportation. For example, many popular Cisco Catalyst switches export flow record data. Likewise, current revisions of Cisco routers and firewalls support the ability to produce flow data and export it to a collector, using Cisco’s proprietary NetFlow format (which we discuss in detail later in this chapter). Other vendors, such as Sonicwall, now support an open, standardized version of the flow record exportation protocol (IPFIX), in addition to NetFlow.^{4, 5} Be careful because some vendors who advertise support for flow record exportation actually base the exported flow record data on packet sampling, rather than a comprehensive inspection of all flows. Sampled data is generally not appropriate for network forensics because it will not provide you with complete information.

4. “NSA 3500 Network Security Appliance—SonicWALL, Inc.,” SonicWall, 2011, http://www.sonicwall.com/us/products/NSA_3500.html#tab=specifications.

5. “NetFlow Ninjas—Attention Firewall Vendors: Please Add NetFlow Support!,” July 20, 2010, http://netflowninjas.lancope.com/blog/2010/07/attention-firewall-vendors-please-add-netflow-support-asap.html.

5.2.1.2 Standalone Appliances

Although it is common to generate flow records using network equipment that is already installed, in some cases the local network equipment may not be configured to export flow record data when the investigator arrives onsite. Furthermore, the existing network equipment may not generate flow data in a format that local administrators or forensic investigators can use for collection or the equipment may not be capable of generating flow records at all. In these instances, network administrators or forensic investigators may choose to deploy a standalone appliance as a sensor for generating and exporting flow record data.

You can deploy a software-based sensor anyplace on the network where you can capture traffic. Organizations can set up port monitoring or a network tap to send traffic to the standalone sensor, which in turn processes traffic, generates statistics, and sends flow record data to the collector. Since the standalone server does not need to concurrently function as a router or perform other functions, there are usually performance benefits to deploying standalone sensors to export flow record data.

Forensic investigators work to maintain a small footprint within any environment under investigation. For this reason, it is often preferable to set up a separate, standalone sensor rather than modifying existing network infrastructure devices and risk impacting production functionality.

5.2.2 Sensor Software

Most modern, enterprise-quality routers and switches can be configured to act as sensors and export flow record data across the network to collectors. This includes devices manufactured by widely used vendors such as Cisco, Juniper, SonicWALL, and others. Alternatively, you can set up standalone sensors using free, open-source tools such as Argus, softflowd, or yaf. We now discuss each of these in turn.

5.2.2.1 Argus

Argus stands for “Audit Record Generation and Utilization System.” It is a mature, libpcap-based network flow sensing, collection, and analysis toolkit. Argus was first developed at Carnegie Mellon and released to the public in 1996. It is open-source software, copyrighted by QoSient, LLC and released under the GNU Public License. Partly due to its long history, it has been tested on Mac OS X, Linux, FreeBSD, OpenBSD, NetBSD, Solaris, and has also been ported to many other operating systems (as well as Cygwin).⁶

6. “ARGUS—Auditing Network Activity—FAQ,” July 1, 2009, http://www.qosient.com/argus/faq.shtml.

Argus is distributed as two packages: the Argus server, which reads packets from a network interface or packet capture file; and the Argus client tools, which are used to collect, distribute, process, and analyze Argus data. The Argus server functions as a sensor and can output flow export data in Argus’ compressed format, as files or over the network via UDP. As a libpcap-based tool, Argus supports BPF filtering, which allows the user to fine tune filtering and maximize performance. It also supports multiple output streams, which can be directed to different collector processes or hosts.

Argus is one of the few flow record toolkits that specifically includes mention of forensic investigation in official documentation:

“Historical netflow data can be used in forensic investigations several months, or years, after an incident has taken place. Argus’ netflow records offer up to a 10,000:1 ratio from the packet size to the record written to disk, which allows installations to save records for much longer than full packet captures. When network security is very important, non-repudiation becomes a very important requirement that must be provided throughout the network. Argus provides the basic data needed to establish a centralized network activity audit system. If done properly, this system can account for all network activity in and out of an enclave, which can provide the basic system needed to assure that someone can’t deny having done something in the network.”⁷

7. “Argus—NSMWiki,” Sguil, 2011, http://nsmwiki.org/index.php?title=Argus#Who_uses_argus.

5.2.2.2 softflowd

“Softflowd” is an open-source flow monitoring tool developed by Damien Miller. It is designed to passively monitor traffic and export flow record data in NetFlow format (versions 1, 5, and 9 are supported, as of the time of this writing).⁸ It can be installed on Linux and OpenBSD operating systems, and is based on libpcap.

8. “Softflowd—Fast Software NetFlow Probe,” March 28, 2011, http://www.mindrot.org/projects/softflowd/.

The hardware requirements are minimal; you need a network card capable of capturing your target volume of traffic and a second network card for exporting flows from the sensor to a collector. Select disk space based on the volume of flow data you would like to retain. (See Chapter 3, “Evidence Acquisition,” for more details on how to passively gain access to network traffic.)

5.2.2.3 yaf

“Yaf,” stands for “Yet Another Flowmeter.” As the name suggests, it is open-source flow sensor software.⁹ Released in 2006 under the GNU GPL, yaf is based on libpcap and can read packets from live interfaces or packet captures. It exports flow data in the IPFIX format, over SCTP, TCP, or UDP transport-layer protocols. One of the most powerful features of yaf is that it supports BPF filters for the purposes of filtering incoming traffic. This can allow investigators and network engineers to dramatically reduce the required processing capabilities and volume of flow export data.¹⁰ Yaf natively supports encrypted flow export using TLS.¹¹

9. Brian Trammell, “YAF—Documentation,” CERT NetSA Security Suite, http://tools.netsa.cert.org/yaf/yaf.html.

10. Christopher M. Inacio and Brian Trammell, “YAF: Yet Another Flowmeter,” August 23, 2010, http://www.usenix.org/event/lisa10/tech/full_papers/Inacio.pdf.

11. “YAF Documentation.”

5.2.3 Sensor Placement

Forensic investigators don’t always have much control over where sensors are placed in an environment under investigation. In the ideal situation, network infrastructures would be designed with flow monitoring in mind, for the purpose of performance tuning, as well as incident response and forensics. However, an investigator may walk into an organization that has little or no flow record collection set up. In some cases, you may have the opportunity to initiate flow record exportation and collection during the course of an investigation.

Network engineers and forensic investigators can work together to design a network that will facilitate flow record collection and analysis. Factors to consider when placing sensors include:

• Duplication Minimize duplication of flows collected. Duplication is inefficient, increases the resources needed for analysis, and adds to network congestion during aggregation. If you collect two of every packet because of where your sensors are placed, your processing and storage capacity is cut in half. There are times when it is necessary to place sensors at points where traffic will pass through multiple sensors, and in these cases, you can use filtering to reduce (but usually not eliminate) duplication. Make sure to place sensors in locations where you will obtain your targeted data, of course, while minimizing redundant data collection. To paraphrase Albert Einstein, “Make things as simple as possible, but no simpler.”

• Time synchronization Time synchronization between sensors is crucial. If the time on a sensor is not accurate, it is difficult to correlate flows exported by that device with any other device, or evidence gathered from other resources. If you collect flow records from a device that is already set up, note the clock skew between the sensor and an accurate source. If you set up your own sensor, make sure to keep the time synchronized using NTP if possible.

• Perimeter versus internal traffic Most enterprises prioritize flow record collection from perimeter devices such as external firewalls. However, if all you do is monitor the perimeter, you’ll never see what’s going on inside the network. Keep in mind that having visibility inside the network is also very valuable. For example, frequently workstations are on a separate subnet from the core data center. Monitoring connections between these two subnets is likely to be of interest in the event of compromise. Often, it is valuable for forensic evaluators to have visibility within a single subnet. Local flow data can help identify compromised workstations that are seeking to propagate to new targets, both internal and external.

• Resources Remember that there are as many bits flowing across the Internet every day as there are grains of sand on all the beaches in the world. You can’t have them all. Every investigator has limited resources. You have to decide, based on your funding, equipment, staff, and time, which of those grains are most important. Review the local network map carefully and choose choke points that will maximize your collection capacity while still fitting within your budget and your ability to process and analyze.

• Capacity Network devices have limited capacity to monitor and process traffic. Enabling flow record processing and exportation can impact the performance of network equipment, especially if it is already at high utilization. Fortunately, most modern, enterprise-class network devices have built-in functionality to help you understand the utilization and capacity in advance. For standalone devices, it is possible to place the sensor in a location where it simply cannot process the volume of traffic passing by. Many switches will allow you to oversubscribe a port while doing mirroring. Try to be aware in advance if capacity will be an issue and, if so, employ more filtering, or partition VLANs to multiple sensing ports and deploy multiple sensors or sensing interfaces.

5.2.4 Modifying the Environment

In most cases, flow record data is not recorded to the extent that an investigator would prefer for the case at hand. It may be appropriate to initiate collection of supplemental flow record data for a specific case, depending on the environment and the nature of the investigation. When an investigative team has decided to gather additional flow record data by modifying the environment, there are a few general options available:

• Leverage existing equipment The existing network equipment within the infrastructure may be capable of exporting flow record data. This can include switches, routers, firewalls, and NIDS/NIPS equipment. In this case, investigators may be able to leverage this built-in capability through minor configuration changes to the equipment. However, it is crucial to examine the network device capacity before initiating additional flow record collection to ensure that the network functionality is not negatively impacted and also that the flow record collection will be sufficient. Remember also to check in advance and make sure that the format of flow record data exported by local equipment is supported by your collection system.

• Upgrade network equipment If existing network equipment does not support flow record exportation or cannot handle the increased capacity, investigators may choose to deploy replacement switches or other network devices that can handle the increased capacity. Depending on the type of network equipment, this transition may be straightforward or may require extensive reconfiguration.

• Deploy additional sensors Many devices that don’t support flow record exportation still support port mirroring. Investigators can use port mirroring to send packets of interest to a standalone sensor (such as Argus or softflowd). Similarly, investigators may choose to deploy a network tap and send data to a standalone sensor for flow record collection.

Make sure that you carefully document any changes that you make to the environment, especially if you are involved in a case that may at some point wind up in court.

5.3 Flow Record Export Protocols

Flow records can be exported in a variety of formats, ranging from proprietary protocols such as Cisco’s NetFlow to the IETF’s open standard, IPFIX. Different formats can store different types of data, and some formats allow for far more local customization than others. In this section, we highlight a few of the most popular protocols.

Flow record processing is still relatively new (particularly in the context of forensics), and because it is not yet mature, tools and equipment typically only support a small subset of flow record export formats and often are not compatible with each other. Before you begin your evidence collection and analysis, be sure to research the flow record exportation formats supported by the equipment and tools used in your investigation.

5.3.1 NetFlow

NetFlow was first developed in 1996 by Cisco Systems, Inc.¹² for caching and exporting traffic flow information. Originally, NetFlow was designed to cache flow information for the purposes of improving performance. The flow statistics generated proved to be useful for network analysts in monitoring network performance, debugging issues, and responding to security threats.

12. Cisco Systems, Inc., “NetFlow Gives Network Managers a Detailed View of Application Flows on the Network,” 2004, https://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps6601/prod_case_study0900aecd80311fc2.pdf.

Many devices can produce NetFlow data, from routers to switches to standalone probes. The sensor maintains a cache that tracks the state of all active flows observed by the device. When a flow is completed or is no longer active, the sensor marks the flow as “expired” and then exports the flow record data in a “NetFlow Export” packet that is transmitted over the network to a collector (or multiple collectors). For NetFlow version 9, the latest release as of this writing, the “NetFlow Export” packet is transport-layer independent and can be transmitted over UDP, TCP, or even SCTP (earlier versions of NetFlow supported transport only over UDP). Although NetFlow was developed by Cisco, it became so popular that other network device manufacturers included support for the format, including Juniper¹³ and Enterasy.¹⁴

13. “Netflow Configuration Juniper,” Caligare, May 10, 2006, http://netflow.caligare.com/configuration_juniper.htm.

14. “Wireless Access Points, Cloud Based Solutions, Data Center Solutions,” Enterasys, July 15, 2010, http://secure.enterasys.com/support/manuals/hardware/NetFlowFeatGde071510.pdf.

NetFlow version 5 (NetFlow v5) is straightforward and widely used on a wide variety of equipment from different manufacturers. However, NetFlow v5 has some important limitations—for example, it supports only IPv4 (not IPv6) and Export Packets must be transported over UDP, which is an unreliable transport-layer protocol.

NetFlow version 9 (Netflow v9) is the latest release from Cisco and was also selected by the IETF as the basis for the IPFIX standard¹⁵ (see “IPFIX” below). In addition to providing support for IPv6 and transport-layer independent exportation, NetFlow v9 introduced support for flexible template-based specification of flow record fields. This allows users to customize the data cached and exported by a sensor, enabling both greater flexibility for analysis as well as performance improvements.

15. B. Claise, “RFC 3954—Cisco Systems NetFlow Services Export Version 9,” IETF, October 2004, http://www.rfc-editor.org/rfc/rfc3954.txt.

5.3.2 IPFIX

The IP Flow Information Export (IPFIX) protocol is a successor to Cisco’s NetFlow, specified originally in RFC 5101¹⁶ and based on NetFlow v9. It is a result of the efforts of an IETF Working Group that sought to extend the NetFlow v9 protocol to handle bidirectional flow reporting,¹⁷ to reduce data redundancy when reporting on flows with similar attributes,¹⁸ and to provide for better interoperability as an open standard.¹⁹

16. B. Claise, “RFC 5101—Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information,” IETF, January 2008, http://www.ietf.org/rfc/rfc5101.txt.

17. B. Trammell and E. Boschi, “RFC 5103—Bidirectional Flow Export Using IP Flow Information Export (IPFIX),” IETF, January 2008, http://www.ietf.org/rfc/rfc5103.txt.

18. E. Boschi, “RFC 5473—Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports,” IETF, March 2009, http://www.ietf.org/rfc/rfc5473.txt.

19. “IP Flow Information Export (ipfix),” Datatracker, 2011, http://datatracker.ietf.org/wg/ipfix/.

The flow record data that IPFIX supports is extensible through data templates. The collector sends the template defining the data to be exported, and the sensor then uses that template to construct the flow data export packets that are sent back to the collector as the flows expire.

5.3.3 sFlow

sFlow is a standard for providing network visibility based on packet sampling. It was developed by InMon Corporation and published as an IETF RFC in 2001²⁰ (the specifications are now maintained at sFlow.org).²¹ sFlow is supported by many network device manufacturers (although notably, not Cisco).²²

20. P. Phaal, “RFC 3176—InMon Corporation’s sFlow: A Method for Monitoring Traffic in Switched and Routed Networks,” IETF, September 2001, http://www.rfc-editor.org/rfc/rfc3176.txt.

21. “Specifications for Developers,” Sflow, 2011, http://www.sflow.org/developers/specifications.php.

22. “sFlow Products: Network Equipment,” 2011, http://www.sflow.org/products/network.php.

sFlow is different from Netflow and IPFIX, in that it is designed to conduct statistical packet sampling and does not support recording and processing information about every single packet. The benefit of packet sampling is that it scales well to very large networks, with high throughput. The down side is that packets that are not sampled are not recorded and cannot be analyzed. For this reason, NetFlow and IPFIX are likely to be better options for the purposes of forensic analysis.

5.4 Collection and Aggregation

Once you have exported flow record data from a sensor, you need a collector or multiple collectors to receive data in the export format and store it on disk. The architecture of the collector system is very flexible. A collector can run on a wide variety of hardware and software platforms as long is it supports the flow export format used to export the flow record data of interest.

5.4.1 Collector Placement and Architecture

It’s important to carefully consider where to place your collectors within the infrastructure, how to secure the system and the flow record data in transit, and how many collectors will be needed. Factors to consider include:

• Congestion Flow record exportation generates network traffic. This can exacerbate network congestion or, conversely, network congestion can cause problems with flow record exportation. When placing collectors, it is wise to choose locations within the network where transit times between sensor and collector are unlikely to be impacted by either routine or incident-based congestion (such as that caused by a worm).

• Security Can flow export data be “sniffed” or modified in transit? It’s important to consider these questions when setting up flow export collectors, particularly if you suspect the local network may be compromised. While there is no “silver bullet,” forensic investigators have options for implementing reasonable security measures, depending on the environment.

To protect the confidentiality of flow export data, investigators can place collectors on segments where the paths between collector and sensor are access-controlled and well protected. Exporting flow record data on a separate VLAN from other traffic is usually a good idea. Collectors and sensors can even be networked across an isolated physical cable, or run on the same physical system, with sensor data exported across the localhost interface.

It is also possible to encrypt the flow export data as it is transmitted across the network, with protocols such as IPSec or TLS.²³ While flow record data is not commonly encrypted in transit, forensic investigators should strongly consider this as an option when it is supported by sensor and collector software, especially when flow export data is transmitted across insecure or shared network segments.

23. S. Leinen, “RFC 3955—Evaluation of Candidate Protocols for IP Flow Information Export (IPFIX),” IETF, October 2004, http://www.rfc-editor.org/rfcs/rfc3955.html.

• Reliability Depending on the transport-layer protocol, packets dropped due to network congestion or denial-of-service attacks may be lost forever. Traditionally, flow export data has been transmitted over UDP, a connectionless protocol with no built-in support for reliability. Over the years, flow export formats have evolved to support reliable transport-layer protocols as well, such as TCP and SCTP. It is wise to use transport-layer protocols with built-in reliability whenever possible, especially when flow collection is being conducted for the purposes of forensic investigation.

• Capacity One of the main design choices to consider is how many collectors to set up. On the one hand, consolidating collection on a single system reduces hardware and software costs and facilitates aggregation and analysis. On the other hand, it’s important to ensure that each collector can keep up with the bandwidth generated by the sensors, based on its network card capabilities, processing power, RAM, and storage space. A single physical system can run multiple collector processes listening for input on different ports, which can help system administrators track usage and facilitate remediation in the event of overload. Since hardware requirements for a collector are generally flexible, adding additional collectors is usually fairly easy to accomplish.

• Strategy for Analysis Your strategy for analysis will also affect placement, software choices, and configuration of collectors. If you are purchasing a commercial tool which has collector and analysis functionality bundled together, you will have different constraints than if you are planning to have a separate analysis system that can receive flow report data from multiple sources and aggregate it. Consider the architecture of your analysis environment when setting up the collectors.

5.4.2 Collection Systems

There are commercial collectors that often double as analysis tools. These include Cisco NetFlow Collector, Manage Engine’s NetFlow Analyzer, WatchPoint NetFlow Collector, and others.

There are also a wide spectrum of free and open-source collector tools. These are often bundled as part of a larger flow record collection and analysis suite. We discuss four open-source collector tools in this section: flowcap (from the SiLK suite), flow-capture (from the flow-tools suite), nfcapd (from the nfdump suite), and the Argus collection utility.

5.4.2.1 SiLK

The “System for Internet Level Knowledge” (SiLK) is a suite of command-line tools for the collection and storage of flow data that provides a very powerful toolset for the filtering and analysis of flow records.²⁴ One of several projects produced by the Network Situational Awareness (NetSA) group at CERT,²⁵ SiLK can process NetFlow versions 5 and 9 and IPFIX data and produce statistical data in a variety of output formats, configurable to a fine level of granularity. Of course, as with most extremely powerful and flexible toolsets, there is a cost in the form of a learning curve.

24. “SiLK,” CERT NetSA Security Suite, 2011, http://tools.netsa.cert.org/silk.

25. “Featured Projects,” CERT NetSA Security Suite, 2011, http://tools.netsa.cert.org.

SiLK includes two collector-specific tools, flowcap and rwflowpack. The “rwflowpack” tool is designed to collect flow record data from a network socket or files and export it to the compressed SiLK Flow format. It supports NetFlow version 5, NetFlow version 9, and IPFIX, although IPFIX and NetFlow v9 are only supported if SiLK is compiled with support for NetSA’s libfixbuf²⁶ software library.²⁷

26. “Fixbuf,” CERT NetSA Security Suite, 2011, http://tools.netsa.cert.org/fixbuf/libfixbuf/index.html.

27. “SiLK sensor.conf,” CERT NetSA Security Suite, 2011, http://tools.netsa.cert.org/silk/sensor.conf.html.

Rwflowpack can also accept input over a TCP socket from the SiLK flowcap tool. Flowcap similarly listens for flow records on a network socket, temporarily stores flow data to disk or RAM, and forwards the compressed stream to a client program such as rwflowpack.²⁸ This architecture allows flow records to be collected close to the sensor and then is transferred via TCP to rwflowpack for processing. The SiLK collector tools support TCP and UDP for IPFIX flow record collection and UDP for NetFlow flow record collection.

28. “Flowcap—Capture, Store, and Forward Flow Data,” 2006, http://silktools.sourceforge.net/flowcap.pub.html.

5.4.2.2 flow-tools

The flow-tools suite is an open-source collection of flow export data collection and analysis tools. It is modular and easily extensible. The flow-capture utility is designed to collect NetFlow traffic exported by sensors via UDP. As of the time of this writing, flow-capture supports NetFlow versions 1, 5, 6, 7, and 8.1-14. However, NetFlow version 9 and IPFIX are not supported.²⁹ This is a serious limitation, especially for forensic investigators, since earlier versions of NetFlow do not provide support for reliable transport protocols. The flow-capture tool only accepts input on UDP ports and does not natively support flow export data collection via TCP or SCTP.

29. Mark Fullmer, “flow-capture(1)—Linux man page,” 2011, http://linux.die.net/man/1/flow-capture.

5.4.2.3 nfdump/NfSen

The nfdump suite includes tools for the collection, display, and analysis of network flow record data. The collector daemon, nfcapd, supports NetFlow versions 5, 7, and 9. It can read flow export data from a UDP network socket or pcap files, and it allows for extensive customization of the data files stored on disk. As with flow-tools, the transport-layer limitation is important for forensic investigators since forensics typically requires a higher degree of reliability than performance analysis. Nfcapd is designed to be integrated with the nfdump and NfSen display and analysis tools.³⁰ The nfdump and NfSen tools are released under the open-source BSD license.³¹

30. “NFDUMP,” SourceForge, July 28, 2010, http://nfdump.sourceforge.net/.

31. Peter Haag, “Watch your Flows with NfSen and NFDUMP” (Switch: The Swiss Education and Research Network, May 3, 2005), http://www.ripe.net/ripe/meetings/ripe-50/presentations/ripe50-plenary-tue-nfsen-nfdump.pdf.

5.4.2.4 Argus

The Argus client tools suite also includes a collector. All of the Argus client programs accept input both in the Argus format and NetFlow versions 1–8.³² Unfortunately, NetFlow version 9 and IPFIX are not yet supported, although Qosient has indicated that support for NetFlow 9 is in development. The Argus “radium” (“Argus Record multiplexor”) client program is particularly useful for collection and distribution of flow export data because it can accept Argus or NetFlow input from multiple network or filesystem sources and sends Argus output to up to 128 client programs via the network or filesystem. As with other Argus client programs, radium supports the use of BPF-style filter expressions.

32. “Auditing Network Activity—Argus and Netflow,” March 26, 2010, http://www.qosient.com/argus/argusnetflow.shtml.

5.5 Analysis

Flow record data lends itself very well to statistical analysis. Although each flow record contains only a very limited set of information about each flow, aggregated over time, this information can be used to create a detailed picture of activity on the network, predict normal behavior, and identify anomalies. The costs of collecting and aggregating network flow record data are remarkably low when compared to the effort it takes to collect and aggregate similarly descriptive data about real-world activities such as highway traffic or pedestrian movements. As network forensic analysts, we are lucky to have access to so much high-fidelity data regarding our environment. The next question is, What can we do with it?

In this section we discuss common analysis techniques that are useful for investigators with access to flow data and also provide you with an overview of some popular tools.

5.5.1 Flow Record Analysis Techniques

The purpose of flow record collection is to store a summary of information about the traffic flowing across the network. As a result, forensic analysis techniques such as carving content out of traffic flows do not apply to flow record data. That said, forensic analysis of flow record data is exceptionally powerful because the limited scope of data stored for each flow allows for a huge number of flows to be analyzed in ways that would simply exceed the scope of available storage and processing capabilties if full packet captures were used for the same purpose.

The precise analysis techniques that you employ will vary on a case-by-case basis, depending on your investigative goals and resources. As we discuss in the next section, the tools you choose will also impact your analysis (and vice versa). There are some analysis techniques, primarily statistical in nature, which are particularly well suited for flow record forensics. Keep in mind that this field of study is still nascent and flow record analysis techniques will quickly advance in the years to come.

Flow record analysis is typically an iterative process. Important highlights include:

1. Goals and Resources As always, the goals of your investigation and available resources shape your analysis. Are you trying to identify compromised systems? Evaluate whether a data breach occurred? Investigate an HR violation?

Make sure to assess your available time, staff, equipment, and tools. How much flow record data can you process? Given the format of the data and your goals, what tools are most appropriate for use? As your investigation begins, strategize and identify the analysis techniques that will best fit with your investigative goals and available resources.

2. Starting Indicators Every investigation begins with some triggering event. Investigators are normally provided with some pieces of evidence that can be used to begin analysis (“starting indicators”). Starting indicators may include:

• IP address of a compromised or malicious system

• Time frame in which malicious activity is suspected, even if the participants are unknown

• Known ports on which a worm was suspected to be active

• Specific flow records that indicate abnormal or unexplained activity

You may be given the IP address of a compromised system, or the domain name of a suspicious web site, or a description of malware behavior to search for in flow record data. Each of these “starting indicators” is useful for different analysis techniques. The volume and type of data you begin with will shape your investigation.

3. Analysis Techniques

(a) Filtering Filtering is a general term for narrowing down a large pool of evidence to a subset—or groups of subsets—that are of interest. This is a fundamental technique in nearly every investigation that involves flow record data, since normally investigators are provided with far more flow record data than is relevant for the case at hand and must select only small percentages for detailed analysis and presentation.

(b) Baselining Using flow record data, network administrators can build a profile of “normal” network activity. Forensic investigators can compare flow record traffic to the baseline profile in order to identify anomalies.

(c) “Dirty Values” A common strategy in digital forensics of all types is to compile a list of suspicious keywords or other values to use when searching for evidence relevant to a case. For flow record data specifically, “dirty values” might include IP addresses, ports, or protocols.

(d) Activity Pattern Matching Every activity leaves a fingerprint on the network. Simple patterns, such as large unidirectional volumes of traffic flow, can indicate specific activities that may be suspicious given the context (such as the presence of a high-volume, Internet-accessible file server on a subnet that should only contain company workstations). More complex patterns may match the behaviors of known worms or viruses. A trained investigator can pick out suspicious activity that manifests as complex and subtle variations in flow record evidence.

In this section, we briefly discuss how the forensic analysis techniques of filtering, baselining, “dirty values” search, and activity pattern matching apply to flow record analysis.

5.5.1.1 Filtering

Filtering is fundamental to flow record analysis. As a forensic investigator, your job is to remove extraneous details and identify events, hosts, ports, and activities of interest. For example, if your starting indicator was a specific IP address, you might start by isolating all activity relating to that IP address and picking out subsets of flows for analysis until you have gathered the evidence you need and built your case. Alternatively, you might filter for flows that match particular patterns of activity indicative of worm behavior, data exfiltration, port scanning, or other suspicious behavior, depending on your investigative goals.

In rare instances, you may be provided with only the specific flows of interest in a case, but the vast majority of the time you will have access to a large volume of flow record data and will need to pick out the flows relevant to your case.

Most analysis techniques used for forensic analysis of flow records are based on filtering.

5.5.1.2 Baselining

One of the advantages of collecting flow record data (as opposed to full traffic captures) is that due to the dramatically smaller size of flow record data, longer historical periods can be kept accessible and perhaps even archived indefinitely.

Network traffic analysts can build, maintain, and reference a baseline of traffic and identify trends and patterns of activity that are considered “normal” for the environment (or at least understood and explainable). By aggregating a large volume of flow information, it’s possible to build a large, detailed set of benchmarks of what should be considered normal over any span of time and across business cycles. Forensic investigators can reference these baselines in order to analyze suspicious activity.

Network baselines

By looking at general trends over time for monitored segments, the traffic seen can be understood even if specific source and destination IP addresses must be abstracted or generalized.

Host baselines

Likewise, when a particular host becomes of interest, investigators can build or refer to a historical baseline of a specific host’s activities in order to identify or investigate anomalous behavior.

In most cases, the network flow patterns of a host change dramatically when it is compromised or under attack. By referring to a baseline of activity over time, you can often pick out approximate times of compromise or other important events relating to the investigation.

5.5.1.3 “Dirty Values”

In hard drive analysis, it’s common to compile a list of “dirty words”—keywords that relate to the investigation—and then search the hard drive for ASCII or Unicode strings that contain any of those keywords. This allows the investigator to pick out relevant sectors of the hard drive, even if the filesystem or partition table has been mangled, and find needles in the haystack.

Similarly, network forensic analysts can compile a list of “dirty values” and search flow record data to pick out relevant entries. When conducting statistical flow analysis, “dirty values” aren’t usually words—they are more likely to be suspicious IP addresses, ports, dates, and times. However, the concept is the same. As you conduct your analysis, you will often find that it is helpful to maintain an updated list of suspicious values that you collect as you move forward.

5.5.1.4 Activity Pattern Matching

Network flow record data, when aggregated, represents complex behavior that is often predictable and can be analyzed mathematically. We have previously compared network flows to traffic on physical roads. As with physical-world phenomena network flows contain patterns that can be empirically measured, mathematically described, and logically analyzed.

The biggest challenge for forensic investigators and anyone who analyzes flow record data is the absence of large publicly available data sets for research and comparison. However, within an organization, it is entirely possible to empirically analyze day-to-day traffic and build statistical models of normal behavior.

In this section, we briefly describe the basic elements of flow record analysis and present a high-level discussion of how to identify and analyze activity patterns that are commonly useful for network forensic investigators.

Elements

Flow record data can be customized, but common elements include source and destination IP address, source and destination ports, protocol and flags, directionality of flow, and volume of data transferred.

• IP address The source and destination IP addresses are great clues that reveal a lot about the cause and purpose of a flow. Consider whether the addresses are on an internal network or Internet-exposed, countries of origin, what companies they are registered to, and other factors.

• Ports Although ports are not required to correspond with particular services, much of the time they do correspond with assigned or well-known ports linked to specific applications or services. Port numbers can also indicate whether a system is port scanning or being scanned and help you identify malicious activity.

• Protocol and Flags Layer 3 and 4 protocols are often tracked in flow record data. These can indicate whether connections were completed and help you tell the difference between connection attempts that were denied by firewalls, successful port scans, successful data transfers, and more. They can also help you make educated guesses as to the content and purpose of a flow.

• Directionality The directionality of the flows are crucial. For example, directionality can indicate whether proprietary data has been leaked or a malicious program was downloaded. Taken in aggregate, the directionality of data transfers can allow you to tell the difference between web-surfing activity and web-serving activity.

• Volume of data transferred The volume of data transferred can help indicate the type of activity and whether or not higher-layer data transfer attempts were successful. For example, many small TCP packets may be indicative of port scanning, whereas larger packets can indicate file exportation. The distribution of the data transferred over time matters; a large volume of data transferred in a very short period of time is usually caused by something different than the same amount of data transferred over a very long period of time.

Simple Patterns

Even novice forensic investigators can identify simple patterns by eye in flow traffic and often logically discern their meanings. Here are a few examples of simple patterns relating to source and destination IP addresses:

• “Many-to-one” IP addresses If you see many IP addresses sending large volumes of traffic to one destination IP address, this may be indicative of a:

– Distributed denial-of-service attack against the destination IP address

– Syslog server

– “Drop box” data repository on the destination IP address

– Email server (at destination)

• One to many IP addresses If you see one IP address sending large volumes of traffic to many IP addresses, this may be indicative of a:

– Web server

– Email server (at source)

– SPAM bot

– Warez server

– Network port scanning

• Many to many IP addresses Many IP addresses sending distributed traffic to many destinations can be indicative of:

– Peer-to-peer filesharing traffic

– Widespread virus infection

• One to one IP address Traffic from one IP address targeted at one other IP address can indicate:

– Targeted attack against a specific system

– Routine server communications

Of course, context is key. Specific ports, protocols, timing information, and other elements can help to narrow down the meaning of the flow record data even further.

Complex Patterns

“Fingerprinting” is the process of matching complex flow record patterns to specific activities. When fingerprinting traffic, we examine multiple elements and their context and develop a hypothesis of the cause of the behavior.

For example, a TCP SYN port scan might have the following characteristics:

• One source IP address

• One or more destination IP addresses

• Destination port numbers increasing incrementally

• Volume of packets surpassing a specified value within a period of time

• TCP protocol

• Outbound protocol flags set to “SYN” (no full TCP connections)

It’s even possible to identify specifically which tool created the port scan, using flow record fingerprinting, since different port scanning tools have different algorithms for selecting source and destination port numbers, TCP window sizes, timing of packets, and other characteristics.

Statistical flow record analysis is still in its infancy. Over time, as this field matures, analysts will develop better tools and mechanisms for identifying the causes of network activity. In the meantime, forensic investigators should at the very least be familiar with the basic elements and simple patterns common in flow record analysis and refer to any baselines that exist within the environment under investigation.

5.5.2 Flow Record Analysis Tools

Many flow record analysis tools have emerged over the years, designed for different purposes and with varying levels of maturity. In this section, we provide you with a brief overview of the analysis capabilities included in selected widely used flow record analysis tools: flow-tools, SiLK, Argus, FlowTraq, nfdump/NfSen. There are many other tools out there, and more are released every day. We encourage you to do your own research to identify tools that are most appropriate for your customized investigative toolkit.

Notably, some of the most popular tools from the past decade, such as flow-tools, do not natively support the IPFIX/NetFlow v9 format, which limits both collection and analysis. Furthermore, some versions of tools do not produce accurate output when data generated using proprietary formats, such as Cisco NetFlow v9, is used as input.

Throughout this book, we generally present open-source tools that are widely accessible. However, developers of open-source tools often have limited access to expensive commercial network equipment that generates flow record data in proprietary formats. In some cases, such as nfdump, specific branches of development have forked from the main project in order to incorporate input from vendors. As a result, if you are analyzing flow record data that is stored in a proprietary format such as Cisco’s NetFlow v9, be sure to carefully research whether your analysis tools, and the specific version you are using, fully support the format you wish to analyze.

5.5.2.1 SiLK

The SiLK suite includes powerful flow export data analysis capabilties. Here is a quick summary of some of the more important analysis tools in the SilK suite.

• rwfilter One of SiLK’s core tools, rwfilter, is designed to extract flows of interest from a particular data repository, filter them by time and category, and then “partition” them further by protocol attributes. It supports a rich syntax that is generally as functional as the BPF, though different.³³

33. “The SiLK Reference Guide (SiLK-2.4.5),” February 25, 2011, http://tools.netsa.cert.org/silk/reference-guide.html.

Initial flow selection typically includes date and time ranges, sensor class (by default all sensors), and a flow type. The SiLK toolkit defines six basic “types” of traffic, although not so much by protocol or layer: in, inweb, innull, out, outweb, and outnull, which correspond to successful versus unsuccessful flow attempts in either direction, further sorted by TCP port 80 versus everything else. (Of course the “successful vs. unsuccessful” part presumes that the reporting sensor is itself a filtering device, which is not uncommon.)

Rwfilter can be configured to output statistical summaries, or simply output detailed results to “stdout,” suitable for input to most of the rest of the filtering and manipulation tools in the SiLK suite.

• rwstats, rwcount, rwcut, rwuniq, et al. SiLK includes an arsenal of basic flow export data manipulation utilities. Rwstats produces statistical aggregations based on the protocol fields specified. Rwcount counts packets and bytes. Rwcut selects the fields that rwuniq can help you sort on.³⁴

34. Timothy Shimeall, “Using SiLK for Network Traffic Analysis” (Software Engineering Institute: Carnegie Mellon, September 2010), http://tools.netsa.cert.org/silk/analysis-handbook.pdf.

• rwidsquery This is a real gem. Let’s suppose there is a Snort sensor somewhere in the environment, most likely on a critical segment. Perhaps some unusual alerts have been detected on that heavily instrumented segment, and there’s concern the activity may have spread to other internal segments where only flow record data is being collected (as opposed to full packet captures). Rwidsquery can be fed either a Snort rule file, or an alert file, and figures out what flows from its input would match the rule or alert, and writes an rwfilter invocation to produce the flows that match.

• rwpmatch This is essentially a libpcap-based program that reads in SiLK-formatted flow metadata and an input packet source and saves out just the packets that match the flow metadata. So if you just happen to have a full packet capture in addition to flow records, and your flow metadata turns up something interesting, you can extract those packets with precision for further inspection.

• Advanced SiLK In addition to the chainable command-line suite, a Python interpreter, “PySiLK,” is available that implements the SiLK functionality by exposing it through a Python API.³⁵ The nice folks at NetSA have provided a “Tooltips” wiki so the user community can share experience and grow better faster.³⁶

35. “PySiLK Reference Guide (SiLK-2.4.5),” February 25, 2011, http://tools.netsa.cert.org/silk/pysilk.pdf.

36. Peter Rush, “NetSA Tools Wiki,” CERT, April 9, 2008, https://tools.netsa.cert.org/confluence/display/tt/Tooltips.

5.5.2.2 flow-tools

The flow-tools suite includes a variety of flow export data collection, storage, processing, and sending tools, including a few tools that are particularly useful for forensic analysis. The “flow-report” tool creates ASCII-readable text reports based on stored flow data. Report contents can be customized by the user through the configuration file, and then sent as input to graphing or analysis programs. The “flow-nfilter” program allows users to filter flow export data based on “primitives,” which are specific to flow-tools.

For forensic investigators, “flow-dscan” is a particularly useful utility, designed to identify suspicious traffic based on flow export data. It includes features for identifying port scanning, host scanning, and denial-of-service attacks.

5.5.2.3 Argus Client Tools

The Argus suite includes a variety of specialized utlities with powerful analysis capabilties. These include, among others:³⁷

37. “Argus—NSMWiki,” Sguil, 2011, http://nsmwiki.org/index.php?title=Argus#St.C3.A9phane_Peters.27s_Cheat_sheet.

• ra—Argus’ basic tool for reading, filtering, and printing Argus data. As with all other Argus client programs, ra can read input from the network or the filesystem, in Argus or Netflow 1-8 format. It includes support for BPF-style filter expressions. Ra allows the user to specify fields for printing, select specific records for processing, match regular expressions, and more.

• racluster—Cluster flow export data based on user-specified criteria. This is very helpful for printing summaries of flow record data.

• rasort—Sort flow data based on user-specified criteria, such as source or destination IP address, time, TTL, flow duration, and more.

• ragrep—Powerful regular expression and pattern matching, based on the GNU “grep” utility.

• rahisto—Generate frequency distribution tables for user-selected metrics such as flow record duration, source and destination port number, bytes transferred, packet count, bits per second, and more.³⁸

38. Carter Bullard, “Argus Development Mailing List—Real Time Flow Monitor (RTFM) for Comprehensive IP Network Traffic Auditing,” November 21, 2010, http://blog.gmane.org/gmane.network.argus/month=20061121.

• ragraph—Create visual plots based on user-specified fields of interest, such as bytes, packet counts, average duration, IP address, ports, and more. Ragraph includes a variety of tools that allow users to customize the graph appearance.

5.5.2.4 FlowTraq

FlowTraq is a commercial flow record analysis tool developed by ProQueSys. Although there are many commercial tools out there for processing flow record data, FlowTraq is a nice option for several reasons. First, it supports a very wide variety of input formats, including NetFlow v9, IPFIX, JFlow, and many others. It can also sniff traffic directly and generate flow records. Once collected, FlowTraq allows users to filter, search, sort, and produce reports based on flow records. In addition, you can specify patterns to generate alerts. FlowTraq supports a variety of operating systems (Windows, Linux, Mac, and Solaris), and is designed and marketed for forensics and incident response (among other purposes).⁴⁰ Figure 5-1 shows an example screenshot of FlowTraq, using data from their online demo.⁴¹

Figure 5-1 A screenshot of the FlowTraq commercial flow record analysis GUI, created using ProQueSys’ online demo.³⁹

39. “Online Demo,” ProQueSys, 2010, http://www.proquesys.com/corporate/flowtraq-demo.

40. “Features,” ProQueSys, 2010, http://www.proquesys.com/corporate/product/features.

41. “Online Demo,” ProQueSys, 2010, http://www.proquesys.com/corporate/flowtraq-demo.

5.5.2.5 nfdump/NfSen

The “nfdump” utility (part of the nfdump suite) is designed to read flow record data, analyze it, and produce customized output. It offers users powerful analysis features, including:⁴²

42. “NFDUMP.”

• Aggregate flow record fields by specific fields

• Limit by time range

• Generate statistics about IP addresses, interfaces, ports, and much more

• Anonymize IP addresses

• Customize output format

• BPF-style filters

“NfSen” (“Netflow Sensor”) provides a graphical, web-based interface for the nfdump suite. It is an open-source tool written in Perl and PHP, designed to run on Linux and POSIX-based operating systems.⁴³

43. Peter Haag, “nfsen,” http://sourceforge.net/projects/nfsen/.

5.5.2.6 EtherApe

EtherApe is an open-source, libpcap-based graphical tool that visually displays network activity in real time. It reads packet data directly from a network interface or pcap file. (Note that EtherApe does not take flow records as input. We are including it here because it provides a nice high-level visualization of traffic patterns, and therefore may be of interest to the reader.) As shown in Figure 5-2, EtherApe visually displays source and destination IP addresses/hostnames, with corresponding volumes of traffic. The GUI provides an intuitive, high-level view of network activity. Colors indicate traffic protocols such as HTTP, SMB, ICMP, IMAPS, and more. EtherApe also provides sortable traffic summaries, as shown in Figure 5-3 and Figure 5-4.⁴⁴

44. Riccardo Ghetta and Juan Toledo, “EtherApe, A Graphical Network Monitor,” May 31, 2011, http://etherape.sourceforge.net/.

Figure 5-2 The main EtherApe screen.

Figure 5-3 The EtherApe “Protocols” window.

Figure 5-4 The EtherApe “Nodes” window.

5.6 Conclusion

Statistical flow record analysis is becoming increasingly important for forensic analysis, as the rate of network traffic generated and the volume of data stored far outpaces the rate at which investigators can process and analyze full packet captures. Although flow records were originally generated for the purposes of monitoring and improving network performance, in recent years investigators have begun to recognize that they are excellent sources of network-based forensic evidence as well.

A variety of sensor, collector, aggregation, and analysis tools exist, ranging from proprietary to free and open-source tools. One of the biggest challenges forensic investigators face is ensuring that the formats used by sensors and collectors are compatible with the analysis tools chosen for the investigation. As the industry matures, there have been movements to standardize flow record exportation formats, but support for flow record export formats is still fragmented and many older tools exist that rely on older protocols. Forensic investigators also need to consider the placement and type of flow sensors and collectors throughout the network environment in order to determine whether the evidence needed is already being collected, or whether it is appropriate to configure additional flow record collection.

Ultimately, statistical flow record analysis is a very powerful field of study that will only grow in importance over the next decades. As flow record processing tools and techniques mature, investigators can expect to have increasingly powerful flow record analysis options at our fingertips.

5.7 Case Study: The Curious Mr. X

5.7.1 Analysis: First Steps

Let’s begin by using nfdump to browse flow records relating to our known attacker system, 172.30.1.77, beginning at the time of interest (2011-04-27 12:51:46.130):

$ nfdump -R cisco-asa-nfcapd/ 'host 172.30.1.77'
Date flow start       Proto  Src IP Addr:Port X-Src IP Addr:Port    Dst IP
    Addr:Port   Event  XEvent Bytes
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:135   DENIED   I-ACL  0
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:139   DENIED   I-ACL  0
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:256   DENIED   I-ACL  0
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:5900  DENIED   I-ACL  0
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:3389  DENIED   I-ACL  0
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:80    DENIED   I-ACL  0
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:995   DENIED   I-ACL  0
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:143   DENIED   I-ACL  0
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:1720  DENIED   I-ACL  0
2011-04-27 12:51:46.130 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:1025  DENIED   I-ACL  0

Notice that the source port on the attacker system remains the same, even as it makes fast connection attempts to different destination ports on the targeted system, 172.30.1.231. This is common behavior for an automated port scanner.

Using nfdump, we can determine which ports the attacker may have found open on the targeted system, 172.30.1.231. Notice that the firewall “DENIED” traffic to most targeted ports. The flows that were not DENIED by the firewall actually reached the targeted system and may have resulted in a response. In this case, the only port that did not result in a DENIED response from the firewall was TCP port 22 on 172.30.1.231, as shown below:

$ nfdump -R cisco-asa-nfcapd/ 'host 172.30.1.77'
Date flow start       Proto  Src IP Addr:Port X-Src IP Addr:Port    Dst IP
    Addr:Port  Event  XEvent Bytes
...
2011-04-27 12:51:47.330 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:443  DENIED   I-ACL    0
2011-04-27 12:51:47.330 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:8080 DENIED   I-ACL    0
2011-04-27 12:51:47.330 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:22   CREATE  Ignore    0
2011-04-27 12:51:47.340 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:22   DELETE  Deleted   0
2011-04-27 12:51:47.340 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:554  DENIED   I-ACL    0
2011-04-27 12:51:47.340 TCP 172.30.1.77:44197  172.30.1.77:44197 ->
    172.30.1.231:21   DENIED   I-ACL    0
...

5.7.2 External Attacker and Port 22 Traffic

Was there any more traffic on port 22 (TCP) relating to the attacker? In the nfdump command below, we filter the flow data to pull out only records pertaining to the attacker system and TCP port 22. Beginning at 2011-04-27 12:52:30.111, we see a series of connection attempts on port 22 of size 3,755 bytes. These occur in rapid succession (roughly every 6 seconds) until 2011-04-27 13:00:45.452.

$ nfdump -R cisco-asa-nfcapd/ 'host 172.30.1.77 and port 22'
Date flow start       Proto  Src IP Addr:Port X-Src IP Addr:Port    Dst IP
    Addr:Port   Event  XEvent Bytes
...
2011-04-27 13:00:16.672 TCP 172.30.1.77:56356  172.30.1.77:56356 ->
    172.30.1.231:22  DELETE Deleted  3755
2011-04-27 13:00:22.722 TCP 172.30.1.77:56358  172.30.1.77:56358 ->
    172.30.1.231:22  CREATE  Ignore     0
2011-04-27 13:00:22.732 TCP 172.30.1.77:56357  172.30.1.77:56357 ->
    172.30.1.231:22  DELETE Deleted  3755
2011-04-27 13:00:29.532 TCP 172.30.1.77:56359  172.30.1.77:56359 ->
    172.30.1.231:22  CREATE  Ignore     0
2011-04-27 13:00:29.532 TCP 172.30.1.77:56358  172.30.1.77:56358 ->
    172.30.1.231:22  DELETE Deleted  3755
2011-04-27 13:00:35.612 TCP 172.30.1.77:56360  172.30.1.77:56360 ->
    172.30.1.231:22  CREATE  Ignore     0
2011-04-27 13:00:35.612 TCP 172.30.1.77:56359  172.30.1.77:56359 ->
    172.30.1.231:22  DELETE Deleted  3755
2011-04-27 13:00:41.962 TCP 172.30.1.77:56361  172.30.1.77:56361 ->
    172.30.1.231:22  CREATE  Ignore     0
2011-04-27 13:00:41.962 TCP 172.30.1.77:56360  172.30.1.77:56360 ->
    172.30.1.231:22  DELETE Deleted  3603
2011-04-27 13:00:45.452 TCP 172.30.1.77:56361  172.30.1.77:56361 ->
    172.30.1.231:22  DELETE Deleted  3335
2011-04-27 13:01:00.133 TCP 172.30.1.77:56362  172.30.1.77:56362 ->
    172.30.1.231:22  CREATE  Ignore     0

What does this pattern mean? First of all, by default TCP port 22 is assigned to SSH protocol by IANA.⁴⁵ SSH is a remote access service that is commonly targeted for brute-force password-guessing attacks when attackers find it exposed. Brute-force password-guessing attacks are typically automated and produce connection attempts at regular intervals with approximately the same, small amount of data transferred each time. While it is possible to set up the SSH server on a different port (and many people do, to obfuscate the attack surface), the pattern of this traffic (both frequency and port numbers) supports the hypothesis that this is a brute-force password-guessing attack against an externally accessible SSH service.

45. “Port Numbers,” July 7, 2011, http://www.iana.org/assignments/port-numbers.

As you can see, at 2011-04-27 13:00:41.962, the bytes transferred suddenly changed. This was followed by one more very quick connection, and then at 2011-04-27 13:01:00.133 a flow was created between the attacker (172.30.1.77) and the target (172.30.1.231) on port TCP 22, which did not expire within the time frame of the recorded evidence. The length of this flow implies that the connection was probably successful. Of course, we will want to follow up by examining the internal flow record data and the authentication logs, if available.

From the timing, port number, and data transfer sizes of the flows, we hypothesize that these flow records indicate an automated brute-force password-guessing attack on an SSH server running on the targeted system, 172.30.1.231. After approximately 8 minutes, this attack probably resulted in a successful SSH login.

Now let’s turn our attention to the internal Argus flow record data. Searching for traffic relating to the attacker, 172.30.1.77 and port 22, we see:

$ ra -z -nn -r argus-collector.ra - 'src host 172.30.1.77 and port 22'
         StartTime  Proto SrcAddr  Sport Dir DstAddr  Dport TotPkts State
 04-27-11 12:51:55  6  172.30.1.77.44197 ->  10.30.30.20.22   3       sSR
 04-27-11 12:51:57  6  172.30.1.77.44208 ->  10.30.30.20.22   3       sSR
 04-27-11 12:51:58  6  172.30.1.77.44209 ->  10.30.30.20.22   3       sSR
 04-27-11 12:52:38  6  172.30.1.77.54348 ->  10.30.30.20.22  43     sSEfF
 04-27-11 12:52:44  6  172.30.1.77.54349 ->  10.30.30.20.22  42     sSEfF
 04-27-11 12:52:50  6  172.30.1.77.54350 ->  10.30.30.20.22  42     sSEfF
 04-27-11 12:52:56  6  172.30.1.77.54351 ->  10.30.30.20.22  42     sSEfF
 04-27-11 12:53:02  6  172.30.1.77.54352 ->  10.30.30.20.22  42     sSEfF
 04-27-11 12:53:09  6  172.30.1.77.54353 ->  10.30.30.20.22  42     sSEfF
 04-27-11 12:53:15  6  172.30.1.77.54354 ->  10.30.30.20.22  42     sSEfF
...

Recall that the internal NAT-ed address 10.30.30.20 corresponds with the external address 172.30.1.231 (they are the same server). Accounting for 8 seconds of time skew between the Cisco ASA and the Argus server, the Argus flow records show the same attempts to connect to TCP port 22 on the same server during the same time frame—correlating evidence. Using the ra utility’s -z flag, we can also see the TCP state changes in the flow records, as described below:

$ man ra
...
       -z  Modify status field to represent TCP state changes. The values
           of the status field when this is enabled are:
             's' - Syn Transmitted
             'S' - Syn Acknowledged
             'E' - TCP Established
             'f' - Fin Transmitted  (FIN Wait State 1)
             'F' - Fin Acknowledged (FIN Wait State 2)
             'R' - TCP Reset

From this information, we can see that the attacker first initiated a connection to 10.30.30.20:22 three times (sent a TCP SYN, received a SYN-ACK, sent a RST). The connection was aborted by the attacker before the TCP handshake was completed. This matches the behavior of a port scanner testing to see if port 22 is open using a TCP SYN scan (notably, the default nmap behavior when run with administrative privileges). Having received a TCP SYN/ACK back from the server, the attacker has verified that indeed the port is open:

$ ra -z -nn -r argus - collector.ra  - 'src host 172.30.1.77 and port 22'
         StartTime  Proto SrcAddr  Sport Dir DstAddr  Dport  TotPkts State
 04-27-11 12:51:55  6  172.30.1.77.44197 ->  10.30.30.20.22  3       sSR
 04-27-11 12:51:57  6  172.30.1.77.44208 ->  10.30.30.20.22  3       sSR
 04-27-11 12:51:58  6  172.30.1.77.44209 ->  10.30.30.20.22  3       sSR

Next, we see the series of short connections to port 22 approximately every 6 seconds. In the flows below, the attacker sent a TCP SYN packet, received a SYN ACK packet, established a full TCP handshake, sent a FIN packet, and received a FIN packet. This is a full, successful Layer 4 TCP communication, over which application data could have been sent. (Note that Argus, unlike nfdump, displays the number of bytes transferred including the lower-layer Ethernet frame, IP, and TCP headers.)

$ ra -z -nn -r argus - collector.ra  - 'src host 172.30.1.77 and port 22'
         StartTime  Proto  SrcAddr Sport Dir DstAddr Dport TotBytes State
...
 04-27-11 12:52:38  6  172.30.1.77.54348 ->  10.30.30.20.22  6609   sSEfF
 04-27-11 12:52:44  6  172.30.1.77.54349 ->  10.30.30.20.22  6543   sSEfF
 04-27-11 12:52:50  6  172.30.1.77.54350 ->  10.30.30.20.22  6543   sSEfF
 04-27-11 12:52:56  6  172.30.1.77.54351 ->  10.30.30.20.22  6543   sSEfF
 04-27-11 12:53:02  6  172.30.1.77.54352 ->  10.30.30.20.22  6543   sSEfF
 04-27-11 12:53:09  6  172.30.1.77.54353 ->  10.30.30.20.22  6543   sSEfF
 04-27-11 12:53:15  6  172.30.1.77.54354 ->  10.30.30.20.22  6543   sSEfF
...

Finally, starting at 04-27-11 13:00:44 (Argus time), we see two smaller, short, completed connections followed by a series of longer established TCP connections on port 22, with no FIN packets logged as part of the flow. These established TCP connections between 172.30.1.77 and 10.30.30.20:22 continue from 04-27-11 13:01:08 through 04-27-11 13:15:55, more than 15 minutes:

$ ra -z -n -r argus-collector.ra  - 'src host 172.30.1.77 and port 22'
         StartTime  Proto  SrcAddr  Sport Dir DstAddr  Dport  TotBytes State
...
 04-27-11 13:00:31  tcp 172.30.1.77.56358 ->  10.30.30.20.22      6543 sSEfF
 04-27-11 13:00:38  tcp 172.30.1.77.56359 ->  10.30.30.20.22      6543 sSEfF
 04-27-11 13:00:44  tcp 172.30.1.77.56360 ->  10.30.30.20.22      6127 sSEfF
 04-27-11 13:00:50  tcp 172.30.1.77.56361 ->  10.30.30.20.22      5661 sSEfF
 04-27-11 13:01:08  tcp 172.30.1.77.56362 ->  10.30.30.20.22    147296   sSE
 04-27-11 13:02:11  tcp 172.30.1.77.56362 ->  10.30.30.20.22     77440   sSE
 04-27-11 13:03:11  tcp 172.30.1.77.56362 ->  10.30.30.20.22     24508   sSE
 04-27-11 13:04:14  tcp 172.30.1.77.56362 ->  10.30.30.20.22    447368   sSE
 04-27-11 13:05:24  tcp 172.30.1.77.56362 ->  10.30.30.20.22    228032   sSE
 04-27-11 13:06:24  tcp 172.30.1.77.56362 ->  10.30.30.20.22     47906   sSE
 04-27-11 13:07:55  tcp 172.30.1.77.56362 ->  10.30.30.20.22      1108   sSE

$ ra -z -n -r argus-collector.ra  - 'src host 172.30.1.77 and port 22'
         StartTime  Proto  SrcAddr  Sport Dir DstAddr  Dport  TotBytes State
...
 04-27-11 13:08:55  tcp 172.30.1.77.56362 ->  10.30.30.20.22      1124   sSE
 04-27-11 13:09:55  tcp 172.30.1.77.56362 ->  10.30.30.20.22      1092   sSE
 04-27-11 13:10:55  tcp 172.30.1.77.56362 ->  10.30.30.20.22     80742   sSE
 04-27-11 13:11:55  tcp 172.30.1.77.56362 ->  10.30.30.20.22      1124   sSE
 04-27-11 13:12:55  tcp 172.30.1.77.56362 ->  10.30.30.20.22      1124   sSE
 04-27-11 13:13:55  tcp 172.30.1.77.56362 ->  10.30.30.20.22      1092   sSE
 04-27-11 13:14:55  tcp 172.30.1.77.56362 ->  10.30.30.20.22      1108   sSE
 04-27-11 13:15:55  tcp 172.30.1.77.56362 ->  10.30.30.20.22      1108   sSE

The sudden halt of short, regular connection attempts, followed by larger, longer flow records with more data associated, fits with the pattern of a brute-force password-guessing attack that was successful and was subsequently followed by an extended SSH login through at least 13:15:55. This pattern is illustrated using the Argus utility “ragraph” in Figure 5-5. The following command was used to generate the graph in Figure 5-5:

$ ragraph bytes -M 1s  -r argus-collector.ra -t 2011/04/27.12:52+10m
  -title "Traffic Snippet from 172.30.1.77 on Port 22 (TCP)"
  - 'src host 172.30.1.77 and dst host 10.30.30.20 and port 22'

Figure 5-5 A graph of Argus flow record data from the attacker (172.30.1.77) to a victim server in the DMZ (10.30.30.20) on port 22. Notice the 8-minute automated brute-force password-guessing attack, followed by a short break, and then followed by a higher-bandwidth exchange of data (implying that the attack was probably successful).

5.7.3 The DMZ Victim—10.30.30.20 (aka 172.30.1.231)

Now let’s get more information about the activities of 10.30.30.20, the targeted DMZ system.

The first noticeably different SSH connection from 172.30.1.77 to 10.30.30.20 occurred at 04-27-11 13:01:08 (in the Argus flow record data). The size of the data transferred during this flow was substantially larger than previous attempts, likely indicating a successful connection. More than a minute later, we see another connection on port 22. The slower timing of these connections also implies a possible manual connection after the automated brute-force tool completed.

Subsequently, we see 10.30.30.20 communicating over port 80 to an external IP address, 91.189.92.166.80. Then, at 13:03:31, it began sending TCP SYN packets to internal systems on ports 80 and 443. At 13:03:44, these packets were targeted at sequential, incrementing IP addresses—very common for an automated port sweep. Notice the sequential destination IP addresses all on the same ports (80 and later 443). The system 10.30.30.20 continued to send TCP SYN segments to ports 80 and 443 on the internal networks until 13:03:49. In most cases, the targeted systems did not respond with a TCP SYN/ACK packet—or any packet at all—as shown in the “State” column:

$ ra -z -nnr argus - collector.ra - 'host 10.30.30.20 and not udp and not port
    514 and not port 53'
      StartTime  Proto    SrcAddr  Sport  Dir   DstAddr  Dport TotPkts  State
...
 04-27-11 13:02:57  6  10.30.30.20.36339  ->  91.189.92.166.80     508   SEfF
 04-27-11 13:02:58  6  10.30.30.20.35194  ->  91.189.92.161.80    1221 sSEfF
 04-27-11 13:03:11  6  172.30.1.77.56362  ->    10.30.30.20.22     214   sSE
 04-27-11 13:03:31  6  10.30.30.20.60833  ->    10.30.30.10.80       1     s
 04-27-11 13:03:33  6  10.30.30.20.32907  ->    10.30.30.10.80       1     s
 04-27-11 13:03:33  6  10.30.30.20.53778  ->   10.30.30.10.443       1     s
 04-27-11 13:03:33  6  10.30.30.20.53808  ->   10.30.30.10.443       1     s
 04-27-11 13:03:44  6  10.30.30.20.41339  ->   192.168.30.1.80       1     s
 04-27-11 13:03:44  6  10.30.30.20.42486  ->   192.168.30.2.80       1     s
 04-27-11 13:03:44  6  10.30.30.20.48181  ->   192.168.30.3.80       1     s
 04-27-11 13:03:44  6  10.30.30.20.56116  ->   192.168.30.4.80       1     s
 04-27-11 13:03:44  6  10.30.30.20.58905  ->   192.168.30.5.80       1     s
 04-27-11 13:03:44  6  10.30.30.20.37560  ->   192.168.30.6.80       1     s
 04-27-11 13:03:44  6  10.30.30.20.36524  ->   192.168.30.7.80       1     s
 04-27-11 13:03:44  6  10.30.30.20.35544  ->   192.168.30.8.80       1     s
 04-27-11 13:03:44  6  10.30.30.20.43568  ->   192.168.30.9.80       1     s
...
 04-27-11 13:03:49  6  10.30.30.20.48728  ->  192.168.30.86.443      1     s
 04-27-11 13:03:49  6  10.30.30.20.55470  -> 192.168.30.116.443      1     s
 04-27-11 13:03:49  6  10.30.30.20.48601  -> 192.168.30.119.443      1     s
 04-27-11 13:03:49  6  10.30.30.20.58087  ->  192.168.30.87.443      1     s

Which systems did respond to the port sweep? We can easily pick these needles out of the haystack using a filter that searches for flow records where the number of packets sent from the targeted systems back to the port scanner was greater than zero:

$ ra -z -t 2011/04/27.13:03:31+18s -nnr argus - collector.ra -s stime saddr
    sport dir daddr dport spkts dpkts state - 'host 10.30.30.20 and port (80
    or 443) and tcp and dst pkts gt 0'
        StartTime    SrcAddr  Sport Dir   DstAddr  Dport SrcPkt DstPkt State
04-27-11 13:03:47  10.30.30.20.39307 ->  192.168.30.30.80  2      2       sR
04-27-11 13:03:47  10.30.30.20.53512 ->  192.168.30.90.80  3      3       sR

From this, we see that the two systems 192.168.30.30 and 192.168.30.90 sent TCP RST packets back to 10.30.30.20. This indicates that port 80 was not open on those systems, but the systems were in fact alive.

At 04-27-11 13:03:49, 10.30.30.20 changed behavior and began sending SYN packets exclusively to a range of ports on the two systems that had responded, 192.168.30.30 and 192.168.30.90. The pattern lasted only through 13:03:50 (2 seconds). This behavior is indicative of a port scan against the two hosts, 192.168.30.30 and 192.168.30.90, as shown below.

$ ra -z -nnr argus - collector.ra  - 'host 10.30.30.20'
...
         StartTime  Proto SrcAddr  Sport  Dir    DstAddr  Dport TotBytes
             State
 04-27-11 13:03:49  6  10.30.30.20.52140  ->  192.168.30.90.143     262     sR
 04-27-11 13:03:49  6  10.30.30.20.45717  ->  192.168.30.30.143     268     sR
 04-27-11 13:03:49  6  10.30.30.20.60813  ->  192.168.30.90.1723    262     sR
 04-27-11 13:03:49  6  10.30.30.20.39655  ->  192.168.30.30.1723    268     sR
 04-27-11 13:03:49  6  10.30.30.20.35499  ->  192.168.30.90.5900    262     sR
 04-27-11 13:03:49  6  10.30.30.20.55132  ->  192.168.30.30.5900    268     sR
 04-27-11 13:03:49  6  10.30.30.20.43935  ->  192.168.30.90.8888    262     sR
 04-27-11 13:03:49  6  10.30.30.20.49207  ->  192.168.30.30.8888    268     sR
 04-27-11 13:03:49  6  10.30.30.20.59231  ->  192.168.30.90.8080    262     sR
 04-27-11 13:03:49  6  10.30.30.20.58646  ->  192.168.30.30.8080    268     sR
 04-27-11 13:03:49  6  10.30.30.20.58594  ->  192.168.30.90.3389    262     sR
 04-27-11 13:03:49  6  10.30.30.20.35855  ->  192.168.30.30.3389    268     sR
...
 04-27-11 13:03:50  6  10.30.30.20.45320  ->  192.168.30.90.24800   262     sR
 04-27-11 13:03:50  6  10.30.30.20.46080  ->  192.168.30.30.24800   268     sR
 04-27-11 13:03:50  6  10.30.30.20.53337  ->  192.168.30.90.9207    262     sR
 04-27-11 13:03:50  6  10.30.30.20.58434  ->  192.168.30.30.9207    268     sR
...

We can use a quick Bash shell command to sort and count the destination ports targeted. As you can see below, 10.30.30.20 targeted exactly 1,000 different port numbers. (Experienced penetration testers will recall that by default Nmap, the world’s most popular port scanner, “scans the most common 1,000 ports for each protocol.”⁴⁶)

46. Gordon “Fyodor” Lyon, “Port Specification and Scan Order,” NMAP, 2011, http://nmap.org/book/man-port-specification.html.

$ ra -z -t 2011/04/27.13:03:49+2s -nnr argus - collector.ra  -s dport - 'host
    10.30.30.20 ' | grep -v Dport | sort -u | wc -l
1000

Which ports did the attacker find open on the targeted internal systems, 192.168.30.30 and 192.168.30.90? By filtering for TCP SYN/ACK packets, we can see that the attacker found 192.168.30.90:22 (TCP), 192.168.30.30:22 (TCP), and 192.168.30.30:514 (TCP) open, as shown below:

$ ra -z -t 2011/04/27.13:03:49+2 s -nnr argus - collector.ra   - 'host
    10.30.30.20 and dst host (192.168.30.30 or 192.168.30.90) and  synack'
         StartTime  Proto SrcAddr  Sport  Dir   DstAddr  Dport TotBytes State
 04-27-11 13:03:49  6  10.30.30.20.46692  ->  192.168.30.90.22      560  sSER
 04-27-11 13:03:49  6  10.30.30.20.36307  ->  192.168.30.30.22      560  sSER
 04-27-11 13:03:49  6  10.30.30.20.39370  -> 192.168.30.30.514      560  sSER

Next, between 13:04:09 through 13:04:14, 10.30.30.20 began sending TCP SYN packets to sequential IP addresses on the 192.168.30.0/24 network, but only targeted at TCP port 3389, as shown below:

$ ra -z -t 2011/04/27.13:04:09+6 s -nnr argus - collector .ra - 'host
    10.30.30.20 and net 192.168.30.0/24 '
         StartTime  Proto SrcAddr  Sport  Dir   DstAddr  Dport TotBytes State
 04-27-11 13:04:09  6  10.30.30.20.56212  ->  192.168.30.1.3389      74     s
 04-27-11 13:04:09  6  10.30.30.20.56255  ->  192.168.30.2.3389      74     s
 04-27-11 13:04:09  6  10.30.30.20.52168  ->  192.168.30.3.3389      74     s
 04-27-11 13:04:09  6  10.30.30.20.45057  ->  192.168.30.4.3389      74     s
 04-27-11 13:04:09  6  10.30.30.20.32898  ->  192.168.30.5.3389      74     s
 04-27-11 13:04:09  6  10.30.30.20.55453  ->  192.168.30.6.3389      74     s
 04-27-11 13:04:09  6  10.30.30.20.48122  ->  192.168.30.7.3389      74     s
 04-27-11 13:04:09  6  10.30.30.20.39936  ->  192.168.30.8.3389      74     s
 04-27-11 13:04:09  6  10.30.30.20.51199  ->  192.168.30.9.3389      74     s
 04-27-11 13:04:09  6  10.30.30.20.40066  ->  192.168.30.10.3389     74     s
...
 04-27-11 13:04:14  6  10.30.30.20.47058  ->  192.168.30.221.3389    74     s
 04-27-11 13:04:14  6  10.30.30.20.47002  ->  192.168.30.232.3389    74     s
 04-27-11 13:04:14  6  10.30.30.20.40560  ->  192.168.30.226.3389    74     s

This is indicative of a port sweep targeting TCP port 3389 (commonly associated with Microsoft’s Remote Desktop Protocol). We can see which systems had ports open by again searching for TCP SYN/ACK responses within that time frame:

$ ra -z -t 2011/04/27.13:04:09+6s -nnr argus - collector.ra  - 'host
    10.30.30.20 and net 192.168.30.0/24 and synack'
         StartTime Proto SrcAddr  Sport Dir     DstAddr  Dport TotBytes State
 04-27-11 13:04:13 6  10.30.30.20.41371 -> 192.168.30.100.3389      560  sSER
 04-27-11 13:04:13 6  10.30.30.20.33786 -> 192.168.30.101.3389      568  sSER
 04-27-11 13:04:13 6  10.30.30.20.57473 -> 192.168.30.102.3389      568  sSER

The attacker would have found that three systems on the 192.168.30.0/24 network have TCP port 3389 open: 192.168.30.100, 192.168.30.101, and 192.168.30.102.

After the port sweep of port 3389 is completed, we see a series of flows from the DMZ system 10.30.30.20 to 192.168.30.101 on port 3389 (RDP), spanning more than 11 minutes:

$ ra -z -nnr argus - collector.ra  - 'host 10.30.30.20 and port 3389'
         StartTime  Proto SrcAddr  Sport Dir    DstAddr  Dport TotBytes State
...
 04-27-11 13:04:32  6  10.30.30.20.34187 -> 192.168.30.101.3389     568  sSER
 04-27-11 13:04:32  6  10.30.30.20.34188 -> 192.168.30.101.3389    1260 sSEfF
 04-27-11 13:04:54  6  10.30.30.20.34189 -> 192.168.30.101.3389  852450   sSE
 04-27-11 13:05:54  6  10.30.30.20.34189 -> 192.168.30.101.3389  285408   sSE
 04-27-11 13:06:54  6  10.30.30.20.34189 -> 192.168.30.101.3389   10032   sSE
 04-27-11 13:07:55  6  10.30.30.20.34189 -> 192.168.30.101.3389    2128   sSE
 04-27-11 13:08:55  6  10.30.30.20.34189 -> 192.168.30.101.3389    2170   sSE
 04-27-11 13:09:55  6  10.30.30.20.34189 -> 192.168.30.101.3389    2116   sSE
 04-27-11 13:10:55  6  10.30.30.20.34189 -> 192.168.30.101.3389  126806   sSE
 04-27-11 13:11:55  6  10.30.30.20.34189 -> 192.168.30.101.3389    2180   sSE
 04-27-11 13:12:55  6  10.30.30.20.34189 -> 192.168.30.101.3389    2164   sSE
 04-27-11 13:13:55  6  10.30.30.20.34189 -> 192.168.30.101.3389    2112   sSE
 04-27-11 13:14:55  6  10.30.30.20.34189 -> 192.168.30.101.3389    2126   sSE
 04-27-11 13:15:55  6  10.30.30.20.34189 -> 192.168.30.101.3389    2140   sSE

Recall that during this same time frame, there were also likely active SSH connections between the external attacker, 172.30.1.77, and the DMZ victim, 10.30.30.20. Based on the flow record data, we have seen that while this was occurring, the DMZ victim system, 10.30.30.20, conducted port scans and port sweeps of the internal ANFRF network, and then subsequently connected to port 3389 (RDP) on an internal system, 192.168.30.101.

5.7.4 The Internal Victim—192.168.30.101

Now let’s filter for traffic relating to 192.168.30.101. In the output below, we again see the internal port scanning traffic, followed by the port 3389 connection. Then at 13:05:33 we see an outbound connection directly from 192.168.30.101 to our attacker system 172.30.1.77 on TCP port 21 (FTP)!

$ ra -z -nn -r argus-collector.ra - 'host 192.168.30.101'
        StartTime Proto    SrcAddr  Sport Dir   DstAddr  Dport TotBytes State
...
 04-27-11 13:03:47 6    10.30.30.20.47537 -> 192.168.30.101.80      148     s
 04-27-11 13:03:47 6    10.30.30.20.57498 -> 192.168.30.101.443     148     s
 04-27-11 13:03:49 6    10.30.30.20.57963 -> 192.168.30.101.443     148     s
 04-27-11 13:03:49 6    10.30.30.20.48054 -> 192.168.30.101.80      148     s
 04-27-11 13:04:04 17 192.168.30.101.1600 -> 192.168.30.30.514    13610   INT
 04-27-11 13:04:13 6    10.30.30.20.33786 -> 192.168.30.101.3389    568  sSER
 04-27-11 13:04:32 6    10.30.30.20.34187 -> 192.168.30.101.3389    568  sSER
 04-27-11 13:04:32 6    10.30.30.20.34188 -> 192.168.30.101.3389   1260 sSEfF
 04-27-11 13:04:54 6    10.30.30.20.34189 -> 192.168.30.101.3389 852450   sSE
 04-27-11 13:05:06 17 192.168.30.101.1600 -> 192.168.30.30.514     4108   INT
 04-27-11 13:05:33 6  192.168.30.101.1603 -> 172.30.1.77.21        1221   sSE
 04-27-11 13:05:54 6    10.30.30.20.34189 -> 192.168.30.101.3389 285408   sSE
 04-27-11 13:06:50 6  192.168.30.101.1603 -> 172.30.1.77.21        1254   sSE
 04-27-11 13:06:50 6       172.30.1.77.20 -> 192.168.30.101.5002    558 sSEfF
...

File Transfer Protocol (FTP) is used for transferring files between systems. This traffic may indicate that data was transferred from our internal system, 192.168.30.101, to the attacker, 172.30.1.77. Filtering for FTP-related traffic (default ports TCP 20/21), we see:

$ ra -z -n -r argus-collector.ra -s stime saddr sport dir daddr dport sbytes
    dbytes state - 'host 192.168.30.101 and port (20 or 21) and tcp'
        StartTime      SrcAddr  Sport Dir  DstAddr  Dport SBytes DBytes State
04-27-11 13:05:33 192.168.30.101.1603 ->      172.30.1.77.21  673   548   sSE
04-27-11 13:06:50 192.168.30.101.1603 ->      172.30.1.77.21  630   624   sSE
04-27-11 13:06:50      172.30.1.77.20 -> 192.168.30.101.5002  348   210 sSEfF
04-27-11 13:07:03      172.30.1.77.20 -> 192.168.30.101.5003  998 16874 sSEfF
04-27-11 13:11:14 192.168.30.101.1603 <?>     172.30.1.77.21  180   128  EfFR

In the port 20 flow at 13:07:03, there were 16,874 bytes of data exported from the internal system, 192.168.30.101, to the external attacker. This is certainly large enough for a decent size file to be exported. What is the risk that data was exfiltrated? High!

Let’s see if we can correlate the internal Argus data relating to FTP with the Cisco ASA flow records. Let’s go back to our original nfdump data. (Remember that the internal address 192.168.30.101 is NAT-ed and corresponds with the external address, 172.30.1.227. These are the same system!) Adjusting for our 8-second time skew, we can indeed see the same connections in the output below. Note that the flow record data from the Cisco ASA shows the Layer 4 payload size (TCP payload size only rather than Ethernet frame size), and so it reports less data transferred than the Argus utility, which includes lower-layer frame and packet headers in its reported data transferred.

It is likely that the last flow record on port 20 indicates an outbound file transfer from 172.30.1.227 (192.168.30.101) to the attacker (172.30.1.77) of a file with a size of 15,872 bytes (or multiple files adding up to that size).

$ nfdump -R cisco-asa-nfcapd/ 'host 172.30.1.77 and (port 20 or port 21)'
Date flow start       Proto    Src IP Addr:Port X-Src IP Addr:Port    Dst IP
    Addr:Port   Event  XEvent Bytes
...
2011-04-27 13:05:24.739 TCP 192.168.30.101:1603  172.30.1.227:1603 ->
    172.30.1.77:21    CREATE  Ignore      0
2011-04-27 13:06:42.070 TCP      172.30.1.77:20     172.30.1.77:20 ->
    172.30.1.227:5002 CREATE  Ignore      0
2011-04-27 13:06:42.081 TCP      172.30.1.77:20     172.30.1.77:20 ->
    172.30.1.227:5002 DELETE Deleted     10
2011-04-27 13:06:55.291 TCP      172.30.1.77:20     172.30.1.77:20 ->
    172.30.1.227:5003 CREATE  Ignore      0
2011-04-27 13:06:55.631 TCP      172.30.1.77:20     172.30.1.77:20 ->
    172.30.1.227:5003 DELETE Deleted  15872
2011-04-27 13:11:05.767 TCP 192.168.30.101:1603  172.30.1.227:1603 ->
    172.30.1.77:21    DELETE Deleted    631

5.7.5 Timeline

Based on our flow record analysis, we can create a working hypothesis of what likely transpired. Of course, we must make some educated guesses in order to interpret the activity we have seen. However, our analysis is strongly supported by the evidence, references, and experience. With that in mind, here is a timeline of events related to the incident at ANFRF April 27, 2011:

[Please note that the times are adjusted to match the internal Argus flow record data, which is 8 seconds ahead of the Cisco ASA.]

• 12:49:33—Flow captures begin.

• 12:51:54—Port scanning begins from 172.30.1.77 (attacker) against 172.30.1.231 (DMZ victim). The attacker likely found that port 22 (TCP) was open on the DMZ victim system.

• 12:52:38—172.30.1.77 begins likely brute-force password-guessing attack against an SSH server on DMZ victim.

• 13:00:45—172.30.1.77 ends likely brute-force password-guessing attack.

• 13:01:08—172.30.1.77 begins extended connection to SSH port on DMZ victim.

• 13:03:31—DMZ victim begins port sweep of internal and DMZ networks on TCP ports 80 and 443. Two systems on the internal network responded: 192.168.30.30 and 192.168.30.90.

• 13:03:49—DMZ victim ends port sweep of internal and DMZ networks on TCP ports 80 and 443.

• 13:03:49—DMZ victim begins port scan of 192.168.30.30 and 192.168.30.90. 1,000 ports were targeted. The attacker found 192.168.30.90:22 (TCP), 192.168.30.30:22 (TCP), and 192.168.30.30:514 (TCP) open.

• 13:03:50—DMZ victim ends port scan of 192.168.30.30 and 192.168.30.90.

• 13:04:09—DMZ victim begins port sweep of internal network, 192.168.30.0/24, on port 3389. Three systems on the 192.168.30.0/24 network appeared to have TCP port 3389 open: 192.168.30.100, 192.168.30.101, and 192.168.30.102.

• 13:04:14—DMZ victim ends port sweep of internal network targeting port 3389.

• 13:04:32—DMZ victim begins a series of connections to 192.168.30.101 on port 3389 (TCP). This port is commonly associated with RDP, a remote connection protocol commonly used on Microsoft Windows systems.

• 13:05:33—192.168.30.101 begins outbound connections on port 21/TCP (FTP) to the attacker, 172.30.1.77.

• 13:07:03—192.168.30.101 conducts a particularly large outbound data transfer to 172.30.1.77, with a likely file size of 15,872 bytes.

• 13:15:55—Last flow record from 172.30.1.77 to DMZ victim (port 22/TCP). Connection still active.

• 13:15:55—Last flow record from DMZ victim to 192.168.30.101 (port 3389/TCP). Connection still active.

• 13:16:09—Flow captures end.

5.7.6 Theory of the Case

Now that we have put together a timeline of events, let’s summarize our theory of the case. Again, this is a working hypothesis strongly supported by the evidence, references, and experience:

• The attacker (172.30.1.77) conducted a port scan of the DMZ victim 172.30.1.231 (10.30.30.20).

• The attacker found that TCP port 22 (SSH) was exposed on the targeted DMZ victim 172.30.1.231.

• The attacker (172.30.1.77) conducted a brute-force password-guessing SSH attack on the DMZ victim, 172.30.1.231 (10.30.30.20). After approximately 8 minutes, this attack was successful.

• The attacker logged into the DMZ victim 172.30.1.231 (10.30.30.20) using SSH and conducted a port scan of the internal network.

• Two systems, 192.168.30.30 and 192.168.30.90, were responsive and had port 22/TCP open.

• From the DMZ victim 172.30.1.231 (10.30.30.20), the attacker also conducted a port sweep of the internal network for open port 3389 (RDP).

• Three systems had port 3389 open: 192.168.30.100, 192.168.30.101, and 192.168.30.102.

• The attacker, pivoting through the DMZ victim 172.30.1.231 (10.30.30.20), logged into 192.168.30.101 via RDP.

• On 192.168.30.101 (172.30.1.227), the attacker used FTP to connect outbound to 172.30.1.77.

• The attacker transferred a file from the internal system 192.168.30.101 (172.30.1.227) to the attacker’s system, 172.30.1.77.

5.7.7 Response to Challenge Questions

Now, let’s answer the investigative questions posed to us at the beginning of the case.

• Identify any compromised systems

The evidence suggests that at least two systems were compromised: the DMZ victim (10.30.30.20) and an internal system (192.168.30.101). The attacker compromised the DMZ victim through a brute-force SSH password-guessing attack, and was able to leverage access to the DMZ in order to access a system on the internal network. The internal system, 192.168.30.101, did not appear to be subjected to a password-guessing attack before the RDP connection, so it may be the case that the attacker was able to reuse the same password in order to log into both systems remotely.

• Determine what the attacker found out about the network architecture

Based on the port scanning activity and responses that we have seen, we can expect that the attacker found out that:

– Firewall rules allow connections via TCP port 22 to the DMZ victim, 172.30.1.231 (10.30.30.20).

– The ANFRF has at least two subnets: 10.30.30.0/24 and 192.168.30.0/24.

– Firewall rules also allow access from the DMZ to various systems on the internal network for TCP ports 22, 80, 443, 514, and 3389.

– FTP traffic (TCP port 20/21) is allowed outbound from the internal network.

• Evaluate the risk of data exfiltration

Based on our analysis, the risk of data exfiltration is high. The flow record data includes strong evidence that an outbound FTP connection was made from an internal system to the external attacker, and that a significant amount of data was transferred over this connection.

5.7.8 Next Steps

Using only flow record data, we have learned a significant amount about the events that likely transpired on April 27, 2011. As forensic investigators, where do we go from here? Although this depends on the resources and goals of the organization initiating the investigation, often the next steps involve a two-pronged attack of gathering additional evidence and containing/eradicating any compromise. In this case, common next steps might include:

• Containment/Eradication What can ANFRF staff do to contain the damage and prevent further compromise? Here are a few options:

– Change all passwords that may have been compromised. This includes any passwords related to the DMZ victim, 10.30.30.20, and the internal system 192.168.30.101. Organizations that “play it safe” may choose to reset all passwords.

– Rebuild the two compromised systems, 10.30.30.20 and 192.168.30.101 (after gathering evidence from them as needed).

– Tighten firewall rules to more strictly limit access from the DMZ to the internal network. Is there really a need for DMZ systems to access internal systems? Limit this access to the greatest extent possible.

– Block outbound connections on TCP ports 20/21 (FTP), and any other ports that are not needed.

– Remove or restrict access to the externally exposed SSH service, if possible.

– Consider using two-factor authentication for external access to the network. Single-factor authentication is risky and leaves the system at much higher risk of compromise due to brute-force password guessing, as we have seen.

• Additional Sources of Evidence Here are some high-priority potential sources of additional evidence that might be useful in the case:

– Central Logging Server—Based on the flow record data, there may be syslog traffic (port 514) sent over the network. (Although IANA has only assigned UDP 514 to syslog, many organizations have switched to using TCP for transport of syslog data, and frequently choose to use TCP 514 for syslog.) It would be wise to inquire with local system administrators to determine if there is a central logging server. If so, logs from this system can be gathered and analyzed. In particular, authentication logs from the two compromised systems (10.30.30.20 and 192.168.30.101) would be of great interest.

– Firewall logs—The firewall logs may provide more granularity regarding denied and allowed network activity relating to the incident.

– Hard drives of compromised systems—Forensic analysis of the compromised system hard drive may reveal detailed information about the attacker’s activities, or at the very least, allow for an inventory of confidential information that may have been compromised.