Chapter 9. Switches, Routers, and Firewalls

Search in book...
Toggle Font Controls
Create new playlist

Name your new playlist

Playlist description (optional)
Sign In

Email address

Password

Forgot Password?

or

Continue with Facebook

Continue with Google
Sign Up

Full Name

Email address

Confirm Email Address

Password

or

Continue with Facebook

Continue with Google

Chapter 9. Switches, Routers, and Firewalls

“The Internet . . . is not a big truck. It’s a series of tubes. And . . . those tubes can be filled and if they are filled, when you put your message in, it gets in line and it’s going to be delayed by anyone that puts into that tube enormous amounts of material, enormous amounts of material.”

—Former U.S. Senator Theodore “Ted” Stevens (R-Alaska)¹

1. T. Stevens, in a speech before the U.S. Senate on “network neutrality,” June 2006, http://media.publicknowledge.org/stevens-on-nn.mp3.

The line between switches, routers, and firewalls has become very blurred. It only exists as a theoretical line, which is no longer strictly implemented at all, if it ever really was. What does that mean for the forensic investigator? The evidence you may expect to find on one device may actually exist on another. A device called a “switch” may actually contain logs that you would expect to find on a “firewall.”

Regardless of their label, network infrastructure devices contain configurations that reflect the state of the network and activities and, with any luck, the policies of the enterprise that’s deployed them. As a result, they often contain evidence that may be of use in an investigation, including descriptive information about the investigative environment, and perhaps evidence relating to a particular event of interest.

In this chapter, we discuss traditional switches, routers, and firewalls, as they pertain to network forensic investigations. We review the types of storage media commonly found in network infrastructure devices, evidence found on different types of devices, common interfaces, and logging setups. Keep in mind, however, that when you are examining a piece of equipment called a “switch” or a “router,” these devices may include functionality normally associated with other types of devices (for that matter, the same holds true for “hubs,” which are nearly always actually switches these days). It is always a good idea to research the specific make and model of the equipment under investigation before beginning your forensic examination.

Don’t worry so much about the label. Whether Cisco says a device is a hub or a switch, the investigator’s job is to understand the feature set and the configuration of the device, whatever it may be called.

9.1 Storage Media

The types of storage used on switches, routers, and firewalls vary between manufacturers and models. It is important for forensic investigators to be familiar with common types of storage used in network equipment in order to properly prioritize evidence collection. Understanding the volatility of data on different storage mediums is paramount, and as a general rule evidence should be collected and preserved in order of volatility (beginning with the most volatile first).

Common types of storage in switches, routers, and firewalls include (in approximate order of volatility):

• Dynamic Random-Access Memory (DRAM) DRAM is very volatile and does not retain data (for long) when power is turned off. It is commonly used to store running operating system configuration, process memory, routing tables, firewall statistics, and more.

• Content-Addressable Memory (CAM) CAM is a special kind of very fast memory used to store information that must be accessed extremely quickly. It is most famously used on switches for storing tables that map MAC addresses to ports (hence the name “CAM tables”). CAM is very volatile and does not retain data when power is turned off.

• Nonvolatile Random-Access Memory (NVRAM) NVRAM retains data when the power is turned off, but can also be easily modified. The most common type of NVRAM found in network equipment is “flash memory.” In routers, this often contains a copy of the operating system used at boot, as well as startup configuration files.

• Hard drive Most switches, routers, and firewalls do not include a hard drive. However, general-purpose servers can be configured to act as routers or firewalls (i.e., a Linux system running iptables). In these cases, the hard drive typically contains the operating system, startup configuration, firewall logs, and an extensive amount of other data. The data on a hard drive remains after the power is turned off.

• Read-Only Memory (ROM) ROM is a type of random-access memory that is designed to permanently store data without modification (hence the name). ROM is not designed to be routinely modified, although nowadays types of memory commonly referred to as “ROM” can be reprogrammed in order to update firmware. For example, on unmanaged switches, the operating system is typically stored in ROM. For more capable and flexible managed switches and routers, the ROM typically contains a boot loader, which loads the operating system and configuration from NVRAM. On fully configurable Linux systems that are used as routers or firewalls, the ROM normally contains the boot loader.

9.2 Switches

Switches are Layer 2/3 devices that connect multiple computers together to form a network. Unlike hubs, switches isolate traffic on different switch ports, so that each switch port is a separate collision domain. This prevents Layer 1 interference between stations on different switch ports and improves performance.

9.2.1 Why Investigate Switches?

Switches are typically involved in investigations for one of a few reasons:

• If you are trying to sniff traffic on a local segment, one of the easiest ways is to set up port mirroring on the switch. See Chapter 3, “Evidence Acquisition,” for details.

• Switches contain tables that map client network card addresses (MAC addresses) to physical ports on a switch. This can help you to physically track down a computer.

• Attackers may launch attacks designed to “confuse” the switch in order to bypass network security restrictions or launch man-in-the-middle attacks. Forensic analysis of the switch may help to identify and isolate attacks of this type.

9.2.2 Content-Addressable Memory Table

Ethernet switches typically contain a special type of very fast memory called CAM. This memory holds a table, referred to as the “CAM table,” that dynamically maps MAC addresses to corresponding physical ports on the switch. When a frame comes into a port, the switch looks up the destination MAC address in the CAM table to see which port it is attached to. Then, it writes a copy of the frame to the port associated with the destination MAC address.

For forensic investigators, the CAM table of an Ethernet switch can be very valuable, since it contains the MAC addresses of the network cards communicating on the local subnet. This table is very volatile and can change quickly, depending on network activity.

When an attacker is trying to sniff local network traffic, the CAM table often contains clear evidence of suspicious activity.

Below is the CAM table from a Cisco ASA 5505 Version 8.3 with the hostname “ant-fw.” Be careful—the CAM table reports “Age” as the number of seconds an entry has left before it expires rather than the number of seconds that have transpired. MAC records expire after five minutes, or 300 seconds.

9.2.3 Address Resolution Protocol

The Ethernet Address Resolution Protocol (ARP)² is designed to dynamically distribute Layer 2 Ethernet addresses to systems on the local subnet. Hosts on the local subnet, and many switches, maintain an ARP table in memory that maps MAC addresses to higher-layer protocol addresses such as IP addresses.

2. David C. Plummer, “RFC 826—Converting Network Protocol Addresses to 48.bit Ethernet Address for Transmission on Ethernet Hardware,” IETF, November 1982, http://www.rfc-editor.org/rfc/rfc826.txt.

On a typical Ethernet/IP network, the sender will use ARP to determine what MAC address is in use by the destination system with which it wants to communicate (or the local router), and will encapsulate an IP packet in an Ethernet frame with the appropriate destination MAC. If the destination IP address is not local, it must encapsulate the frame with the MAC address of the local gateway system.

Typically, ARP replies are cached for some period of time in an ARP table on each system that participates on the local area network. This ARP cache can be inspected on both Windows and Linux/UNIX systems via the ‘arp -a’ command. This is often the quickest way to determine what MAC-to-IP mappings may be in place on a local host, although this information could be spoofed.

Here is an example of an ARP table on a Cisco ASA 5505 Version 8.3 with the hostname “ant-fw.” The lines shown are dynamic ARP entries. The number at the end of each line represents the age of the ARP entry, in seconds.

ant-fw# show arp
  inside 192.168.30.30 0008.742d.2f94 94
  inside 192.168.30.100 0008.74fa.a6cc 99
  inside 192.168.30.102 0012.7964.f718 470
  inside 192.168.30.101 000b.cdc2.e491 480
  inside 192.168.30.90 0008.74a0.2e02 4091
  outside 172.30.1.5 0001.031a.d5f6 94
  outside 172.30.1.254 5475.d0ba.522a 2160
  dmz 10.30.30.20 0008.74d5.e0c4 409

In contrast, here is an ARP table from an Ubuntu Linux server:

$ arp -na
? (192.168.30.101) at 00:0b:cd:c2:e4:91 [ether] on eth0
? (10.30.30.20) at 00:08:74:d5:e0:c4 [ether] on eth1
? (172.30.1.5) at 00:01:03:1a:d5:f6 [ether] on eth2
? (172.30.1.254) at 54:75:d0:ba:52:2a [ether] on eth2

Both systems show the MAC address, the corresponding IP address, and indicate the associated interface.

9.2.4 Types of Switches

Since the introduction of the first Ethernet switch (essentially a multiport bridge) by Kalpana in 1990, commercial options for switched technology have multiplied.³

3. Robert J. Kohlhepp, “Top 10 Most Important Products of the Decade: Number 5,” Network Computing, October 2, 2000, archived in the Internet Archive: Wayback Machine, http://web.archive.org/web/20090218115819/http://www.networkcomputing.com/1119/1119f1products_5.html.

9.2.4.1 Managed Switches

Managed switches are designed for enterprise environments. They typically include a variety of features, such as:

• Support for VLANs

• Access control lists (ACLs)

• ARP caching

• 802.1x authentication

• Port mirroring

• Performance monitoring capabilities

• Event logging

Managed switches also typically support a variety of configuration interfaces, such as:

• Console (command-line interface [CLI])

• Remote console (SSH/Telnet)

• SNMP

• Web interface

• Proprietary interface (i.e., Cisco’s ASDM)

9.2.4.2 Smart Switches

“Smart” switches are a middle-of-the-road option for organizations that need some of the features of managed switches, but may not want to purchase or configure an expensive, fully-managed switch. Typically, smart switches include a limited set of features, such as:

• Support for VLANs

• ARP caching

• Port mirroring

• Performance monitoring capabilities

The exact features vary extensively between manufacturers and models. Commonly, smart switches offer at least a web interface, and often a console and remote console (SSH/Telnet) interface as well.

9.2.4.3 Unmanaged (or “Not-So-Smart”) Switches

Unmanaged switches are designed for “small office/home office” (SO/HO) or personal use. They are normally plug-and-play, with no configuration interface, and tend to have limited forensic value for this reason.

9.2.5 Switch Evidence

In this section, we list the types of evidence that can be gathered from switches and discuss relative volatility.

9.2.5.1 Volatile

The majority of evidence on a switch tends to be volatile, stored in very fast memory such as CAM, or RAM for less time-sensitive applications. Volatile evidence on a switch can include:⁴

4. “Catalyst 6500/6000 Switches ARP or CAM Table Issues Troubleshooting—Cisco Systems,” October 27, 2009, http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00807347ab.shtml.

• Stored packets before they are forwarded

• CAM tables—MAC address to port mappings

• ARP table—MAC address to IP address mappings

• Access control lists

• I/O memory

• Running configuration

• Processor memory

• Flow data and related statistics

9.2.5.2 Persistent

Switches also include persistent storage media that contain evidence such as:

• Operating system image

• Boot loader

• Startup configuration files

9.2.5.3 Off-System

Managed switches (and some smart switches) may support automatic exportation of log data or flow data. Review the configuration files and architecture documents to identify these remote storage systems and retrieve the data. In addition, backup copies of configuration files are often stored on an external TFTP server.

9.3 Routers

Routers are (by definition) Layer 3 devices designed to pass packets between networks. These days, pretty much every router has the ability to do some filtering at Layer 3 as well. Most can examine Layer 4 sockets to some degree, and at least filter traffic based on source/destination port.

9.3.1 Why Investigate Routers?

Routers are typically involved in investigations for one of a few reasons:

• Traffic of interest may traverse the router, resulting in associated flow data and related records. (A router is one of the most basic logging devices you will find on any network and also one of the most fundamental.)

• The network topology is key to understanding evidence and incidents, and is described at Layer 3 by the aggregate of routing tables.

• The router itself may be compromised.

9.3.2 Types of Routers

There are many types of routers, designed and marketed for specilized uses, ranging from core routers that provide a backbone of communication across an entire enterprise, to small-scale routers designed to connect home computers to an ISP. Routers tend to be dedicated physical appliances, not computers with GUI consoles (although they can be). The functionality, form factor, and interfaces of routers vary extensively depending on the usage model.

Much like switches, routers are available with a variety of features depending on the class and expense of the device.

Broadly speaking, general classes of routers include enterprise, consumer, and “roll-your-own.”

9.3.2.1 Enterprise

Enterprise-class routers are designed for use inside the enterprise, across WANs, or by ISPs. They tend to have extensive capabilities, such as:

• Stateful packet filtering

• Support for a wide variety of routing protocols (BGP, OSPF, etc)

• Multiprotocol Label Switching (MPLS)

• High-availability and high-throughput through failover and load balancing

• DHCP

• Network address translation (NAT)

• Quality-of-service (QoS)

• Performance monitoring capabilities

• Event logging

Very large-scale environments such as ISPs commonly buy extremely large arrays of routers that provide all of this functionality with tremendous throughput.

The interface options for enterprise-class routers include:

• Console (CLI)

• Remote console (SSH/Telnet)

• SNMP

• Web interface

• Proprietary interface (i.e., Cisco’s ASDM)

• Central management interface such as the CiscoWorks Management Center

Enterprise-class devices are often managed using many-to-one interface models, such as the CiscoWorks Management Center, which allow IT staff to centrally manage a fleet of routers. Common enterprise routers include Juniper’s J-Series and Cisco’s ASA series (replacing the PIX series, which has been deprecated).

9.3.2.2 Consumer

Individual home users and small offices use consumer-class routers to connect small LANs to ISPs for Internet access. These devices are typically available at local retail stores, and require little or no technical knowledge to set up and manage. The router may also be an ISP-provided device, not something selected by the end-user or local business. The feature set of consumer-class devices is limited, although modern routers of this type often include features that were limited to enterprise-class devices only a few years ago.

Capabilities of consumer-class routers typically include:

• ISP connectivity (i.e., PPPoE)

• Port filtering

• DHCP

• Bundled 802.11 (wireless) interface

• NAT

The configuration interface options for consumer-class routers tend to be limited to a web interface. A few consumer-class devices may also include limited command-line access. Typical consumer-class devices include Qwest DSL modem/router, cable modem/routers, Linksys WRT54G wireless router, Apple Airport Express, etc.

9.3.2.3 Roll-Your-Own

It is possible to build your own router using general-purpose hardware and operating systems (including Windows, Linux, and UNIX). This requires at least two network interfaces for forwarding packets. The operating system must be configured to support IP forwarding.

Beyond this, the complexity, features, and configuration interfaces of the router are limited only by the software running on the specific system, and the needs and imagination of the administrator. Common software for roll-your-own routers includes iptables on Linux and Zebra on UNIX.

9.3.3 Router Evidence

In this section, we list the types of evidence that can be gathered from routers, categorized by expected volatility.

9.3.3.1 Volatile

As with switches, much of the evidence on a router tends to be volatile. This includes:⁵

5. “Cisco IOS Configuration Fundamentals Configuration Guide, Release 12.2—Maintaining System Memory [Cisco IOS Software Releases 12.2 Mainline]—Cisco Systems,” 2011, http://www.cisco.com/en/US/docs/ios/12_2/configfun/configuration/guide/fcf009_ps1835_TSD_Products_Configuration_Guide_Chapter.html_#wp1008722.

• Routing tables

• Stored packets before they are forwarded

• Packet counts and statistics

• ARP table—MAC address to IP address mappings

• DHCP lease assignments

• Access control lists

• I/O memory

• Running configuration

• Processor memory

• Flow data and related statistics

9.3.3.2 Persistent

Commonly, the following files are maintained in persistent storage:

• Operating system image

• Boot loader

• Startup configuration files

• Access logs, DHCP logs, etc. (primarily for roll-your-own routers with hard drives)

9.3.3.3 Off-System

Routers tend to include very little, if any, writable persistent storage on board. However, most enterprise-class devices can be configured to automatically export data to external systems for storage, through syslog, FTP, TFTP, SNMP, and other means. Review the capabilities of the specific make and model you are investigating, as well as the running configuration information, to determine what, if any, data is being exported for storage elsewhere. Review of external storage media can provide you with a wealth of information such as the router access history, DHCP logs, backup configuration, flow records, and more.

9.4 Firewalls

Firewalls are essentially souped-up routers capable of inspecting and filtering traffic to a much higher degree. Early firewalls were most often built and configured by local system administrators using general operating system tools and commercial or open-source firewall software packages. However, general-purpose hardware introduced significant latency, and as a result inspection capabilities were limited. Furthermore, system administrators were not always well versed in operating system hardening procedures.

While early commercial and open-source firewall applications may have been well secured, the underlying operating systems often contained major security vulnerabilities. As that result, vendors began to offer firewalls as commercial standalone appliances. These were designed to provide optimized hardware and a hardened, embedded operating system.

Today, the term “firewall” can refer to an extremely wide variety of equipment and functionality, ranging from simple stateless packet filters to application-layer firewalls that include content inspection.

9.4.1 Why Investigate Firewalls?

Firewalls often play a central role in network forensic investigations, for one of a few reasons:

• Firewall logs tend to include extensive information about connection attempts, whether or not these were successful, and if so, how much data was transferred from source to destination. Firewall logs may also include extensive details regarding protocols and applications in use, or even packet contents.

• Firewall configuration can reveal whether services or data were exposed to the world, or to systems of interest. It can also inform an investigator as to the type of evidence that logs may or may not include.

• An investigator may need to modify firewall configuration in order to configure the firewall to collect more evidence, or to gain access to systems of interest during the course of an investigation.

• The firewall itself may be compromised.

9.4.2 Types of Firewalls

In 1994, in their seminal book on the topic, Cheswick and Bellovin defined three basic types of firewalls. Although the technology has advanced in each of these categories, the same general distinctions apply today:⁶

6. William Cheswick and Steven Bellovin, Firewalls and Internet Security: Repelling the Wily Hacker (Boston: Addison-Wesley, 1994).

• Packet Filters Devices that route packets and can “allow” or “deny” traffic based on source and destination addresses (at Layer 3) and Layer 4 protocol header information such as TCP ports and flags. In 1994, technology did not allow for truly stateful inspection of a connection, and so state was inferred based on TCP flags.

• Session-Layer Proxies A device between the source and destination that intercepts connections in order to make stateful decisions about whether the firewall will establish or continue a connection on behalf of the endpoints. Note that when session-layer proxies are in use, the source and destination do not establish connections directly with each other. Rather, they each establish a connection with the proxy, and the proxy maintains state. Session-layer proxies are often designed to provide a man-in-the-middle inteception of TLS- or SSL-encrypted traffic, to allow for decrypted content inspection.

• Application Proxies Application proxies take this concept even further by inspecting traffic all the way up to Layer 7. The precise protocols inspected and reconstructed vary depending on the manufacturer, model, and purpose of the device.

These distinctions are important for forensic examiners because different types of firewalls generate and retain different types of evidence. The types of evidence available from firewalls is usually constrained by the manner in which they inspect traffic and make decisions.

For example, packet filters do the most shallow inspection of only a few Layer 3 and 4 protocol fields, and as a result can usually only record or log those values. On the other end of the spectrum, an application proxy parsing and inspecting web traffic might be capable of recording or logging URLs, content types of files transferred, file sizes, authentication information, or even interesting snippets of page content.

General classes of firewalls include enterprise, consumer, and “roll-your-own.”

9.4.2.1 Enterprise-Class Firewalls

Enterprise-class firewalls are typically offered as part of an appliance that also includes router and/or IDS functionality. However, more powerful enterprise-class firewalls, such as session- or application-layer proxies, may be offered as standalone devices. Common features of enterprise-class firewalls include:

• NAT

• DHCP

• VPN tunneling (such as IPsec, SSL, and others)

• Load balancing

• Fail-over

• Fragmentation reassembly

• Stateful packet filtering

• Performance monitoring capabilities

• Centralized management features

• Event logging

• Expansion slots for increased flash storage

• Content reassembly and inspection

The interface options for enterprise-class firewalls include:

• Console (CLI)

• Remote console (SSH/Telnet)

• SNMP

• Web interface

• Proprietary interface (i.e., Cisco’s ASDM)

• Central management interface such as Cisco’s MARS

9.4.2.2 Consumer-Class Firewalls

With consumer-class devices, a small set of firewall features are typically bundled into what are primarily routers for ISP access. Common features of consumer-class firewalls include:

• NAT

• Packet filtering (perhaps semi-stateful, perhaps not)

• 802.11 interface

• DHCP

Some vendors do market dedicated “firewalls” for the small office/home office (SO/HO) niche, but even these tend to be multipurpose devices. For example, Cisco’s ASA5505 is a firewalling router with built-in switch ports, intended to be an “all-in-one” network device. While this appliance supports many of the enterprise-class features of the larger, rack-mounted ASAs, they are nonetheless more limited; VLANs are supported, but only a few, and routing between more than three segments is not supported (despite having eight ports).

As discussed above, the configuration interface options for these consumer-class devices tend to be limited to a web interface. A few consumer-class devices may also include limited command-line access. Typical consumer-class devices include Qwest DSL modem/router, cable modem/routers, Linksys WRT54G wireless router, Apple Airport Express, etc. (Perhaps few people would consider an Airport Express a “firewall,” but to the extent that it performs NAT and some amount of filtering, it can be used as such.)

9.4.2.3 Roll-Your-Own

In the early days, many firewalls (unlike routers) were distributed as commercial or open-source software applications configured by administrators to run on general-purpose hardware. As a result, there are many mature software firewall distributions that run on top of general-purpose operating systems and equipment.

It is possible to build your own firewall using general-purpose hardware and operating systems (including Windows, Linux, and UNIX). With Windows, this is typically done by installing an ISA server on top of a Windows Server operating system. With Linux, iptables is the native firewalling software included with common distributions. BSD uses ipfw, and Solaris is distributed with ipf.

9.4.3 Firewall Evidence

As with switches and routers, a great deal of evidence generated by firewalls is stored in volatile memory. “Roll-your-own” firewalls, which are built on top of a general-purpose operating system and hardware, tend to have more persistent storage available, since they are usually built with COTS hard drives.

9.4.3.1 Volatile

Volatile evidence gathered from a firewall can include:

• Current, running configuration, including:

– Interface configurations

– ACLs and other firewall rules

– Tunnels and their current states

– Routing table and ARP cache

• Packet- and frame-level statistics

• Processor and I/O memory

• Command history

Note: Some aspects of the running configuration of routers is expected to be dynamic, and therefore to diverge from the initial boot configuration—particularly routing tables, which are constantly renegotiated through the Border Gateway Protocol (BGP) and other protocols. Conversely, firewall configurations are intended to be fairly static in most ways. For example, as with all networked devices, firewalls have routing tables too, but they are usually statically defined in the startup configuration. That said, network administrators sometimes make changes to firewall rules in memory without writing them to persistent storage.

This can lead to a common error among inexperienced investigators, assuming that the firewall configuration can be inspected by simply acquiring the startup configuration (typically backed up somewhere more easily accessible than the firewall itself). However, a savvy investigator will realize that the running configuration might have diverged from the initial configuration—either through mishap or malice, and that any discrepancies found between them might be useful evidence indeed!

9.4.3.2 Persistent

Firewalls do store evidence in less volatile forms, including:

• Initial boot load

• Startup configuration

• Other static binaries

• Access logs, DHCP logs, etc. (primarily for roll-your-own firewalls with hard drives)

9.4.3.3 Off-System

The evidence that any given firewall can produce varies by make and model. As a general rule, however, devices that include firewall capabilities tend to produce much richer logs and other data than more rudimentary appliances such as routers and switches.

The process of recording activities, formatting such data into logs, and sending logs to a central log host takes CPU cycles and memory away from the firewall’s primary task of accurately filtering traffic without introducing unacceptable latency for the packets flowing through it. Consequently, firewall software engineers usually refrain from enabling extensive event logging by default. On the other hand, any of the protocol details that the firewall has already had to parse, packet-by-packet, layer-by-layer, may be included in default logging.

Firewall appliances usually have little, if any, local writable storage. Fortunately, modern enterprise-class firewalls typically include built-in capabilities for exportation of firewall logs and other information of interest. Research the make and model of firewall that you are investigating to determine its precise capabilities, and then review active configuration files to identify remote systems that are used as off-system storage.

Evidence of an exported off-system may include, among other things:

• Firewall logs

• Access history

• Backups of configurations

9.5 Interfaces

We’ve discussed the various types of routers, switches, and firewalls in common use, and the types of evidence found on each. Now let’s discuss how you can actually get access to evidence.

As before, the types of interfaces and supported features vary greatly based on the manufacturer and model of a particular device. Switch, router, and firewall web interfaces are usually password-protected, so you will need to obtain the appropriate username and password before logging in. If the password is unknown, it may be necessary to reboot or even reset the device in order to gain access to configuration details. This is highly undesirable since it results in the loss of volatile data. If you do not have access credentials for a device, it is generally worth trying to use default manufacturer passwords, which are often published online, or at least conduct an inspection without access (see Section 3.3.2, “Inspection Without Access”).

For managed or enterprise-class devices, there may be multiple levels of privilege available. Forensic analysts typically need the highest level of privilege, which allows full access to all configuration details, although this can vary depending on the investigation.

Types of interfaces on switches, routers, and firewalls are discussed next.

9.5.1 Web Interface

Nearly all consumer and enterprise-class routers and firewalls, as well as managed switches, include a web interface. Web interfaces are very portable and easy for users to manage.

While web interfaces often allow you to view (and modify) device configuration and logs, in many cases, they do not allow you to easily export data in a format suitable for convenient analysis. For example, configuration details may be split across multiple pages of the site. If you only need to retrieve a small amount of data, a simple screenshot or manually saving a page’s source code might suffice. However, if you are not sure what you are looking for, or if your goal is to retrieve extensive amounts of information, manual page-by-page collection does not scale. In these situations, if there is no easy export function, a client-side web proxy such as Burp can be very helpful for spidering the site after login and automatically retrieving data stored on all pages of the site.⁷

7. “Burp Suite,” PortSwigger Web Security, 2011, http://www.portswigger.net/burp/.

Simply browsing a web interface significantly modifies the volatile memory of a device. The device necessarily runs a web server, which copies web pages into memory in order to serve them to the end-user. Accessing the web interface also generates network traffic, and may modify router statistics, flow record data, and access logs. If the web interface is not encrypted, this can lead to exposure of login credentials or evidence. Always keep a careful record of your actions. If you must modify the system configuration, record your changes and collect screen captures or other documentation whenever possible.

9.5.2 Console Command-Line Interface (CLI)

Most enterprise-class routers and firewalls, as well as many managed switches, include a serial interface that allows direct command-line access to the system console. (There are USB-to-serial connectors available that allow laptops or forensic workstations that do not have serial ports to connect to the serial console via USB.) The functionality available through this command-line interface varies greatly depending on the device’s operating system. On enterprise-class devices running, for example, Cisco IOS, the command-line interface allows you to retrieve extensive running configuration details, list the contents of flash memory, output CAM and ARP tables, retrieve VLAN configuration, and much more. It also allows you to modify the device configuration and export data to external systems.

When you connect via console, it is best to keep a record of all your commands and output. This can be done using client-side tools; for example, the “screen” command on Linux includes an option (-L) for keeping a full log of the session. Not only will it save you the effort of having to manually export command output, it will also provide excellent documentation of your forensic investigative techniques.

Here is an example of a console connection from a Linux laptop to a Cisco ASA 5505 with the hostname “ant-fw.” The connection was made using “screen” and a Keyspan serial-to-USB connector:

$ screen -L /dev/ttyUSB0
ant-fw> enable
Password:
ant-fw# show clock
16:50:25.364 MDT Tue Apr 26 2011
ant-fw# show version
Cisco Adaptive Security Appliance Software Version 8.3(2)
Device Manager Version 5.2(4)
Compiled on Fri 30-Jul-10 17:49 by builders
System image file is "disk0:/asa832-k8.bin"
Config file at boot was "startup-config"
ant-fw up 1 hour 48 mins
Hardware:   ASA5505, 512 MB RAM, CPU Geode 500 MHz
Internal ATA Compact Flash, 128MB
BIOS Flash M50FW016 @ 0xfff00000, 2048KB
Encryption hardware device : Cisco ASA-5505 on-board accelerator (revision 0
    x0)
                             Boot microcode   : CN1000-MC-BOOT-2.00
                             SSL/IKE microcode: CNLite-MC-SSLm-PLUS-2.03
                             IPSec microcode  : CNlite-MC-IPSECm-MAIN-2.06
0: Int: Internal-Data0/0    : address is d0d0.fdc4.0994, irq 11
1: Ext: Ethernet0/0         : address is d0d0.fdc4.098c, irq 255
...
ant-fw(config)# show run
: Saved
ASA Version 8.3(2)
hostname ant-fw
domain-name example.com
enable password XXXXXXXXXXXXXXXX encrypted
passwd XXXXXXXXXXXXXXXX encrypted
names
interface Vlan1
nameif inside
security-level 100
ip address 192.168.30.10 255.255.255.0
interface Vlan2
nameif outside
security-level 0
ip address 172.30.1.253 255.255.255.0
interface Vlan3
no forward interface Vlan1
nameif dmz
security-level 50
ip address 10.30.30.10 255.255.255.0
interface Ethernet0/0
...

Connecting to a device via serial console is usually the least impactful way to conduct a live examination of a device and retrieve volatile evidence. While console connections do, of course, create changes in the device’s dynamic memory, the memory usage is far less than that of a web server. Furthermore, console connections do not create network traffic, and therefore do not modify router statistics or flow record data.

9.5.3 Remote Command-Line Interface

Managed switches and enterprise-class routers and firewalls may include network-accessible command-line interfaces. Typically, these are accessed over the network through a Telnet or SSH service. The functionality and commands are usually the same as a serial console connection.

When accessing a device over the network, it is important to recognize that the act of connecting to the device in this manner necessarily generates network traffic, and can modify router statistics and access logs in addition to the device memory itself. Also, if the device only supports unencrypted Telnet connections and does not include an encrypted remote command-line service such as SSH, then device credentials and evidence may be exposed in transit. Always be cautious when connecting to a router, switch, or firewall over the network, and use a direct console connection instead whenever possible.

9.5.4 Simple Network Management Protocol (SNMP)

Many network devices are configured to use SNMP for central management, both with respect to gathering information and setting configuration values remotely. For forensic investigators, SNMP can also also be a very useful tool for collecting evidence stored on a device. Typically, SNMP agents are password-protected by “community strings” that limit read and write access. By default, these are often set to “public” and “private,” where the former string allows read access only, and the latter also allows write access. Many organizations never change the default SNMP community strings for a device.

Forensic investigators can leverage SNMP to gather information ranging from device software version and current time, to routing and network interface details. This is one way that you may be able to retrieve CAM table information from switches, routing tables from routers, and ACLs from a firewall. With the appropriate authentication credentials, it may also be possible to modify the device configuration.

Many SNMP agents are not configured to support encryption. Be aware that by connecting to the agent, you may reveal the authentication credentials and subsequent data transfers to others monitoring the local network. As of SNMPv3, encryption is supported. However, many network devices and management applications still do not support SNMPv3.

Common SNMP managers in use include SolarWinds Network Management Software and the open-source Net-SNMP suite. (Please see Chapter 3, “Evidence Acquisition,” for more details.)

9.5.5 Proprietary Interface

Many device manufacturers develop their own proprietary interfaces for managing switches, routers, and firewalls. Often, these require special client-side software for connecting to the device. This can be challenging for forensic examiners because your examiner’s workstation may not support the software required.

Portable web-based interfaces are gaining in popularity. However, because network devices often stay in production for many years, forensic examiners will still encounter proprietary interfaces for many years to come.

Apple’s AirPort Express wireless router/switch interface is a very common example. The AirPort Utility application ships natively with all current Mac OS X operating systems, and a Windows equivalent is freely available. At the time of publication of this book, Apple has not released a version of the AirPort utility for Linux.

Cisco’s VMS, CiscoWorks, and ASDM are examples of Java-based proprietary interfaces designed for managing Cisco network devices. You will need to have a compatible JVM installed on your client system in order for these to work.

9.6 Logging

Forensic investigators may find device logs to be useful repositories of evidence relevant to a case. You may also wish to enable or modify a device’s logging configuration so as to gather new evidence, or to keep an accurate record of your own activities.

Here is an example of the logging configuration gathered from a Cisco ASA 5505 Version 8.3 with the hostname “ant-fw”:

ant-fw(config)# show logging
Syslog logging: enabled
    Facility: 20
    Timestamp logging: enabled
    Standby logging: disabled
    Deny Conn when Queue Full: disabled
    Console logging: level notifications, 39 messages logged
    Monitor logging: level notifications, 39 messages logged
    Buffer logging: disabled
    Trap logging: level informational, facility 20, 78 messages logged
        Logging to inside 192.168.30.30
    History logging: disabled
    Device ID: hostname "ant-fw"
    Mail logging: disabled
    ASDM logging: level informational, 78 messages logged

In the example above, the firewall (“ant-fw”) is configured to send syslog logs to a remote host, 192.168.30.30. Various other logs are also enabled, including console logging, monitor logging, and ASDM logging.

9.6.1 Local Logging

Virtually all switches, routers, and firewalls perform some amount of event recording by default, and do so by writing out event logs to whatever local media is available. Unfortunately for the investigator, this is usually highly volatile and extremely limited in volume. In some cases, unless you happen to be watching the console when the event occurs, the information will be lost forever.

9.6.1.1 Console

A common example of local logging is “console” logging, in which events are displayed in real-time on the CLI console, interrupting whatever interactive session was occurring there—a practice that persists as a legacy of earlier times. Many devices (still including many UNIX and UNIX-like systems) default to logging the most critical events to the console on the notion that an operator is perched in front of a single system’s console, ready to respond. (In the case of networking equipment it is entirely possible that a console hasn’t been connected and viewed since the device was first configured and deployed, if ever.)

Below is an example of enabling console logs with timestamps on a Cisco ASA 5505 Version 8.3. Notice that once the command “logging console 5” was executed, the system immediately began printing event logs to the console (see the text that begins with a date).

ant-fw(config-if)# logging enable
ant-fw(config)# logging timestamp
ant-fw(config)# logging console 5
ant-fw(config)# Apr 13 2011 06:32:29: %ASA-5-111008: User 'enable_15'
executed the 'logging console 5' command.
Apr 13 2011 06:32:29: %ASA-5-111010: User 'enable_15', running 'CLI' from IP
0.0.0.0, executed 'logging console 5'

If you are connected to the console via an intermediary system (recommended), you can capture console logs by recording your console session with a tool such as “screen,” as follows:

$ screen -L /dev/ttyUSB0
ant-fw>

On the other hand, if you are connected to the console via a dumb terminal, your best bet for capturing console logs may literally be a camera, provided they aren’t scrolling by so fast as to preclude such methods.

9.6.1.2 Terminal

Similar to console logging is terminal logging, which simply logs events in the same real-time interruptive method to whichever CLI happens to be presently connected, locally or remotely. These can be recorded by a terminal capturing program on the client, such as “script,” which will record all of the input and output of a terminal session to a file on the investigator’s workstation.

The example below demonstrates use of the program “script,” which records a copy of all subsequent input and output in the terminal. You can see that right after the “script” command was run, the user ran “SSH” to connect to 192.168.1.1. The SSH command shown, as well as all input and output from the SSH session to follow, would be recorded in the local logfile “logfile.txt.”

$ script -a logfile.txt
Script started, file is logfile.txt
$ ssh [email protected]

9.6.1.3 Buffered

Depending on the switch, router, or firewall’s capabilities, you may be able to designate a portion of the device’s memory to be used for storing buffered logs, which users can opt to view when they are logged into the device. Buffered logs are volatile and are lost when the device loses power. They may also be overwritten when memory space is low. Flooding buffered logs by generating extraneous events is a time-honored way for attackers to obscure their events from responders, analysts, and investigators alike.

9.6.2 Simple Network Management Protocol

Previously we’ve discussed how SNMP can be used to actively collect data from devices (through the GET, GETNEXT, and GETBULK methods) or to modify device configurations (via SET methods). However, the protocol is also designed to allow SNMP agents to push data outbound in an event-driven manner using “traps.” Since SNMP is perhaps the most widely supported management protocol deployed on network equipment, SNMP traps are probably the most widely available means of exporting event data from them. (Then again, “most widely available” doesn’t necessarily mean the “most widely used.”)

From an investigator’s perspective, there are a few ways this medium can be useful. If SNMP traps are already being sent to a central logging server, one can be hopeful that they can be obtained from the central logging server and inspected within the context of other activity. If, on the other hand, the network layer is more immediately accessible than the log host, the SNMP traps can be captured just like any other traffic. This is especially easy because SNMP traps are normally sent unencrypted via UDP across the network.

If you’re in search of real-time data, our advice is to sniff for SNMP traps, even if they’re not being intentionally utilized by enterprise staff.

9.6.3 syslog

Syslog is one of the oldest and most widely used remote logging systems. It is supported by many types of network devices, inclusing those running Cisco IOS and most Linux- and UNIX-based operating systems. You can use syslog to record a wide variety of system events, from authentication to VPN to DHCP to system errors.

Years ago, one of the most notorious problems with Cisco IOS devices was that they only gave you the ability to specify one log host and one log facility, with logs from many different types of events all mixed together. Older versions of IOS did not give administrators the ability to separate different types of events to different facilities. As a result, network operators did not use Syslog as much as they otherwise might have.

Modern versions of Cisco IOS, and many other types of network devices, now allow administrators to specify different classes of events and multiple remote logging hosts.⁸ Many even support TLS/SSL-encrypted syslog transport. The precise capabilities of each device vary depending on the manufacturer, model, and software version.

8. “Cisco ASA 5500 Series Command Reference, 8.2—logging asdm—logout message [Cisco ASA 5500 Series Adaptive Security Appliances]—Cisco Systems,” 2011, http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/l2.html#wp1772754.

Below is a simple example of configuring remote syslog logging on a Cisco ASA 5505 Version 8.3 named “ant-fw.” In this case, we set the logging facility to 19 (local3) and configured the firewall to send syslog messages of severity levels 0, 1, 2, and 3 to a syslog server (192.168.30.30) on the “inside” interface.

ant-fw(config)# logging facility 19
ant-fw(config)# logging trap errors
ant-fw(config)# logging host inside 192.168.30.30

By default, this sends Syslog messages to UDP port 514 on the remote host. Modern versions of Cisco IOS also support TCP. (Please see Chapter 8, “Event Log Aggregation, Correlation, and Analysis” for more information about event logging.)

9.6.4 Authentication, Authorization, and Accounting Logging

It is increasingly common to find networking equipment that leverages central authentication, authorization, and accounting (AAA) servers as a back-end to what would otherwise appear to be front-end decisions as to whom to allow onto the network in the first place. This topic could easily include a bevy of protocols and acronyms (from 802.1X to TACACS/TACACS+/RADIUS to PEAP and LEAP). From an investigator’s perspective, however, the concept is fairly simple, and the strategy fairly straightforward.

In a well-architected and centrally managed environment, switches, routers, and firewalls typically handle authentication attempts by forwarding requests to a central authentication authority. Subsequent requests for specific access will likewise be brokered by the device, relying on the central authorization authority. All of these transactions, successful or not, are recorded in a central accounting database.

As an investigator, such a system can be a positive boon, even if you don’t know much (or anything) about the various 802.1X protocols involved.

9.7 Conclusion

Switches, routers, and firewalls connect, route, and filter traffic of all natures across networks. As a result, these devices may contain a wide range of evidence, from network configurations to logs relating to specific events of interest.

In this chapter, we reviewed the different types of evidence found on switches, routers, and firewalls, and discussed strategies for approaching evidence collection and analysis from network infrastructure devices. We also discussed the different types of storage media found in networking equipment, as well as common interfaces and log configurations.

Throughout, we studied forensic investigation of different types of networking devices, classified according to their traditional functions as “switches,” “routers,” and “firewalls.” Although this distinction may be blurred in real life, it is helpful for the purposes of discussion to examine different types of functionality and the kinds of evidence they produce.

9.8 Case Study: Ann’s Coffee Ring

The Case: Ann is on the move again, looking to poach secret recipes. She’s discovered that she can walk into the main lobby of the International Chaos Cookie Company (ICCC), and that if she walks through as though she has somewhere to be and knows where she’s going, she can breeze past the receptionist who is busy with other concerns.

After wandering around purposefully for a while, Ann chooses an out-of-the-way and unused conference room, sits down, jacks in to an Ethernet panel, and begins looking for the database server she discovered during some previous reconnaissance . . .

Meanwhile . . . Ann has forgotten that she configured her laptop to maintain a persistent connection to an IRC channel where she chats with other data thieves. When she jacks into the ICCC’s internal network her laptop (192.168.30.105) tries to make the outbound IRC connection (destination port 6667/TCP), which is prohibited by policy. Local security staff are alerted. Presuming a compromised system on the network, they try to locate it and track the port to the conference room. Realizing that there shouldn’t even be a computer in that room, security staff rush over, but when they get there it is empty. A seat is still warm, though, and there is a coffee mug ring on the table, still wet.

Challenge: The ICCC has for the moment an open-ended question. There’s been an anomalous event associated with suspicious device. An IDS alert was logged at 16:47:48 on 4/29/11 concerning an outbound IRC communication attempt from internal host 192.168.30.105 to port 6667/TCP on an external host.

You are the investigator . . .

• Map the suspect IP address back to a MAC address.

• Track the MAC address to the physical ports it used.

• Build a timeline of the suspicious system’s activities in order to determine if it was a bad actor, and if so, to scope the extent of compromise.

Addressing the following questions will help guide your investigation:

• When the suspicious system came online, it might have attempted to acquire a DHCP address. Are there lease assignment logs and, if so, what evidence do they contain?

• Typically, DHCP servers in enterprises are configured to provide clients with not just an IP address, but also important local configuration information such as the IP address of the local gateway and DNS server(s). Once received, the client must send broadcast ARP traffic to the local network in order to obtain the corresponding MAC address(es) of the local gateway, DNS server, or other important servers. Can you find evidence that the suspicious system did this?

• The CAM table from the switch should be able to tell you which physical port the system is connected to—or was—if you move quickly enough. Can you identify the physical port?

• Based on the timeline you developed, what activities was the suspicious system engaged in during the time frame of interest?

Evidence: Security staff at the ICCC collect DHCP lease logs, and ARP tables, and CAM tables as expediently as they can at your request. You are provided with the following files containing data to analyze:

• fw-evidence.txt—A text file containing the output of an inspection of the running Cisco ASA firewall. This is the device that provides DHCP service on the internal segment (192.168.30.0/24) and is also the core switch for that subnet.

• dhcp.log—The text file containing the remotely logged DHCP lease history from the firewall.

• firewall.log—A text file containing the remotely aggregated logs from the firewall.

9.8.1 Firewall Diagnostic Commands

A quick survey of fw-evidence.txt shows that someone has been kind enough to run some diagnostic commands for us on a Cisco ASA v8.3(2) device:

[...]
ant-fw# show run
: Saved
:
ASA Version 8.3(2)
[...]

Along with the running configuration, we’ve got both the ARP and CAM tables, preceded by a system clock reading to establish the beginning bounds of the time frame for the inspection. Looking first at the clock and the ARP table below, we learn that as of 16:52:48, the suspect system (192.168.30.105) was linked to the MAC address 00:26:22:cb:10:17, and the last ARP reply was just 138 seconds prior:

ant-fw# show clock
16:52:48.927 MDT Fri Apr 29 2011
ant-fw# show arp
        inside 192.168.30.30 0008.742d.2f94 38
        inside 192.168.30.50 0012.3f65.a7e1 51
        inside 192.168.30.105 0026.22cb.1017 138
[...]

Looking further at the CAM table as it was captured thereafter (shown below), we see that the MAC address in question was dynamically assigned as a member of VLAN 0001 to port Et0/6. Note also that the reported “age” associated with 00:26:cb:10:17 was 205 seconds. Recall that this represents the remaining number of seconds until the entry expires (300 seconds total), and so this entry was last updated 95 seconds prior, at 16:51:13.

ant-fw# show switch mac-address-table
Legend: Age - entry expiration time in seconds

   Mac Address  | VLAN |       Type       | Age | Port
-------------------------------------------------------
0008.742d.2f94 | 0001 |     dynamic      | 287 | Et0/5
0008.74a0.2e02 | 0001 |     dynamic      | 082 | Et0/7
000b.cdc2.e491 | 0001 |     dynamic      | 082 | Et0/3
0012.3f65.a7e1 | 0001 |     dynamic      | 246 | Et0/2
0012.7964.f718 | 0001 |     dynamic      | 082 | Et0/4
0026.22cb.1017 | 0001 |     dynamic      | 205 | Et0/6
d0d0.fdc4.0994 | 0001 |     static       |  -  | In0/1
ffff.ffff.ffff | 0001 | static broadcast |  -  | In0/1, Et0/0-7
0001.031a.d5f6 | 0002 |     dynamic      | 164 | Et0/0
5475.d0ba.522a | 0002 |     dynamic      | 164 | Et0/0
d0d0.fdc4.0994 | 0002 |     static       |  -  | In0/1
ffff.ffff.ffff | 0002 | static broadcast |  -  | In0/1, Et0/0-7
0008.74d5.e0c4 | 0003 |     dynamic      | 246 | Et0/1
d0d0.fdc4.0994 | 0003 |     static       |  -  | In0/1
ffff.ffff.ffff | 0003 | static broadcast |  -  | In0/1, Et0/0-7
Total Entries: 15

For further reference in mapping ports to MAC addresses to IP addresses, the VLAN declarations are seen below as follows:

ant-fw# show switch vlan
VLAN Name                             Status    Ports
---- -------------------------------- --------- -----------------------------
1    inside                           up        Et0/2, Et0/3, Et0/4, Et0/5
                                                Et0/6, Et0/7
2    outside                          up        Et0/0
3    dmz                              up        Et0/1

From the configuration, we can see that the ASA has eight interfaces and is configured with three VLANs (named “inside,” “outside,” and “dmz”).

9.8.2 DHCP Server Logs

Looking further at the running firewall config, we see the following lines (among many others):

[...]
logging enable
logging timestamp
logging list notification-dhcp-fw level notifications
[...]
logging host inside 192.168.30.30
[...]
dhcpd address 192.168.30.100-192.168.30.131 inside
dhcpd enable inside
[...]

The firewall does indeed appear to be providing DHCP on the “inside” interface, and it appears that it is logging lease assignments to a remote syslog server (port 514/UDP) on inside host 192.168.30.30. That’s where we could go to get confirmation of the lease assignment to 192.168.30.105.

Sure enough, we find confirmation in the “dhcp.log” file from the central log server by using the “grep” command to find all the entries for the suspicious IP address. There is only one matching entry:

$ grep '192.168.30.105' dhcp.log
2011-04-29T16:47:35-06:00 ant-fw : %ASA-6-604103: DHCP daemon interface
inside: address granted 0026.22cb.1017 (192.168.30.105)

As we assumed, the firewall assigned the IP address in question to the MAC address we’ve identified. This happened at 16:47:35, which corresponds with what we’ve seen so far. Since we suspect the system to be either compromised or at least to be behaving anomalously, we might also want to determine what other IP addresses it’s been assigned and when (if any).

Again, we use the “grep” command to search through the log file, but this time we’re searching for the MAC address, using the format recorded by the Cisco device:

$ grep '0026.22cb.1017' dhcp.log
2011-04-29T16:47:35-06:00 ant-fw : %ASA-6-604103: DHCP daemon interface
inside: address granted 0026.22cb.1017 (192.168.30.105)
$ head -1 dhcp.log
2011-04-29T16:02:39-06:00 ant-fw : %ASA-6-604103: [...]

Above we can see that the system in question has not shown up elsewhere in the DHCP logs since this file began on 4/29/11 sometime prior to 16:02:39. (The “head” command shows us just the first entry in the file, so we can see the starting timestamp.)

9.8.3 The Firewall ACLs

Now that we’ve established that 192.168.30.105 was a member of the “inside” VLAN and was topologically on an internal port, let’s return to the firewall configuration to have a look at the applicable ACLs. From this we should be able to figure out which actions should have been allowed, denied, and/or logged for future inspection.

Shown below, we can see that there are three access lists (inside, dmz, and outside) and that they are assigned to each of the three interfaces as follows:

access-list inside extended permit icmp 192.168.30.0 255.255.255.0 any log
access-list inside extended permit tcp 192.168.30.0 255.255.255.0 172.30.1.0
    255.255.255.0 eq ftp log
access-list inside extended permit tcp 192.168.30.0 255.255.255.0 host
    10.30.30.20 eq domain log
access-list inside extended permit udp 192.168.30.0 255.255.255.0 host
    10.30.30.20 eq domain log
access-list inside extended permit udp 192.168.30.0 255.255.255.0 host
    10.30.30.20 eq ntp log
access-list inside extended permit tcp 192.168.30.0 255.255.255.0 any eq www
    log
access-list inside extended permit tcp 192.168.30.0 255.255.255.0 host
    10.30.30.20 eq ssh log
access-list dmz extended [...]
access-list outside extended [...]
[...]
access-group inside in interface inside
access-group outside in interface outside
access-group dmz in interface dmz

Translated into English, what we see is that the ICCC’s internal hosts are allowed to send ICMP and web traffic anywhere; connect via FTP to external systems; exchange DNS, NTP, and SSH traffic with the DMZ server (10.30.30.20); and that all permitted actions shown above are logged.

9.8.4 Firewall Log Analysis

Previously, in our running firewall configuration, we saw that logging is enabled generally (not just for DHCP leases). Consequently, we should explore the logging host (192.168.30.30) for other logs from the firewall that might be able to shed light on the suspicious IP’s activities. We should expect to see any actions that are permitted, as well as logs for those denied.

Looking at “firewall.log” from the loghost (which has been pared down to the time frame of interest to make it simpler to work through), we see a number of important events. The first one is the firewall/switch reporting that port Ethernet0/6 changed state to “up” at 16:45:13. That is a definitive record that the port’s state changed from disconnected to connected:

2011-04-29T16:45:13-06:00 ant-fw : %ASA-4-411001: Line protocol on Interface
Ethernet0/6, changed state to up

We see that 23 seconds later, the suspicious system sent DNS traffic (perhaps DNS queries) to “dmz/10.30.30.20(53).”

As you can see in the logs below, 192.168.30.105 sent traffic on UDP port 53 to the local DNS server (10.30.30.20) a total of 16 times over the course of one minute and 15 seconds, between 16:47:36 and 16:48:51.

$ cat firewall.log |grep (53)|grep '192.168.30.105'
2011-04-29T16:47:36-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(44724) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:36-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(42410) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:36-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(36088) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:36-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(33475) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:48-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(48153) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:48-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(41901) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:48-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(56884) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:48-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(60365) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(36196) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(43763) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(47979) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(47465) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(38776) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(33464) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:48:51-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(37752) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:48:51-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(44679) -> dmz/10.30.30.20(53) [...]

Our suspect, 192.168.30.105, seems to be a chatty system. Let’s see if we can figure out what it’s doing. Here is the beginning of the firewall log:

$ head firewall.log
2011-04-29T16:45:13-06:00 ant-fw : %ASA-4-411001: Line protocol on Interface
    Ethernet0/6, changed state to up
2011-04-29T16:47:36-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(44724) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:36-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(42410) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:36-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(36088) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:36-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(33475) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:37-06:00 ant-fw : %ASA-4-106023: Deny udp src inside
    :192.168.30.105/123 dst outside:91.189.94.4/123 by access-group "inside"
    [0x0, 0x0]
2011-04-29T16:47:48.009774-06:00 ant-fw : last message repeated 3 times
[...]

In the first line above, we see Et0/6 go up at 16:45:13. We then see 192.168.30.105 successfully send four UDP datagrams to 10.30.30.20:53 (likely DNS queries) at 16:47:36. Next, we see 192.168.30.105 unsuccessfully attempt to send four UDP datagrams to 91.189.94.4:123 (likely NTP requests). Referring back to our ACLs, we should expect this result: internal systems are not allowed to use external systems for NTP. So then the question is, what sort of internal system would try such a thing? What is 91.189.94.4 and why would it be used for NTP by an internal system?

$ dig -x 91.189.94.4
[...]
;; QUESTION SECTION:
;4.94.189.91.in-addr.arpa. IN PTR

;; ANSWER SECTION:
4.94.189.91.in-addr.arpa. 3600 IN PTR europium.canonical.com.
[...]

So who is this “canonical.com”? That’s where Ubuntu Linux lives (see Figure 9-1). Often, Ubuntu Linux clients are configured by to obtain NTP information from Ubuntu servers, so it’s reasonable to hypothesize that the suspicious system was running Ubuntu Linux. According to ICCC staff, internal corporate clients are configured to use the DMZ server 10.30.30.20 for NTP. This is a pretty strong indication that we have a noncorporate system in our midst!

Figure 9-1 Canonical.com, the home of Ubuntu.

Below we see the next things that happen—all in the same few seconds:

2011-04-29T16:47:48-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(48153) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:48-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(41901) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:48-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(56884) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:48-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(60365) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:48-06:00 ant-fw : %ASA-4-106023: Deny tcp src inside
    :192.168.30.105/50885 dst outside:140.211.167.99/6667 by access-group "
    inside" [0x0, 0x0]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(36196) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(43763) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(47979) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(47465) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(38776) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:47:49-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(33464) -> dmz/10.30.30.20(53) [...]

We could use “grep” to easily filter out all that port 53/UDP traffic to the DNS server, but keeping it in view for the moment puts the other connection attempts in some context. Besides the fact that a process on the suspecious system likely attempted to resolve names/addresses, a process on the same system also tried to connect outbound to port 6667/TCP on host 140.211.167.99. This happened at 16:47:48, which corresponds with the timestamp that ICCC’s security folks reported was on the initial IDS alert. The firewall denied the attempt, and also provided corroboration that we’re on the right path.

Meanwhile, let’s learn more about the destination 140.211.167.99:

$ whois 140.211.167.99
[...]
NetRange:       140.211.0.0 - 140.211.255.255
CIDR:           140.211.0.0/16
[...]
OrgName:        Oregon State System of Higher Education
OrgId:          OSSHE-1
Address:        1225 Kincaid, UO Campus
City:           Eugene
StateProv:      OR
PostalCode:     97403
Country:        US
[...]

So what might be hosted in Oregon that is of interest to our intruder at that specific address? Does it have a name?

$ dig -x 140.211.167.99
[...]
;; QUESTION SECTION:
;99.167.211.140.in-addr.arpa. IN PTR

;; ANSWER SECTION:
99.167.211.140.in-addr.arpa. 43200 IN PTR zelazny.freenode.net.
[...]

That’s a published freenode IRC node,⁹ which makes sense since port 6667/TCP is by default assigned to IRC. The connection was blocked, and so we have no way of knowing what channel the suspect system was trying to join.

9. “About freenode: IRC Servers,” 2011, http://freenode.net/irc_servers.shtml.

About a full minute later, we see the following entries in the logs, all occurring in the same second:

2011-04-29T16:48:51-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(37752) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:48:51-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.105(44679) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:48:51-06:00 ant-fw : %ASA-3-710003: TCP access denied by ACL
    from 192.168.30.105/63019 to inside:192.168.30.10/22
2011-04-29T16:48:51-06:00 ant-fw : %ASA-3-710003: TCP access denied by ACL
    from 192.168.30.105/63020 to inside:192.168.30.10/22

It makes sense that connection attempts from the suspicious system to port 22/TCP of the firewall’s internal interface would be denied. There is nothing within the firewall’s running configuration that indicated that such attempts should be allowed. This raises the question of why the suspicious system attempted to connect directly to the firewall on port 22/TCP. Did it target that system specifically, or did it merely probe the local segment for port 22/TCP listeners? Based on the ACLs listed above, it doesn’t appear that this log file alone can tell us.

A few seconds later we see some port 53/UDP activity originating from the loghost to the DNS server, which may or may not be related to anything else we’re seeing here. After all, the loghost may potentially need to perform DNS lookups as part of its routine activity.

Then about 90 seconds later, we see something unusual:

2011-04-29T16:50:28-06:00 ant-fw : %ASA-4-106023: Deny tcp src inside
    :192.168.30.105/56145 dst outside:192.168.1.50/22 by access-group "inside"
    [0x0, 0x0]
2011-04-29T16:50:35-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.50(44420) -> dmz/10.30.30.20(53) [...]
2011-04-29T16:50:51-06:00 ant-fw : %ASA-6-106100: access-list inside
    permitted udp inside/192.168.30.50(38004) -> dmz/10.30.30.20(53) [...]

Let’s consider the first line above. Our suspicious system (192.168.30.105) attempted to connect to port 22/TCP on host 192.168.1.50 and was denied. Inspection of the interface definitions within the ASA’s running configuration show that the “inside” interface is on a “192.168.30.0/24” network, distinct from 192.168.1.0. Given that “192.168.1.50” is a nonroutable IP address which is not part of the local network configuration, perhaps the user of the suspicious system fat-fingered the destination address and typed “192.168.1.50” when he or she meant to type “192.168.30.50.” Lucky for us! If the suspicious system (192.168.30.105) subsequently attempted to access 192.168.30.50, this would not have appeared in the firewall logs because the systems are on the same local subnet. Seven seconds later we see an internal system, 192.168.30.50, made a DNS query, and it made another 16 seconds after that.

While DNS traffic is rightly seen as a query/response protocol, keep in mind that DNS queries themselves often occur in response to some other stimulus on the system. Some process or other needs to resolve something, and triggers a request. From this perspective, all DNS queries should be considered a potential indication of some system- or network-based event, whether easily discoverable or obscure. Commonly, DNS requests occur when a system user attempts to make an outbound connection to a destination station by its name.

Finally, six and a half minutes after port Et0/6 came up on the Cisco ASA, the state changed to “down.”

9.8.5 Timeline

Based on what we’ve discovered so far, we can begin to interleave our evidence into a time-based narrative. Here’s what transpired on 4/29/11 that seems to be of interest to us:

• 16:45:13—Physical port “Ethernet0/6” on the switch changed state to “up” (from firewalls.txt). Working backward from the Cisco ASA’s ARP and CAM table entries (in fw-evidence.txt), we know that this port was subsequently mapped to MAC address 00:26:22:cb:10:17. We also note that the state of this port did not appear to change for the duration of this timeline of events of interest.

• 16:47:35—DHCP address 192.168.30.105 was leased to 00:26:22:cb:10:17 (from dhcp .log).

• 16:47:36-16:47:48—Unknown system with MAC address 00:26:22:cb:10:17 and IP address 192.168.30.105 made what may be DNS queries, followed immediately by NTP connection attempts to an outside server (from firewall.log). The NTP traffic was blocked by policy. The destination address of the NTP traffic implies that this is a non-ICCC Ubuntu system.

• 16:47:48—192.168.30.105 attempted to connect to 6667/TCP on outside “freenode” zelazny.freenode.net (140.211.167.99) and was denied by the firewall (from firewall.log). This is the event that triggered the IDS alert, which was the impetus of this investigation.

• 16:47:49—192.168.30.105 made six more likely DNS queries (from firewall.log).

• 16:48:51—192.168.30.105 made two more likely DNS queries. It also attempted to connect directly to port 22/TCP on the firewall’s inside interface (from firewall.log).

• 16:50:28—192.168.30.105 attempted to connect to port 22/TCP on 192.168.1.50, and was denied by the firewall (from firewall.log).

• 16:50:30—The last ARP reply seen from 00:26:22:cb:10:17 (192.168.30.105) by the switch. This is from fw-evidence.txt, whereby we subtract the age of the last entry for that mapping (138 seconds) from the value of the system clock from the preceding command (16:52:48).

• 16:50:35—192.168.30.50 made a likely DNS query (from firewall.log).

• 16:50:51—Another likely DNS query originated from 192.168.30.50 (from firewall.log).

• 16:51:13—The last time the switch recorded traffic from the suspicious system 00:26:22:cb:10:17, based on the CAM table.

• 16:51:33—Physical port “Ethernet0/6” on the switch changed state to “down.”

• 16:52:48—Diagnostic script pulled data from running firewall. The hunt begins . . .

9.8.6 Theory of the Case

Based on the events of interest that we’ve been able to stitch into a timeline, we need to construct a theory, or theories, that sufficiently explain the observed events, and that can be evaluated with some form of rigor.

9.8.6.1 Summary of Events

It is often best to first summarize what is known about the events observed. These statements should essentially be literal translations of digital records into human-language items, such that anyone familiar with the format of the records would agree as to their meaning. Here is such a summary of our survey of evidence so far:

• At 16:45:13 on 4/29/11, a previously unseen and unidentified device (00:26:22:cb:10:17) physically connected to the Ethernet port in an unoccupied conference room. A few minutes later the device tripped IDS alerts and firewall logs by attempting an outbound connection on port 6667/UDP (likely IRC).

• The ICCC leveraged the data and evidence generated by the “conference area” switch/router/firewall (the Cisco ASA) and determined:

– The device, 00:26:22:cb:10:17, successfully used DHCP to obtain IP address 192.168.30.105 at 16:47:35.

– Categorized as a “noncorporate system” due to its NTP traffic, it attempted policy-prohibited outbound connections for likely NTP, IRC, and SSH services (in that sequence) from 16:47:36 to 16:50:28.

– The rogue device attempted SSH connections to the inside interface of the firewall, which were denied and logged at 16:48:51, and another to nonroutable host 192.168.1.50, roughly a minute and a half later, which was also denied and logged.

– Perhaps related: 192.168.30.50 began sending likely DNS queries approximately six seconds later.

• Roughly one minute later, at approximately 16:51:33, the suspicious device (00:26:22:cb:10:17) physically disconnected from the port in the conference room and was not seen to reconnect to the network.

9.8.6.2 Potential Explanations

Rogue system

It appears that the suspicious device in question is a “noncorporate” or “rogue” system. Based on the evidence, it behaved outside of what would fit with the normal corporate device configuration or corporate policy. It may be the case that the device was a corporate system that was modified to behave inappropriately, or it may be an outside system entirely, trespassing on the internal network. The device’s physical location in an unused conference room further underscores the potential for unauthorized access.

Prohibited connection attempts

The suspicious system attempted to connect via SSH to the internal interface of the firewall. Not only is this disallowed by policy, it is unusual in practice under all but a few circumstances. There are many possible explanations for this unauthorized behavior, but the more common ones include:

• An authorized user (i.e., someone with knowledge of connection parameters) attempted to connect from an unauthorized device.

• An unauthorized user attempted to connect from an unauthorized device. This in turn could be a result of several different activities, including:

– Scanning of the network segment for active and accessible systems.

– Scanning of a host of interest for active listeners.

– Actively attempting to access a system identified as listening, either with knowledge of credentials or by brute-force guessing.

One of the biggest challenges we face in this instance—at least at this moment—is that we haven’t yet gained access to event log sources that might help us distinguish between these activities in order to better understand and interpret the level of threat that we face. If we’re going to sort out what really happened, we need to identify ways to test these competing explanations.

9.8.7 Responses to Challenge Questions

Let’s look at how far we got with the questions we set out to answer to guide our investigation:

• When the suspicious system came online, it might have attempted to acquire a DHCP address. Are there lease assignment logs and, if so, what evidence do they contain?

There were, and from them we were able to see that at 16:47:35 the suspect IP address 192.168.30.105 was leased to 00:26:22:cb:10:17.

• Typically, DHCP servers in enterprises are configured to provide clients with not just an IP address, but also important local configuration information, such as the IP address of the local gateway and DNS server(s). Once received, the client must send broadcast ARP traffic to the local network in order to obtain the corresponding MAC address(es) of the local gateway, DNS server, or other important servers. Can you find evidence that the suspicious system did this?

Yes, we did. The ARP entries in the cache of the firewall were useful in framing the time of our events of interest.

• The CAM table from the switch should be able to tell you which physical port the system is connected to—or was—if you move quickly enough. Can you identify the physical port?

We were able to inspect the switch and, knowing the MAC address we were looking for, identified the physical port it was connected to (Ethernet0/6). (From there, of course, it becomes a physical game of “follow the cable.”¹⁰)

10. Incidentally, this is why it is a good idea for a network forensic investigator to keep a good “tone-and-trace” kit in his or her jump bag.

• Based on the timeline you developed, what activities was the suspicious system engaged in during the time frame of interest?

We were able to see that during the time frame of interest, the suspicious system generated likely DNS, NTP, and SSH traffic. It attempted to make connections that were prohibited by policy. The destination of the logged packets gave us further insight as to the nature of the system. This helped us develop our theory of the case.

9.8.8 Next Steps

Let’s explore some ways we can test each of the possible theories we’ve posited above:

• Interview Authorized Personnel Hopefully, the list of personnel authorized to access the ICCC’s firewalls via SSH is short, so that it is relatively easy to poll each individual to find out whether they attempted to SSH to the firewall from the conference room.

Consider that if our poll is negative, we know that we have, or have had, a likely bad actor present inside the trusted perimeter, both logically and physically. We cannot yet distinguish whether this person is a trusted insider or an external intruder.

• Review IDS and Packet Logs Malicious activities tend to be identifiable by the patterns of stimulus and response upon which they depend to be effective. IDS and other packet logs should be consulted to determine if such patterns exist. If we’re lucky, we can correlate and interpret IDS logs in such a way that we can concentrate our further investigation on the targeted systems, and those most at risk. It may, however, be the unfortunate case that the ICCC has not instrumented their network sufficiently to have recorded scanning activities on the segment of concern.

• Review Authentication Logs It’s probably time to begin looking at host-based authentication logs to determine whether or not accesses were attempted on any other of ICCC’s systems from this suspicious device. Correlating the logs of the host-based firewall with other authentication records for systems on the local segment could reveal patterns that show not only the attacker’s intent, but perhaps also the amount of previous knowledge, and can help scope the extent of the compromise.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.

Table of Contents for Chapter 9. Switches, Routers, and Firewalls

Create new playlist

Sign In

Sign Up

Chapter 9. Switches, Routers, and Firewalls

9.1 Storage Media

9.2 Switches

9.2.1 Why Investigate Switches?

9.2.2 Content-Addressable Memory Table

9.2.3 Address Resolution Protocol

9.2.4 Types of Switches

9.2.4.1 Managed Switches

9.2.4.2 Smart Switches

9.2.4.3 Unmanaged (or “Not-So-Smart”) Switches

9.2.5 Switch Evidence

9.2.5.1 Volatile

9.2.5.2 Persistent

9.2.5.3 Off-System

9.3 Routers

9.3.1 Why Investigate Routers?

9.3.2 Types of Routers

9.3.2.1 Enterprise

9.3.2.2 Consumer

9.3.2.3 Roll-Your-Own

9.3.3 Router Evidence

9.3.3.1 Volatile

9.3.3.2 Persistent

9.3.3.3 Off-System

9.4 Firewalls

9.4.1 Why Investigate Firewalls?

9.4.2 Types of Firewalls

9.4.2.1 Enterprise-Class Firewalls

9.4.2.2 Consumer-Class Firewalls

9.4.2.3 Roll-Your-Own

9.4.3 Firewall Evidence

9.4.3.1 Volatile

9.4.3.2 Persistent

9.4.3.3 Off-System

9.5 Interfaces

9.5.1 Web Interface

9.5.2 Console Command-Line Interface (CLI)

9.5.3 Remote Command-Line Interface

9.5.4 Simple Network Management Protocol (SNMP)

9.5.5 Proprietary Interface

9.6 Logging

9.6.1 Local Logging

9.6.1.1 Console

9.6.1.2 Terminal

9.6.1.3 Buffered

9.6.2 Simple Network Management Protocol

9.6.3 syslog

9.6.4 Authentication, Authorization, and Accounting Logging

9.7 Conclusion

9.8 Case Study: Ann’s Coffee Ring

9.8.1 Firewall Diagnostic Commands

9.8.2 DHCP Server Logs

9.8.3 The Firewall ACLs

9.8.4 Firewall Log Analysis

9.8.5 Timeline

9.8.6 Theory of the Case

9.8.6.1 Summary of Events

9.8.6.2 Potential Explanations

Rogue system

Prohibited connection attempts

9.8.7 Responses to Challenge Questions

9.8.8 Next Steps

Table of Contents for
Chapter 9. Switches, Routers, and Firewalls