Chapter 7. Network Intrusion Detection and Analysis
“IDS is dead.”
—Gartner, 2003
It may seem, based on the title of this chapter, that we’re somewhat behind the times. After all, Gartner famously pronounced intrusion detection dead many years ago,1 asserting in 2003 that intrusion detection systems (IDSs) would be obsolete by 2005 and that everyone would be better off putting their money into preventative technologies (i.e., firewalls). Subsequently, most vendors followed suit, rebranding all of their detection solutions as “intrusion prevention systems (IPSs).” This wasn’t all that difficult to do, as many already included automated remediative actions as configurable options. It wasn’t a magical new technology so much as a marketing strategy for selling a “new” product to existing consumers to replace the “old” in a way that generated revenues that mere feature upgrades couldn’t.
1. Allison Haines, “Gartner Information Security Hype Cycle Declares Intrusion Detection Systems a Market Failure: Money Slated for Intrusion Detection Should Be Invested in Firewalls,” June 11, 2003, http://www.gartner.com/press_releases/pr11june2003c.html.
Meanwhile, on the ground, we’ve been facing the same issues as always, regardless of the acronym. In a very dynamic and fluid environment (say, the Internet) it is sometimes difficult to detect adverse and malicious events, particularly when the authors of such events seek not to have them noticed.
In some cases we can construct sophisticated pattern- or behavioral-matching algorithms that are exceedingly unlikely to miss the bad actor, rarely producing “false positives” (alerting when nothing bad actually happened) or “false negatives” (failing to alert when something bad really did happen)—the worst kind of failure. In such cases automated remediation is a really good idea. However, automated remediation in response to false positives typically results in a self-inflicted denial-of-service condition. The result is typically that the IPS gets “tuned down” a bit, until it is simply less capable of alerting on events that might be of interest.
As investigators, we have to be prepared to evaluate events that might be of interest, and so we often prefer to have the most data available, even that which may not be deemed automatically actionable by an IPS. As a result, we must sift events of interest from the noise.
7.1 Why Investigate NIDS/NIPS?
In an environment that has been appropriately instrumented to detect potentially adverse events via network monitoring, it’s very likely that a NIDS/NIPS alert is the starting point of our investigation. After all, that’s what such deployments are designed to do: tell us when there’s something going on that we ought to look into. Unfortunately NIDS/NIPS can’t always reconstruct a sequence of events and explain them to us, at least not easily. When present, though, they can be as important an event recorder as the “little black box” aviation authorities search for and analyze after every crash, for similar reasons. NIDS/NIPS are often involved in network forensics investigations for the following reasons:
• NIDS/NIPS alerts/logs may include details regarding illicit connections—or even attempts—that are not recorded anywhere else. NIDS/NIPS tend to inspect traffic in ways that even application-layer firewalls may not, and so can provide a richer source of information.
• NIDS/NIPS can be configured to alert on, or at least log traffic that firewalls deem perfectly acceptable. For example, a firewall may not record a port-80 connection to the web server, but the NIDS/NIPS may note if it contains a malicious URI.
• An investigator could potentially modify a NIDS/NIPS configuration to begin detecting events it wasn’t previously configured to record. These devices are usually well positioned as inspection points for network traffic.
• Rarely, the NIDS/NIPS itself might be suspected of compromise.
7.2 Typical NIDS/NIPS Functionality
NIDS/NIPS are specialized sniffers, with the added capability of evaluating captured traffic to determine whether it is malicious or legitimate. There are many different commercial NIDS/NIPS systems on the market. They all behave differently but provide many of the same features. NIDS/NIPS typically include:
• Rules Descriptions of how to compare a packet or stream with known malicious traffic.
• Alerts Lists of suspicious packets/streams.
• Packet captures Certain NIDS/NIPS can be configured to capture suspicious packets and save them for later analysis. They are not always configured to do this by default, due to storage constraints.
7.2.1 Sniffing
At a minimum, a network-based intrusion detection system must have access to network traffic to collect and inspect. This can be done passively, either by connecting it to a mirroring port on a switch or through the use of an inline tap. This can also be done by putting the NIDS/NIPS itself inline between two devices in a choke point position—typically between a set of firewalls or routers.
Once the traffic has been collected, several approaches to its inspection can be leveraged, all of which take some amount of time. If the device is sniffing traffic passively—meaning that the traffic is not purturbed or detained but merely copied on its way by—this delay is of little to no consequence. The delay caused by the inspection is likely to be miniscule compared to the time lag between an alert being sent and a human responder beginning an investigation.
However, if the device is inline in the traffic’s path (typical for a NIPS), and is actively filtering the traffic as it inspects it, the delay can cause noticeable latency. Consequently, many NIPS are not configured to perform as much analysis as a passive NIDS because the latency introduced by the analysis might interfere with business operations.
This is important for the network investigator to understand. While it might be advantageous to begin sniffing and analyzing traffic during the course of an investigation, particularly in ways that weren’t being done previously, modifying an existing NIPS might not be feasible without disrupting the business. In such situations it might be much better to temporarily install a separate passive NIDS.
7.2.2 Higher-Layer Protocol Awareness
In the arms race between attackers and network defenders, attackers began to break malicious traffic into multiple pieces, or encode traffic in sneaky ways, in order to evade detection. In response, NIDS and NIPS developers incorporated higher-layer protocol analysis into their systems. Protocol reassembly and normalization of content based on protocol awareness are two ways in which NIDS and NIPS have evolved to protect against attacks. These techniques are costly in terms of processing power, but effective.
7.2.2.1 Protocol Reassembly
In order for either a NIDS or a NIPS to accurately detect a wide range of malicious traffic, these devices must be programmed to properly parse a wide range of protocols. For example, many years ago a common technique that attackers would use to evade detection by a NIDS was to intentionally fragment the attack traffic. The assumption was that only the end recipient of the traffic—presumably the target—would reassemble the fragments. Consequently we learned that we had to build our NIDS to perform fragment reassembly in order to be able to see and evaluate what the target would receive and process. Likewise it is common for the telltale signs of malicious activity to span multiple segments in a TCP stream. In order for a NIDS/NIPS to accurately inspect such traffic, it must be able to understand TCP’s statefulness and to perform stream reassembly prior to doing content inspection.
It is this actual content inspection that lends the NIDS/NIPS its power. Most simple firewalls can easily be programmed to allow only port 80 traffic to a web server, but if the “legitimate” port 80 connection contains a malicious request, the firewall has no way of knowing. Because the NIDS/NIPS understands the HTTP protocol, it can look inside the request and make sure that it is formatted properly, and alert if it contains something it shouldn’t.
7.2.2.2 Normalization
For certain types of traffic, most notably HTTP, the protocol allows for many different ways of encoding data. For example, the following GET requests are functionally identical, and would be handled by a web server in the same manner (hopefully not allowed!):2
2. Tim Berners-Lee, “RFC 3986—Uniform Resource Identifier (URI): Generic Syntax,” IETF, January 2005, http://rfc-editor.org/rfc/rfc3986.txt.
GET /../../../../etc/passwd HTTP/1.1
GET %2f..%2f..%2f..%2f..%2fetc%2fpasswd HTTP/1.1
GET %2f%2e%2e%2f%2e%2e%2f%2e%2e%2f%2e%2e%2f%65%74%63%2f%70%61%73%73%77%64
HTTP/1.1
This is an example of a “directory traversal” attack, whereby if the webserver is buggy, it can be convinced to traverse up and out of its document hierarchy, and then display the system /etc/passwd file. A NIDS/NIPS that understands the GET method of the HTTP protocol should have no problem identifying such an attempt. However, given that there are a nearly infinite number of ways to encode any particular malicious string, we have two options: either write a “signature” to detect each and every one of the various ways such a string can be represented, or first “normalize” each string to its “canonical” form, and only then compare them against the list of “known bad” things.
7.2.3 Alerting on Suspicious Bits
The different ways that NIDS/NIPS are typically built to perform detection are described below. All NIDS/NIPS can be configured to communicate alerts via various means. The most common of these include:
• Sending email alerts
• Logging events to a syslog server
• Sending SNMP traps
• Logging events directly to a queriable database
Additionally, a NIDS/NIPS sensor may store alert and event data locally, although as with other network devices, storage space is likely to be fairly limited, and so these logs will likely be kept for only short periods of time.
7.2.3.1 Fidelity
The manner in which the alerts are recorded and delivered can matter greatly when considering the value of the data/evidence produced. Alerts logged via syslog or SNMP traps are normally fairly “lo-fidelity” in that very little detail about the event is available. Database logging (usually as part of some sort of proprietary event aggregation system) can often produce much more detail about how and why the event was detected and determined to be of interest.
At the other end of the spectrum, some NIDS/NIPS can produce very “hi-fidelity” event logs, capturing and storing the packets in their entirety (usually in libpcap format). This can allow an analyst or investigator the ability to see the actual traffic that caused the alerts, and perhaps best understand why it was deemed “bad” by the NIDS/NIPS.
7.3 Modes of Detection
NIDS/NIPS typically work in three basic ways: using signature-based analysis, protocol awareness, and/or behavioral analysis techniques.
7.3.1 Signature-Based Analysis
Signature-based analysis is the oldest and most common strategy. This method compares headers, contents of packets, and streams of packets against databases of known, malicious byte sequences in order to identify suspicious traffic. Individual packet analysis does not always work because malicious activity may be spread across groups of packets. Hence, signature-based NIDS/NIPS have evolved to inspect related traffic that spans multiple fragments of packets, or multiple segments in a stream.
7.3.2 Protocol Awareness
Normal traffic typically conforms to RFC specifications. Malicious traffic often does not. Attackers often intentionally bend protocols in order to evade detection. Just as the arms race exists between attacker and defender, it also exists between attacker and detector. Protocol-aware NIDS/NIPS reassemble fragments (Layer 3), reassemble streams (Layer 4), and even reconstruct entire protocols (Layer 7) because they have to understand how the endpoint of the communication will interpret the data. It’s possible for protocols to be broken in a nonmalicious way by a misconfigured or malfunctioning device. However, protocol abuse is often a symptom of malicious intent. Either way, it is worth detecting.
7.3.3 Behavioral Analysis
Behavioral analysis is a more recent development in NIDS. The idea behind this approach is to spend time building a “baseline” of normal network behavior against which to compare future behavior. The system “learns” what is normal, in order to later detect aberrations. Alerts may be triggered, for example, if the volume of data per transaction going out of your web server is two standard deviations greater than the expected mean. Though there is no indication of protocol violations, or that the content of the web transactions matches any known bad behavior, this could nonetheless be a very important event to be made aware of.
7.4 Types of NIDS/NIPSs
NIDS/NIPS can be sold as a commercial product or built in-house. Commercial NIDS/NIPS are usually made from special hardware designed to improve performance. “Roll-your-own” NIDS are often built from open-source software such as Snort, which gives users the option to download free signature updates, or purchase a subscription that provides more timely access to updates.
7.4.1 Commercial
Regardless of what Gartner may have predicted in 2003, the commercial market for IDS is very much alive and well, with many of the top IT vendors having made significant acquisitions to cover their positions relative to competitors. One of the early leaders in detective technologies was Marcus Ranum’s “Network Flight Recorder” (NFR), which was bought by Check Point (after a failed attempt to purchase Marty Roesch’s “Snort”) and which presumably has been adopted as the core technology behind Check Point’s IPS-1 product. Notably, NFR was one of the few NIDS that offered an open language for specifying packet and traffic signatures, allowing NFR Security’s customers to build their own custom rules (though the open-source Snort was evolving a simpler and more widely adopted language around the same time).
Other long-time players in this market have evolved their product lines as well. Cisco’s “Secure IDS” has morphed into “Cisco IPS” and Enterasys’ Dragon is now “Enterasys IPS.” IBM purchased Internet Security Systems (ISS), a market leader in both IDS and network vulnerability scanning technologies. ISS’s RealSecure sensor has morphed into the “IBM Security NIPS,” which is now offered as a component of IBM’s Tivoli suite of enterprise management tools. Meanwhile, TippingPoint’s popular product was subsumed by Hewlett-Packard.
Sourcefire continues to stand alone as the commercial vendor/licenser of Snort, which in its various for-profit and open-source forms undoubtedly leads the market as the single most widely deployed NIDS/NIPS technology on the planet. Consequently, we’ll look at its architecture in depth below.
As of the time of this writing, some common commercial products network forensic investigators are likely to encounter include (listed alphabetically):
• Check Point IPS-13
3. “Check Point IPS-1—intrusion detection & prevention system (IDS/IPS),” 2011, http://www.checkpoint.com/products/ips-1/index.html.
• Cisco IPS4
4. “Cisco Intrusion Prevention System—Products & Services—Cisco Systems,” 2011, http://www.cisco.com/en/US/products/sw/secursw/ps2113/index.html.
• Corero Network Security5
5. “Intrusion Prevention System (IPS)—Stop DoS, DDoS, SYN Flood, and Cyber Attacks,” 2011, http://www.corero.com/content/products/intrusion_detection/attack_mitigator.jsp.
• Enterasys IPS6
6. “Intrusion Prevention System, Secure Networks—Enterasys,” 2011, http://www.enterasys.com/products/advanced-security-apps/dragon-intrusion-detection-protection.aspx.
• HP TippingPoint IPS7
7. “HP Networking products: switches, wireless, routers, TippingPoint security, network managment, uc&c,” 2011, http://h10163.www1.hp.com/products_ips.html.
• IBM Security NIPS8
8. “IBM—IBM Security Network Intrusion Prevention System—Software,” 2011, http://www-01.ibm.com/software/tivoli/products/security-network-intrusion-prevention/.
• Sourcefire 3D System9
9. “Next-Generation Intrusion Prevention System (NGIPS) | Sourcefire Cybersecurity,” 2011, http://www.sourcefire.com/security-technologies/cyber-security-products/3d-system.
7.4.2 Roll-Your-Own
It is clear why Snort is the most deployed NIDS/NIPS technology: it is both free for use under the GNU Public License (GPL), and at the same time supported by commercially funded research and development efforts. Consequently you may encounter Snort used “unofficially” even within corporate and government environments that have opted to purchase another commercial vendor’s offering as an “official” solution.
As with any other IT product niche, commercial support, development, and ease of integration can provide compelling reasons to write checks. However, as an investigator, you should be aware that you may find custom, “roll-your-own” solutions sitting side-by-side with a commercial deployment. These devices are often more flexible in their use (being “unofficial” anyhow), and can more easily be refactored or repurposed during the course of an investigation—and yet the evidence they collect can be no less valid and useful.
For a long time, Snort was the only real open-source NIDS/NIPS capable of industrial-scale use, though it appears that with some National Science Foundation funding and an increasingly energetic following, the Bro system is gaining both in competitiveness and adoption.
Again, here are the most common noncommercial systems you should expect to encounter, with references to more information about each. Furthermore, these are systems that you can and should be prepared to work with during an investigation. Pretty much everything you need to know about how to do so is available in online manuals, and both systems have robust communities surrounding them:
• Snort10
10. “Snort,” 2011, http://www.snort.org/.
• Bro11
11. “The Bro Network Security Monitor,” 2011, http://bro-ids.org/.
Though both are open-source and free for use, they are really not equivalent projects in either scope or goal. Due to its commercial ownership and support, Snort is much more targeted toward enterprise, production deployment. Meanwhile Bro’s development team states overtly that it “has been developed primarily as a research platform for intrusion detection and traffic analysis.” These are very complementary goals, and by all appearances, are very complementary projects where either research or raw investigation is the goal.
7.5 NIDS/NIPS Evidence Acquisition
Given the wide variation in hardware and software used to build NIDS/NIPS, the precise evidence that forensic investigators can gather, and the methods for gathering it, can vary considerably. In this section, we review the general categories of evidence that can be gathered from NIDS/NIPS and discuss the value to investigators and common interfaces for devices.
7.5.1 Types of Evidence
Depending on the vendor, alerts, rules, and packet captures may be accessed via local devices’ file systems, web management interfaces, client software, email, etc. In general, the evidence that we may be able to gather from NIDS/NIPS includes:
• Packet header and/or flow record information
• Packet payloads
• Activities correlated across multiple sensors
7.5.1.1 Configuration
The configuration of the NIDS/NIPS is fundamentally important for any analysis of alerts from the device. For example, suppose your investigative environment has a number of IDS sensors deployed throughout the environment. A very critical alert resulted in escalation and investigation. The alert in question was only seen in one critical segment. Is that because the events that triggered it only occurred in that segment, or because sensors in other segments weren’t configured to look for those events? How would your strategy and the outcome differ depending on that answer? Without evaluating the relative configurations of the sensors in the various segments, you have no way of knowing what the possible scope of events may be.
If a NIDS/NIPS has not been configured to alert upon a particular event, then the absence of alerts for that event is meaningless. Similarly, it is not possible to determine the meaning of an alert with certainty, or evaluate whether it is a false positive or negative, unless you can see precisely what rule triggered the alert. What you do or do not see coming from an IDS system is wholly dependent upon where the device is placed and what it has been configured to alert upon. For this reason, we recommend that you gather device configuration as part of NIDS/NIPS evidence acquisition.
In addition to gathering the boot configuration of the device (usually found in persistent storage on the device), it’s important to gather the current running configuration whenever possible. Depending on the NIDS/NIPS software, an administrator may be able to modify the configuration while the system is running, without writing these changes to disk. In these instances, the running configuration may be quite volatile, and unless it is manually written to persistent storage, it will be lost when the device is powered down.
7.5.1.2 Alert Data
The purpose of a NIDS is to evaluate network traffic based on a set of rules or learned patterns, identify traffic of interest, and produce alerts that humans can act upon. NIDS/NIPS produce alerts in various formats, ranging from text files to emails to messages in pretty graphical user interfaces.
Different NIDS/NIPS systems categorize, label, and prioritize alert data differently. Even when alerts are aggregated from multiple sources, it can be a real challenge to discover that the manner in which a NIDS/NIPS reports an event on one segment of a network is not at all the same way an older model on a different segment reports it. There are Security Information and Event Management (SIEM) systems that seek to “normalize” event data from disparate sources, but this is often a problem that ends up being solved by humans, hunkered over lots of events, trying to puzzle them out.
7.5.1.3 Packet Header and/or Flow Record Information
When a packet or flow triggers a NIDS/NIPS alert, the device may log corresponding packet header and/or flow record data. This can help you identify the origin, destination, and patterns of activity. It can also help you correlate events from multiple sources.
7.5.1.4 Content Data
In some instances, the NIDS/NIPS may be configured to capture full packet contents from packets that trigger alerts, and often a certain number of subsequent packets. Packet contents can be invaluable to the investigator, allowing you to carve out the malware that triggered an alert or view the actual obfuscated Javascript that an attacker attempted to serve a client.
Regardless of how hard it can be to sort through volumes of data to figure out what’s going on, it is usually infinitely more difficult to figure out the nature of events given too little information about them. Full content captures are limited by the amount of disk space available on the NIDS/NIPS, in addition to processing power.
7.5.1.5 Activities Correlated Across Multiple Sensors
Topologically and temporally correlated event data is exceptionally useful. Unfortunately, it is not always available. Relatively few organizations are set up to properly aggregate security event data, and for good reason: The products that can correlate NIDS/NIPS data effectively are often prohibitively expensive, and require even more expensive staff and skills to monitor and maintain. This is challenging for small- and mid-sized operations. Even large-scale enterprises don’t always go all in with such technologies.
An increasingly common practice is for enterprises to deploy NIDS/NIPS sensor instrumentation, but then to direct all of its output to a third-party monitoring service. As an investigator, you’ll need to determine very quickly if this is being done, as you’ll almost certainly want to work with the outsourced service provider to gain access to a comprehensive view of correlated events. After all, this is what they’re being paid to provide, and it sure beats repeating their work. Even if you are working with an outsourced provider, be aware that NIDS/NIPS data can be either inaccurate or incomplete, or both. Corroboration is important no matter where your evidence comes from.
7.5.2 NIDS/NIPS Interfaces
These days, NIDS/NIPS are almost uniformly designed to be information aggregation devices, so they are typically deployed with a central analysis console. This is certainly convenient for the forensic investigator; it’s far better to collect evidence from one central source than from multiple disparate devices, perhaps in far-flung locations.
A standalone NIDS/NIPS sensor may be used without aggregation, as is common with point-deployments of Snort and other low-cost systems. In this situation, you may find a great deal of data without a convenient central analysis tool. The biggest challenge will be to manually aggregate and correlate the data.
Depending on the particular NIDS/NIPS, evidence can be gathered via a graphical user interface (GUI), command-line interface (CLI), or a remote log server. Please see Chapter 3 for a detailed discussion of active evidence acquisition.
7.5.2.1 GUI Interfaces
Nearly all modern NIDS/NIPS systems can be accessed, inspected, and configured via some sort of graphical user interface, whether it’s a web application that can be accessed by a web client, or a proprietary interface that requires a special client. The degree of inspection and configuration supported by such interfaces varies, of course, but some vendors strive to expose every last bit of functionality this way, from signature configuration to alerts generated. Sometimes you can use the GUI to download the actual packet traces.
7.5.2.2 CLI Interfaces
It may be possible to log onto the NIDS/NIPS via SSH or direct console connection and use the command-line interface to view configuration, logs, and alerts. Some devices have historically provided constrained access to a subset of functionality via the command-line, while others expose entire application programming interfaces (APIs) for nearly infinitely extensible customizations.
As an example, the NIPS sensors that Sourcefire sells commercially run the same version of Snort that can be downloaded for free, though they provide an array of configuration and management utilities in the commercial devices that make enterprise deployments much simpler and more manageable. Many of these utilities leverage a library of custom Perl objects that can be used by anyone with the appropriate access credentials and the skills to employ them (check licensing terms and support contracts before doing so).
7.5.2.3 Off-System Logging
These days, both commercial offerings and open-source systems alike are capable of sending at least some of the critical evidence to an aggregation repository. Many NIDS/NIPS are also configured to send alerts via email to system administrators. You may need to inspect the NIDS/NIPS configuration in order to figure out where the system is sending logs (passive analysis of the traffic from the device may also do the trick). The local device configuration will also provide you with an understanding of the level of detail and fidelity of the remote logging.
7.6 Comprehensive Packet Logging
We’ve discussed various ways that packet activity can be logged: routers and firewalls can log the packet event with source and destination information; NIDS/NIPSs can record the total packet content from metadata (header info) down to the payload contents.
However, where full content packet logging is occurring on NIDS/NIPS, almost uniformly the only packets being captured and made available for future inspection are the ones that trigger actual alerts. NIDS/NIPS generally are not configured (or configurable) to capture the packets that preceded an incident, nor the ones that followed. That can pose a serious problem for the investigator: often very little can be gleaned from a single packet, as too much of the context is unavailable.
One way of countering this problem is to intentionally construct a persistent packet sniffing device. This is a sniffer configured to record every one and zero that traverses the link it monitors. It’s job is not to alert on any specific condition, but rather to ensure that it has recorded all of the data available, along with critically accurate timestamp information.
The benefits of such a device are instantly obvious. Whereas NIDS/NIPS rarely capture much more than a single packet per event (the one that tripped the trigger), that single packet can subsequently be located in a comprehensive packet log, and its context examined. The stream that contained it, if it was part of a stateful session, can be isolated and reconstructed. If it was part of a stimulus, the response may be located and investigated. If it was part of a response, perhaps the stimulus can be determined.
All of the previous caveats with respect to packet captures still apply, however:
• It can require a lot of CPU.
• It can require a lot of disk.
• It can also pose a huge security risk!
7.6.0.4 Evidence that May Be Available
The possession of full-content packet captures, that contain packets which correlate with NIDS/NIPS alerts, is pretty much the dream scenario. There is virtually nothing left wanting—so long as time skew and time zone can be dealt with. As to the former, many tools can be programmatically configured to adjust for skews—either on input, or on display, or both—and to aggregate based on an absolute time (such as GMT-0). We discuss the latter issue in Chapter 8. To summarize the bounty:
• Packet headers
– Suitable for traffic analysis
– Similar to pen registers, trap/trace
• Packet payloads
– Suitable for full transaction reconstruction
– Similar to full wiretap
When such packet captures are available, they can be used in conjunction with other event data (such as NIDS logs) in order to reconstruct entire events. If full payloads are stored, entire sessions can be reconstructed, and files recovered intact. Perfect-fidelity copies of evidence can be acquired in this way, every bit as successfully as from the hard drive to which they were written, or from which they were transferred.
Since packet capturing is usually performed with tcpdump or another libpcap-based tool retrieval and analysis of the logs is usually performed with libpcap-based tool as well. The Berkeley Packet Filter language becomes critical for such analysis, as finding the critical traffic can pose an overwhelming task. See Chapter 3 for details.
7.7 Snort
As discussed, the open-source Snort product is arguably the single most widely distributed NIDS in the world. It is sometimes used alongside commercial products. In this section we focus on Snort in particular depth because of its large market share.
Snort is a libpcap-based utility, and therefore shares all of the functionality that tcpdump and others possess. However, rather than being limited to Layer 2 through Layer 4 analysis, Snort has the capability to do much deeper packet inspection. It can analyze arbitrary network traffic using two basic techniques: protocol analysis and signature analysis.
Snort’s rule language is open source, so we can easily discuss how rules are written to find individual packets. Anyone who has used tcpdump on a busy network interface is quickly overwhelmed by the fact that there is so much traffic it’s difficult to find what you’re looking for. Snort is a tool that illustrates our ability to find the needle in the haystack.
As our friend Rob Lee says, “If you know what you’re looking for and where to look, nothing is hidden.”
An overview of Snort:
• Most widely used NIDS
• Open-source code
• Open rule language
• Extremely versatile
• Actively improving, partly due to commercial support
7.7.1 Basic Architecture
Snort’s basic architecture is as follows:
• Snort pulls packets in using libpcap, from whichever interface is specified, either at the command-line or in the configuration file.
• Snort passes all packets through preprocessors for reassembly and protocol analysis. These preprocessors roughly mirror the OSI layers:
– At Layer 3, it reassembles fragments
– At Layer 4, it reassembles streams
– At Layer 5, it reassembles circuits/sessions
– At Layer 7, it reassembles transactions
• If at any of those layers, Snort detects anomalies, it can alert. The degree to which Snort does protocol analysis and reconstruction is configurable.
• After all of the preprocessing has been completed, the analyzed information is handed off to the Snort rule engine. This engine can then use any and all of the protocol information, as well as the payload contents (parsed or raw) to detect malicious traffic.
• The output engine is then invoked to determine how any resulting alerts will be communicated to the end-user. This can range from Snort’s native text-based alerting to syslog and SNMP alerts.
Any packet that triggers an alert, either through preprocessors or signatures, is by default captured in libpcap format, and can be used by analysts to further investigate the cause of the alert. Snort can be configured to capture subsequent related traffic.
7.7.2 Configuration
By default on Linux systems, the important Snort files and directories are:
• /etc/snort/snort.conf This file is where global values for Snort are declared, including the internal/external network definitions, how the preprocessors will be configured, how the output processors will be configured for logging, and the inclusion of major chunks of rules.
• /etc/snort/rules/ This directory contains the rules files themselves. Within each file, rules can be individually disabled or enabled.
• /var/log/snort/ This directory contains the native alerts file, where Snort records text-based alerts, and the libpcap files with corresponding packet captures.
The locations of these Snort files/directories are fully configurable.
7.7.3 Snort Rule Language
Snort rules are defined by text-based language, one line per rule. These rules are contained in numerous files, which by default contain thousands of available rules distributed by Sourcefire’s Vulnerability Research Team (VRT). Each rule may be enabled or disabled by default, based on what the VRT deems is useful in a typical environment. All Snort rules must contain a Snort ID (SID), which identifies that particular rule. SIDs lower than 1 million are reserved for the Sourcefire VRT; SIDs between 1 and 2 million are reserved for a freelance, open-source community called “Emerging Threats.”12 Local rules can be built, but should always be numbered greater than or equal to 2 million. When creating customized local rules, best practice is to copy an existing rule to the local rules file, disable the commercial rule, and enable the modified local version. This way if the original rule is updated by the VRT, the update won’t clobber your local changes.
12. “Emerging Threats,” 2011, http://www.emergingthreats.net/.
7.7.3.1 Rule Header
The Snort rule header consists of seven fields, space delimited, as follows:
• Action: This field describes what the Snort sensor should do when a packet is discovered to match the rule. Typical actions are alert, log, pass, and drop.
• Protocol: This field delineates the protocol the packet must be employing to match the rule. Possible values are icmp, tcp, udp, or the general case, ip.
• Source IP/network and port: These fields describe what the source of a packet must be to qualify as a match.
• Directionality operator: This field allows the rule author to specify whether a match can occur in one direction only (from source to destination) or bidirectionally. The allowed values are “->” and “<>”.
• Destination IP/network and port: These fields describe what the destination of a packet must be to qualify as a match.
Here are three examples of Snort rule headers, which we examine in turn.
• alert tcp any any -> 192.168.2.1 80 (...)
For any TCP segment that is from any source at all and that is destined specifically to port 80 on host 192.168.2.1, compare it with the rule body. If a match is found, both log the packet and issue an alert.
• log udp 192.168.1.1 53 -> !192.168.1.0/24 any (...)
For any UDP datagram that originates from port 53 on host 192.168.1.1 and that is destined for anywhere except the 192.168.1.0/24 network, compare it with the rule body. If a match is found, log the packet, but do not alert on the event.
• drop ip $EXTERNAL_NET any <> $HTTP_SERVERS $HTTP_PORTS (...)
Here we notice variable names instead of sources and destinations. This is the most common practice when writing rules, as keeping specific network ranges defined in a single place (typically the snort.conf file) allows us to change the network topology or the location of the sensor without having to edit all the rules. In this case we assume the rule we’re looking at is used in an in-line IPS, because the action is “DROP.” This rule will evaluate any IP packet against the rule body if it is coming from an external source and destined for the predefined HTTP servers. If the rule body also matches the packet, the packet is dropped.
7.7.3.2 Rule Body
The Snort rule body consists of a semicolon-delimited list of rule “options” bracketed by parentheses:
• General options (metadata about events) provide a way of specifying information about the rule, it’s purpose, revision, etc. These are often referred to as “metadata” options.
• Detection options (stepwise instructions for matching packets or streams) include:
– “nonpayload” detection specifications (based on Layer 3 and Layer 4 protocol fields) such as: TTL, TOS, sequence and acknowledgment numbers (for TCP), ICMP types and codes, etc.
– “payload” detection specifications (based on Layer 5 content and above in IPv4) such as: content, pcre (Perl-compatible regular expression), depth, offset, and many more.
• Post-detection options (what to do if there is a match) describe on a rule-by-rule basis actions to take if a match is encountered.
General Rule Options (metadata about events) include:
• msg: This is the descriptive title that will be attached to any resulting alerts. It should make clear to the intrusion analyst what the event is about, at a glance.
• sid: The required Snort ID number that uniquely identifies the rule.
• rev: The rule’s revision number. The combination of “sid” and “rev” define rules uniquely.
• reference: An optional pointer to background information, often a CVE number, Bugtraq ID, or URL. While not required, this option is typically one of the most useful metadata options, as it can help the analyst understand why a rule was written in the first place, and what it’s testing.
Note: Not all “options” are optional!
Nonpayload Detection Rule Options essentially expose for examination and comparison any and all of the protocol data that has been dissected and evaluated by the various preprocessors as the packet is passed up the stack. These options are the most efficient for the Snort engine to evaluate, as they operate on data that has already been structurally parsed and preserved (rather than on laborious searches through unparsed payload data). As a consequence, they are often relied upon as heavily as possible in order to weed out the packets that should not be given further, less efficient deep-packet inspection. The best example of this is the use of flow tracking by the TCP state engine. If a particular type of maliciousness can only manifest itself as part of a connection from a server to a client, and not vice versa, then monitoring the appropriate simplex side of the connection is all that is required.
Nonpayload detection rule options include:
• Comparison operators for all of the protocol fields in the:
– IP packet header (TTL, fragmentation information, embedded protocol, IP options, etc.)
– TCP segment header (TCP flags and stateful flow inspection, sequence, and acknowledgment numbers, window size, etc.)
– ICMP header (ICMP type and code, as well as sequence values)
• Comparison operators for sameip, stream_size, RPC call number
Payload Detection Rule Options are used to inspect the contents of the payload of packets and streams in intelligent ways in order to determine if any packets match known bad signatures. The payload contents can be matched against ASCII strings, hexadecimal (binary) sequences (including for non-ASCII encodings), and also Perl-compatible regular expressions (PCRE). Such comparisons can be made against the payload as a whole (from end to end) or against sections of the payload selected by either relative or absolute position in order to facilitate Layer 7 protocol parsing. Additionally, some amount of Layer 7 protocol awareness comes built in. For efficiency’s sake, you can search for content in the parsed-out URL of HTTP transactions, rather than having to search for it in every packet’s entire payload.
Payload detection rule options include:
• Content matching for:
– ASCII strings
– Binary sequences
– PCREs
• Layer 7-specific protocol data, such as HTTP URIs and SMTP commands
• Absolute- and relative-positional searches, based on previous content matches
Post-Detection Rule Options exist to give rule authors the ability to translate rule matches into specific actions on a rule-by-rule basis, which then overrides the global Snort configuration. Such actions include:
• Causing the alert and packet logging for the given rule to be handled in a different way than other alerts.
• Triggering the capture of some or all of the subsequent packets in a stream after a single packet has caused a rule to fire.
• Active response mechanisms, including the bidirectional reset of TCP connections and the sending of ICMP Destination Unreachable packets, spoofed at each end of the virtual circuit.
7.7.4 Examples
In this section we consider a couple of fairly simple Snort rules, examine packets that would trigger them, and then inspect the alerts that would result. Understanding how these three items correlate is very useful for any investigator.
• Snort Rule #1
alert icmp $EXTERNAL_NET any -> $HOME_NET any (msg:"ICMP PING"; icode:0;
itype:8; classtype:misc-activity; sid:384; rev:5;)
Above, we see a rule that will alert on any inbound ICMP traffic that is of type 8 code 0: an “Echo Request.” It is the fifth revision of the VRT’s rule number 384, and has been assigned a “classtype” that confers a priority of “3” (although you’d have to look at the Snort configuration to know that last part).
• Snort Packet #1
03:12:08.359790 IP 10.0.1.10 > 10.0.1.254: ICMP echo request, id 32335,
seq 0, length 64
0x0000: 4500 0054 eac8 0000 4001 78d9 0a00 010a [email protected].....
0x0010: 0a00 01fe 0800 75e4 7e4f 0000 e801 e349 ......u.~O.....I
0x0020: 487d 0500 0809 0a0b 0c0d 0e0f 1011 1213 H}..............
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
Here we see an ICMP packet that is indeed an Echo Request. If we presume that 10.0.1.10 is part of the $EXTERNAL NET definition, and 10.0.1.254 is within the $HOME NET range, we would expect a match and an alert.
• Snort Alert #1
[**] [1:384:5] ICMP PING [**]
[Classification: Misc activity] [Priority: 3]
04/13-03:12:08.359790 10.0.1.10 -> 10.0.1.254
ICMP TTL:64 TOS:0x0 ID:38125 IpLen:20 DgmLen:84
Type:8 Code:0 ID:32335 Seq:1 ECHO
This is the resulting alert. The first line of all Snort alerts, as presented in the native text-based output, contains four items. The first and last are reputed to be ASCII-art representations of the Snort pig’s snout. Between them is a square-bracketed, colon-delimited set of three numbers.
The first of these three numbers is the “generator ID” (GID), which specifies the part of the Snort architecture that produced the alert. In this case, the GID is “1,” which denotes the Snort rule engine. Keep in mind that all of the preprocessors can generate alerts as well. Each has its own GID. The second number is the SID, and the third number is the revision, both exactly what we’d expect to see: SID 384 Rev 5. The remainder of the first line is the message option, which also matches what we’d expect to see based on the rule itself.
The remaining information in the alert paragraph above should be fairly straightforward to interpret. In the second line, we see the classtype and priority, and in the third line, we see the date and time stamp, followed by the source and destination IP address. On the fourth line, we see more Layer 3 header data, including encapsulated protocol (ICMP), time-to-live value, type-of-service (TOS) field value (presented in hexadecimal, as the TOS byte is really a series of smaller fields that must be interpreted in binary), the IP identification value (in decimal), and the IP header and total datagram lengths (also in decimal). Finally, on the last line we see the higher-layer (encapsulated) protocol information, which in this case is the ICMP type and code, along with the ICMP identification and sequence values.
• Snort Rule #2
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"WEB-MISC /
etc/passwd"; flow:to_server,established; content:"/etc/passwd";
nocase; classtype:attempted-recon; sid:1122; rev:5;)
Let’s look at a rule with a few more options. The rule header should be no more difficult; without looking in the Snort config, we can’t know what the $EXTERNAL_NET, $HTTP_SERVERS, and $HTTP_PORTS variables are specifically defined to be, but we can certainly understand the intent. Looking at the rule body, we see that a match can only occur on a properly stateful TCP segment sent in the simplex channel from the client to the server as part of an established connection. If and only if that first condition is met, Snort will next evaluate whether the string “/etc/passwd” is found within the packet payload, regardless of it being uppercase, lowercase, or a mix of either. We also see a classtype assignment, and of course the SID and revision numbers.
• Snort Packet #2
05:11:46.015420 IP 192.168.1.50.38097 > 172.16.16.217.80: P
2098193630:2098193743(113) ack 1055208924 win 1460 <nop,nop,timestamp
109823750 109820452>
0x0000: 4500 00a5 23dd 4000 4006 97b2 c0a8 0132 E...#.@[email protected]
0x0010: ac10 10d9 94d1 0050 7d0f e4de 3ee5 35dc .......P}...>.5.
0x0020: 8018 05b4 8803 0000 0101 080a 068b c706 ................
0x0030: 068b ba24 4745 5420 2f65 7463 2f70 6173 ...$GET./etc/pas
0x0040: 7377 6420 4854 5450 2f31 2e30 0d0a 5573 swd.HTTP/1.0..Us
0x0050: 6572 2d41 6765 6e74 3a20 5767 6574 2f31 er-Agent:.Wget/1
0x0060: 2e31 302e 320d 0a41 6363 6570 743a 202a .10.2..Accept:.*
0x0070: 2f2a 0d0a 486f 7374 3a20 7777 772e 736b /*..Host:.www.sk
0x0080: 6574 6368 792e 6576 6c0d 0a43 6f6e 6e65 etchy.evl..Conne
0x0090: 6374 696f 6e3a 204b 6565 702d 416c 6976 ction:.Keep-Aliv
0x00a0: 650d 0a0d 0a e....
The output above represents a single packet. If the source, destination, protocol, and directionality match the rule’s header, the packet will be compared against the rule body. If we assume that the packet contains a segment in an established TCP stream, and it was sent from a client to a server, then the packet contents will be inspected for the string “/etc/passwd.” (The etc/passwd file on Linux and UNIX systems contains sensitive account information that is useful to attackers.) If the inspected packet matches this rule, then the system will log an alert with the message “WEBMISC /etc/passwd.” Inspecting the payload, we do in fact see the string “/etc/passwd” in the payload beginning at byte offset 56 (0x0038), as part of a GET request.
• Snort Alert #2
[**] [1:1122:5] WEB-MISC /etc/passwd [**]
[Classification: Attempted Information Leak] [Priority: 2]
04/06-05:11:46.015420 192.168.1.50:38097 -> 172.16.16.217:80
TCP TTL:64 TOS:0x0 ID:9181 IpLen:20 DgmLen:165 DF
***AP*** Seq: 0x7D0FE4DE Ack: 0x3EE535DC Win: 0x5B4 TcpLen: 32
TCP Options (3) => NOP NOP TS: 109823750 109820452
Finally, we see the alert that results. The basic structure of the information in the first three lines of the art has not changed. We can identify the rule that caused the alert by its SID and Revision numbers (1122 and 5, respectively), and see the name of the alert clearly. Its classtype and priority follow, as does the timestamp and socket data, after which we see the Layer 3 information presented as before. Again we see the Layer 4 protocol data—this time TCP protocol values.
7.8 Conclusion
NIDS/NIPS are specifically designed to sift through large amounts of network traffic and pick out specific events of interest, particularly those that relate to security. As a result, they are often the first place investigators turn for information on a case. In fact, NIDS/NIPS alerts are often the triggers that cause the investigation to be launched. Investigators can leverage evidence gathered from NIPS/NIDS to find the needle in the haystack.
In this chapter, we reviewed typical NIDS/NIPS functionality, including sniffing and higher-layer protocol analysis. We discussed modes of detection, different types of NIDS/NIPS, and the various types of evidence that can be recovered from them. We examined Snort, a popular open-source NIDS/NIPS, and concluded with examples of Snort rules, captured packets, and alerts.
7.9 Case Study: InterOptic Saves the Planet (Part 1 of 2)
7.9.1 Analysis: Snort Alert
Using the Snort alert file provided, let’s examine any instances of the “SHELLCODE x86 NOOP” alert:
$ grep -A 4 'x86 NOOP' alert
[**] [1:10000648:2] SHELLCODE x86 NOOP [**]
[Classification: Executable code was detected] [Priority: 1]
05/18-08:01:45.591840 172.16.16.218:80 -> 192.168.1.169:2493
TCP TTL:63 TOS:0x0 ID:53309 IpLen:20 DgmLen:1127 DF
***AP*** Seq: 0x1B2C3517 Ack: 0x9F9E0666 Win: 0x1920 TcpLen: 20
This is the alert that security staff initially provided us and, based on the output of “grep,” it appears that there are no other alerts of this type in the packet capture.
From this alert, we can see that a remote server, 172.16.16.218:80, sent traffic to the local system 192.168.1.169:2493. “Executable code” was detected in this traffic. The payload of the IP packet appears to be a TCP packet. According to IANA, port 80/TCP is assigned to “World Wide Web HTTP.”13 Although this is not a guarantee that the traffic is HTTP, it is a strong clue.
13. “Port Numbers,” July 11, 2011, http://www.iana.org/assignments/port-numbers.
7.9.2 Initial Packet Analysis
Let’s pull the corresponding packet out of the associated capture so that we can examine it more thoroughly. Using tcpdump and the BPF language we can filter on the IP ID field using the value shown in the alert (“53309”):
$ tcpdump -nnvr tcpdump.log 'ip[4:2] = 53309'
reading from file tcpdump.log, link-type EN10MB (Ethernet)
09:01:45.591840 IP (tos 0x0, ttl 63, id 53309, offset 0, flags [DF], proto
TCP (6), length 1127) 172.16.16.218.80 > 192.168.1.169.2493: P, cksum 0
x2de5 (correct), 455882007:455883094(1087) ack 2677933670 win 6432
This certainly appears to be the packet that caused the alert. In addition to the IP ID number that matches the alert, further corroboration is provided by the sequence and acknowledgment numbers. The TCP sequence number shown in the tcpdump output above (“455882007”) matches the TCP sequence number in the alert (“0x1B2C3517,” which is “455882007” in decimal). Similarly, the acknowledgment number in the tcpdump output is “2677933670,” which matches the acknowledgment number in the Snort alert (“0x9F9E0666,” or “2677933670” in decimal).
Now let’s examine the packet’s contents (in ASCII to begin with):
$ tcpdump -nnAr tcpdump.log 'ip[4:2] = 53309'
reading from file tcpdump.log, link-type EN10MB (Ethernet)
09:01:45.591840 IP 172.16.16.218.80 > 192.168.1.169.2493: P
455882007:455883094(1087) ack 2677933670 win 6432
E..g.=@.?............P ..,5....fP.. -...HTTP/1.0 200 OK
Date: Wed, 18 May 2011 15:01:45 GMT
Server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.5 with Suhosin-Patch
Last-Modified: Wed, 18 May 2011 00:46:10 GMT
ETag: "1238-27b-4a38236f5d880"
Accept-Ranges: bytes
Content-Length: 635
Content-Type: image/jpeg
X-Cache: MISS from www-proxy.example.com
X-Cache-Lookup: MISS from www-proxy.example.com:3128
Via: 1.0 www-proxy.example.com:3128 (squid/2.6.STABLE18)
Connection: keep-alive
......Look! A beautiful pwny!............... ... .......
.
...............................................................................
...........................
.................s.......!.1AQ..a"q..2.....B#.R..3.b.$r..%C4S...cs.5D'...6.
Tdt....&.
....EF..V.U(........eu........fv........7GWgw........8HXhx........)9IYiy
........*:JZjz........................m......!.1A.Q.a".q..2.......#B.Rbr.3
$4C...S%.c...s.5.D..T..
..&6E.'dtU7....()...........eu.....................Wgw........8HXhx........9
IYiy........*:JZjz......................?...*..?..
The packet payload appears to contain HTTP headers, which makes sense given that the source port is 80/TCP (“World Wide Web HTTP”). The server’s HTTP headers indicate that the packet contains a 635-byte JPEG image:
Content-Length: 635
Content-Type: image/jpeg
The HTTP headers also indicate that the web page was provided through a Squid web proxy, “www-proxy.example.com:3128,” as shown below. Based on the “MISS” result listed in the X-Cache and X-Cache-Lookup headers, the requested page was not already in the Squid cache when requested.
X-Cache: MISS from www-proxy.example.com
X-Cache-Lookup: MISS from www-proxy.example.com:3128
Via: 1.0 www-proxy.example.com:3128 (squid/2.6.STABLE18)
As shown in the “Etag” HTTP header, the ETag of the delivered content was “1238-27b-4a38236f5d880.” This may come in handy for web proxy cache analysis later.
7.9.3 Snort Rule Analysis
Why would this packet trigger the “SHELLCODE x86 NOOP” alert? Let’s find the rule and see. In the snort.conf file, we can see where the rules in use can be found:
[...]
var RULE_PATH /etc/snort/rules
[...]
We also have a copy of the “rules” directory taken from /etc/snort/rules. Let’s use “grep” to extract the rule of interest, based on the Snort ID (SID) 10000648:
$ grep -r sid:10000648 rules
rules/local.rules:alert ip $EXTERNAL_NET any -> $HOME_NET any (msg:"SHELLCODE
x86 NOOP"; content:"|90 90 90 90 90 90 90 90 90 90 90 90 90 90|";
classtype:shellcode-detect; sid:10000648; rev:2;)
We find that the matching rule is in the local.rules file, which is typically used for custom rules added by local staff. The SID, 10000648, also indicates that this is a local rule (according to the “SNORT Users Manual,” Snort SIDs greater or equal to 1,000,000 are “Used for local rules”14).
14. Sourcefire, Inc., “Snort Users Manual 2.9.0,” March 25, 2011, http://www.snort.org/assets/166/snort_manual.pdf.
As you can see above, this is a very simple rule that fires on any inbound IP content that contains a string of at least 14 contiguous bytes of 0x90, which is a “NOOP” instruction on the x86 architecture. This is a common feature of a buffer overflow attack attempt.
Based on the rule that was triggered, we would expect the corresponding packet to contain a series of bytes set to 0x90. Let’s view the packet in both hex adecimal and ASCII to see if we can eyeball the match:
$ tcpdump -nnXr tcpdump.log 'ip[4:2] = 53309'
reading from file tcpdump.log, link-type EN10MB (Ethernet)
09:01:45.591840 IP 172.16.16.218.80 > 192.168.1.169.2493: P
455882007:455883094(1087) ack 2677933670 win 6432
0x0000: 4500 0467 d03d 4000 3f06 e817 ac10 10da E..g.=@.?.......
0x0010: c0a8 01a9 0050 09bd 1b2c 3517 9f9e 0666 .....P...,5....f
0x0020: 5018 1920 2de5 0000 4854 5450 2f31 2e30 P...-...HTTP/1.0
0x0030: 2032 3030 204f 4b0d 0a44 6174 653a 2057 .200.OK..Date:.W
0x0040: 6564 2c20 3138 204d 6179 2032 3031 3120 ed,.18.May.2011.
0x0050: 3135 3a30 313a 3435 2047 4d54 0d0a 5365 15:01:45.GMT..Se
0x0060: 7276 6572 3a20 4170 6163 6865 2f32 2e32 rver:.Apache/2.2
0x0070: 2e38 2028 5562 756e 7475 2920 5048 502f .8.(Ubuntu).PHP/
0x0080: 352e 322e 342d 3275 6275 6e74 7535 2e35 5.2.4-2ubuntu5.5
0x0090: 2077 6974 6820 5375 686f 7369 6e2d 5061 .with.Suhosin-Pa
0x00a0: 7463 680d 0a4c 6173 742d 4d6f 6469 6669 tch..Last-Modifi
0x00b0: 6564 3a20 5765 642c 2031 3820 4d61 7920 ed:.Wed,.18.May.
0x00c0: 3230 3131 2030 303a 3436 3a31 3020 474d 2011.00:46:10.GM
0x00d0: 540d 0a45 5461 673a 2022 3132 3338 2d32 T..ETag:."1238-2
0x00e0: 3762 2d34 6133 3832 3336 6635 6438 3830 7b-4a38236f5d880
0x00f0: 220d 0a41 6363 6570 742d 5261 6e67 6573 "..Accept-Ranges
0x0100: 3a20 6279 7465 730d 0a43 6f6e 7465 6e74 :.bytes..Content
0x0110: 2d4c 656e 6774 683a 2036 3335 0d0a 436f -Length:.635..Co
0x0120: 6e74 656e 742d 5479 7065 3a20 696d 6167 ntent-Type:.imag
0x0130: 652f 6a70 6567 0d0a 582d 4361 6368 653a e/jpeg..X-Cache:
0x0140: 204d 4953 5320 6672 6f6d 2077 7777 2d70 .MISS.from.www-p
0x0150: 726f 7879 2e65 7861 6d70 6c65 2e63 6f6d roxy.example.com
0x0160: 0d0a 582d 4361 6368 652d 4c6f 6f6b 7570 ..X-Cache-Lookup
0x0170: 3a20 4d49 5353 2066 726f 6d20 7777 772d :.MISS.from.www-
0x0180: 7072 6f78 792e 6578 616d 706c 652e 636f proxy.example.co
0x0190: 6d3a 3331 3238 0d0a 5669 613a 2031 2e30 m:3128..Via:.1.0
0x01a0: 2077 7777 2d70 726f 7879 2e65 7861 6d70 .www-proxy.examp
0x01b0: 6c65 2e63 6f6d 3a33 3132 3820 2873 7175 le.com:3128.(squ
0x01c0: 6964 2f32 2e36 2e53 5441 424c 4531 3829 id/2.6.STABLE18)
0x01d0: 0d0a 436f 6e6e 6563 7469 6f6e 3a20 6b65 ..Connection:.ke
0x01e0: 6570 2d61 6c69 7665 0d0a 0d0a ffd8 fffe ep-alive........
0x01f0: 0019 4c6f 6f6b 2120 4120 6265 6175 7469 ..Look!.A.beauti
0x0200: 6675 6c20 7077 6e79 21ff db00 8400 0604 ful.pwny!.......
0x0210: 0404 0504 0605 0506 0906 0506 090b 0806 ................
0x0220: 0608 0b0c 0a0a 0b0a 0a0c 100c 0c0c 0c0c ................
0x0230: 0c10 0c0c 0c0c 0c0c 0c0c 0c0c 0c0c 0c0c ................
0x0240: 0c0c 0c0c 0c0c 0c0c 0c0c 0c0c 0c0c 0107 ................
0x0250: 0707 0d0c 0d18 1010 1814 0e0e 0e14 140e ................
0x0260: 0e0e 0e14 110c 0c0c 0c0c 1111 0c0c 0c0c ................
0x0270: 0c0c 110c 0c0c 0c0c 0c0c 0c0c 0c0c 0c0c ................
0x0280: 0c0c 0c0c 0c0c 0c0c 0c0c 0c0c 0c0c 0cff ................
0x0290: c000 1408 0005 0005 0401 1100 0211 0103 ................
0x02a0: 1101 0411 00ff dd00 0400 01ff c401 a200 ................
0x02b0: 0000 0701 0101 0101 0000 0000 0000 0000 ................
0x02c0: 0405 0302 0601 0007 0809 0a0b 0100 0202 ................
0x02d0: 0301 0101 0101 0000 0000 0000 0001 0002 ................
0x02e0: 0304 0506 0708 090a 0b10 0002 0103 0302 ................
0x02f0: 0402 0607 0304 0206 0273 0102 0311 0400 .........s......
0x0300: 0521 1231 4151 0613 6122 7181 1432 91a1 .!.1AQ..a"q..2..
0x0310: 0715 b142 23c1 52d1 e133 1662 f024 7282 ...B#.R..3.b.$r.
0x0320: f125 4334 5392 a2b2 6373 c235 4427 93a3 .%C4S...cs.5D'..
0x0330: b336 1754 6474 c3d2 e208 2683 090a 1819 .6.Tdt....&.....
0x0340: 8494 4546 a4b4 56d3 5528 1af2 e3f3 c4d4 ..EF..V.U(......
0x0350: e4f4 6575 8595 a5b5 c5d5 e5f5 6676 8696 ..eu........fv..
0x0360: a6b6 c6d6 e6f6 3747 5767 7787 97a7 b7c7 ......7GWgw.....
0x0370: d7e7 f738 4858 6878 8898 a8b8 c8d8 e8f8 ...8HXhx........
0x0380: 2939 4959 6979 8999 a9b9 c9d9 e9f9 2a3a )9IYiy........*:
0x0390: 4a5a 6a7a 8a9a aaba cada eafa 1100 0202 JZjz............
0x03a0: 0102 0305 0504 0506 0408 0303 6d01 0002 ............m...
0x03b0: 1103 0421 1231 4105 5113 6122 0671 8191 ...!.1A.Q.a".q..
0x03c0: 32a1 b1f0 14c1 d1e1 2342 1552 6272 f133 2.......#B.Rbr.3
0x03d0: 2434 4382 1692 5325 a263 b2c2 0773 d235 $4C...S%.c...s.5
0x03e0: e244 8317 5493 0809 0a18 1926 3645 1a27 .D..T......&6E.'
0x03f0: 6474 5537 f2a3 b3c3 2829 d3e3 f384 94a4 dtU7....()......
0x0400: b4c4 d4e4 f465 7585 95a5 b5c5 9090 9090 .....eu.........
0x0410: 9090 9090 9090 9090 9090 9090 5767 7787 ............Wgw.
0x0420: 97a7 b7c7 d7e7 f738 4858 6878 8898 a8b8 .......8HXhx....
0x0430: c8d8 e8f8 3949 5969 7989 99a9 b9c9 d9e9 ....9IYiy.......
0x0440: f92a 3a4a 5a6a 7a8a 9aaa baca daea faff .*:JZjz.........
0x0450: da00 0e04 0100 0211 0311 0400 003f 00f2 .............?..
0x0460: a62a faa7 3fff d9 .*..?..
Toward the end of the packet, at byte offset 0x040c (1036), we see 16 consecutive bytes of value 0x90. This would appear to be part of the binary JPEG file, which begins at offset 0x01ec (492). (Note that 492 + 635 bytes of JPEG data = 1127, the datagram length stated in both the Snort alert and the tcpdump output.)
Whether that string of 0x90 values is a coincidence (used to describe pixel colors) or whether it is a malicious NOOP sled may be up to a reverse-engineering malware (REM) specialist to determine. There are certainly known instances of attacks on JPEG parsing/rendering engines that leverage buffer overflow issues.
7.9.4 Carving a Suspicous File from Snort Capture
Let’s carve out the suspicious JPEG so that we can provide it to a REM specialist. Since the JPEG appears to be contained in a single packet, we can easily use Wireshark to export the packet of interest. Figure 1-1 shows the packet of interest in Wireshark. Notice that we have used the display filter “ip.id == 53309” to quickly pick out the packet of interest. Wireshark automatically recognizes the JPEG file in the HTTP packet and allows us to right-click on it and “Export Selected Packet Bytes.” Notice the JPEG “magic number” (0xFFD8) at the beginning of the highlighted data in the Packet Bytes frame.
Figure 7-1 The packet of interest in Wireshark. Notice that we have used the display filter “ip.id == 53309” to quickly pick out the packet of interest. Wireshark automatically recognizes the JPEG file in the HTTP packet and allows us to right-click on it and “Export Selected Packet Bytes.”.
Here are the cryptographic checksums of the file we just exported:
$ md5sum tcpdump.log-53309-exported.jpg
13c303f746a0e8826b749fce56a5c126 tcpdump.log-53309-exported.jpg
$ sha256sum tcpdump.log-53309-exported.jpg
fc5d6f18c3ed01d2aacd64aaf1b51a539ff95c3eb6b8d2767387a67bc5fe8699 tcpdump.log
-53309-exported.jpg
7.9.5 “INFO Web Bug” Alert
We can also look at the subsequent behaviors of the systems in question, based on the evidence sources we already have. For instance, if we think the source of the JPEG (172.16.16.218) might be a bad actor, then it would make good sense to see if there are any other NIDS alerts associated with the IP address:
$ grep -A 3 -B 2 '172.16.16.218' alert
[**] [1:10000648:2] SHELLCODE x86 NOOP [**]
[Classification: Executable code was detected] [Priority: 1]
05/18-08:01:45.591840 172.16.16.218:80 -> 192.168.1.169:2493
TCP TTL:63 TOS:0x0 ID:53309 IpLen:20 DgmLen:1127 DF
***AP*** Seq: 0x1B2C3517 Ack: 0x9F9E0666 Win: 0x1920 TcpLen: 20
That’s the only one. What about the presumed target, 192.168.1.169? Has it been implicated in any other alerts? Let’s grep out all alerts relating to the target IP address 192.168.1.169, and include the preceding two lines of each alert so that we can then just filter down to the message/SID line. Then we can get a count of unique instances, and sort the alert messages from most common to least:
$ grep -B 2 '192.168.1.169' alert | grep '[**]' | sort -nr | uniq -c
108 [**] [1:2925:3] INFO web bug 0x0 gif attempt [**]
18 [**] [116:59:1] (snort_decoder): Tcp Window Scale Option found with
length > 14 [**]
2 [**] [1:100000137:1] COMMUNITY MISC BAD-SSL tcp detect [**]
1 [**] [1:10000648:2] SHELLCODE x86 NOOP [**]
By volume, the most common alert relating to 192.168.1.169 is the “INFO web bug 0x0 gif attempt.” This alert is triggered by an external web server sending an invisible GIF file to an internal client—a common behavior of web servers trying to track a user as they navigate the web site. (A “web bug” is an object, such as a tiny image file, which is included in a web page or email in order to track user behavior.) Let’s see what else we can learn about these 108 events. First, we use “grep” to extract all instances of the “INFO web bug 0x0 gif attempt” alert (SID 2925, revision 3), and then we filter down to just our host of interest, 192.168.1.169. This way we can see when the alerts start and stop:
$ grep -A 5 1:2925:3 alert | grep '192.168.1.169' | wc -l
108
$ grep -A 5 1:2925:3 alert | grep '192.168.1.169' | head -1
05/18-07:45:09.351488 207.46.140.21:80 -> 192.168.1.169:2127
$ grep -A 5 1:2925:3 alert | grep '192.168.1.169' | tail -1
05/18-08:15:08.361442 138.108.28.10:80 -> 192.168.1.169:2649
So we see that they took place over a half hour of web browsing on 05/18 from 07:45:09 to 08:15:08. It might be interesting to get an aggregate count of the various sources of these web bugs:
$ grep -A 5 1:2925:3 alert | grep '192.168.1.169' | awk '{print $2}' |
sort | uniq -c | sort -nr
15 205.188.60.65:80
13 72.14.213.102:80
8 208.71.198.133:80
8 204.203.18.154:80
[...]
$ grep -A 5 1:2925:3 alert | grep '192.168.1.169' | awk '{ print $2 }' |
sort -u | wc -l
42
It appears that the 108 alerts resulted from traffic from 42 distinct web servers. Looking up the associated domains and IP address owners, we find that these web servers were owned by companies including AOL, Google, Monster, and others.
7.9.6 “Tcp Window Scale Option” Alert
Now, let’s turn our attention to the next most common alert relating to 192.168.1.169: “Tcp Window Scale Option found with length > 14.” This is an alert produced by the Snort preprocessor that parses TCP options and watches for anomalous values. In the command below, we extract all instances of the alert by searching for its “GID:SID:Rev,” and then filter only for matches that relate to 192.168.1.169:
$ grep -A 6 116:59:1 alert | grep -A 4 -B 2 '192.168.1.169'
[**] [116:59:1] (snort_decoder): Tcp Window Scale Option found with length >
14 [**]
[Priority: 3]
05/18-08:04:28.974574 192.168.1.169:36953 -> 192.168.1.10:38288
TCP TTL:40 TOS:0x0 ID:58145 IpLen:20 DgmLen:60
**U*P**F Seq: 0x49B898B0 Ack: 0x9EF411B1 Win: 0xFFFF TcpLen: 40 UrgPtr: 0
x0
TCP Options (5) => WS: 15 NOP MSS: 265 TS: 4294967295 0 SackOK
[**] [116:59:1] (snort_decoder): Tcp Window Scale Option found with length >
14 [**]
[Priority: 3]
05/18-08:04:29.092667 192.168.1.169:36953 -> 192.168.1.2:43687
TCP TTL:56 TOS:0x0 ID:55927 IpLen:20 DgmLen:60
**U*P**F Seq: 0x49B898B0 Ack: 0x9EF411B1 Win: 0xFFFF TcpLen: 40 UrgPtr: 0
x0
TCP Options (5) => WS: 15 NOP MSS: 265 TS: 4294967295 0 SackOK
[**] [116:59:1] (snort_decoder): Tcp Window Scale Option found with length >
14 [**]
[Priority: 3]
05/18-08:04:29.160160 192.168.1.169:36953 -> 192.168.1.10:38288
TCP TTL:46 TOS:0x0 ID:51951 IpLen:20 DgmLen:60
**U*P**F Seq: 0x49B898B0 Ack: 0x9EF411B1 Win: 0xFFFF TcpLen: 40 UrgPtr: 0
x0
TCP Options (5) => WS: 15 NOP MSS: 265 TS: 4294967295 0 SackOK
[...]
Looking just at the first few results above, we can see a number of things that are suspicious about the traffic that caused them, in addition to the size of the window scale option:
• While the destination host varies, certain other values do not, including the source port and the setting of the Urg, Push, and Fin flags.
• All of the packets have identical sequence and acknowledgment numbers!
• Although the sources and destinations are all on the same network, the time to live (TTL) values are completely inconsistent.
These are clearly crafted packets. Judging from the nonsensical values and flags, they were likely generated by some sort of reconnaissance tool, probably for the purposes of operating system fingerprinting. Often, fingerprinting tools send strange packets to their targets because different operating systems and applications respond in different ways to unexpected input. By sending strange packets and evaluating the target’s response, fingerprinting tools can compare the target’s response to known values and determine likely software versions.
Let’s look more closely at the timing and sockets of the packets that caused these alerts:
$ grep -A 5 116:59:1 alert | grep '192.168.1.169'
05/18-08:04:28.974574 192.168.1.169:36953 -> 192.168.1.10:38288
05/18-08:04:29.092667 192.168.1.169:36953 -> 192.168.1.2:43687
05/18-08:04:29.160160 192.168.1.169:36953 -> 192.168.1.10:38288
05/18-08:04:29.341094 192.168.1.169:36953 -> 192.168.1.10:38288
05/18-08:04:29.362479 192.168.1.169:36953 -> 192.168.1.30:31935
05/18-08:04:29.529108 192.168.1.169:36953 -> 192.168.1.10:38288
05/18-08:04:29.886795 192.168.1.169:36953 -> 192.168.1.30:31935
05/18-08:04:30.419140 192.168.1.169:36953 -> 192.168.1.30:31935
05/18-08:04:30.948974 192.168.1.169:36953 -> 192.168.1.30:31935
05/18-08:04:33.076313 192.168.1.169:36953 -> 192.168.1.10:40350
05/18-08:04:33.268762 192.168.1.169:36953 -> 192.168.1.10:40350
05/18-08:04:33.525746 192.168.1.169:36953 -> 192.168.1.30:36430
05/18-08:04:33.526513 192.168.1.169:36953 -> 192.168.1.10:40350
05/18-08:04:33.718214 192.168.1.169:36953 -> 192.168.1.10:40350
05/18-08:04:34.057618 192.168.1.169:36953 -> 192.168.1.30:36430
05/18-08:04:34.587735 192.168.1.169:36953 -> 192.168.1.30:36430
05/18-08:04:35.119021 192.168.1.169:36953 -> 192.168.1.30:36430
05/18-08:04:38.805728 192.168.1.169:36891 -> 192.168.1.170:32165
They begin at 08:04:28, target four different hosts on the local segment, and complete within 10 seconds. The four hosts were:
192.168.1.170
192.168.1.30
192.168.1.10
192.168.1.2
It’s likely that 192.168.1.169 sent crafted packets to these four hosts for the purposes of reconnaissance.
7.9.7 Timeline
Based on what we’ve discovered so far, we can begin to interpret our NIDS alerts to construct a time-based narrative. Here’s what transpired on 5/18/11 that seems to be of interest to us (times are in MST):
• 07:45:09 NIDS alerts for 192.168.1.169 begin. Though these initial alerts—for web bug downloads—do not themselves indicate any particularly adverse behavior, they do serve to establish a known commencement of web browsing activity by 192.168.1.169.
• 08:01:45 There is a NIDS alert for possible shellcode being downloaded by 192.168.1.169 from an unknown external web server. This is the NIDS alert that was the impetus of our investigation.
• 08:04:28-08:04:38 Multiple NIDS alerts (18) report crafted packets sent from 192.168.1.169 to multiple internal hosts.
• 08:15:08 NIDS alerts for 192.168.1.169 end. The end of the web bug download alerts does not definitively indicate that 192.168.1.169 has ceased to be active on the network, but it does at least indicate a possible change in the operator’s web browsing activities.
7.9.8 Theory of the Case
Based on the timeline of NIDS events that we’ve identified as being of interest, let’s try to construct a theory, or theories, to explain the observed events. Such theories should be posed in such a way that they can be tested or evaluated to determine the actual nature of the events that transpired. The next step in our analysis is interpretation.
Here is a theory of the case that we’ve constructed based on the evidence obtained so far:
• From 07:45:09 until at least 08:15:08 on 5/18/11, internal host 192.168.1.169 was used to browse external web sites, some of which delivered web bugs, which were detected and logged by the NIDS.
• At 08:01:45, an unknown external web server (172.16.16.218) delivered what it stated was a JPEG image to 192.168.1.169, which contained an unusual binary sequence that is commonly associated with buffer overflow exploits.
• The ETag in the external web server’s HTTP response was: 1238-27b-4a38236f5d880
• The MD5sum of the suspicious JPEG was: 13c303f746a0e8826b749fce56a5c126
• Less than three minutes later, at 08:04:28, internal host 192.168.1.169 spent roughly 10 seconds sending crafted packets to other internal hosts on the 192.168.1.0/24 network. Based on their nonstandard nature, the packets are consistent with those used to conduct reconnaissance via scanning and operating system fingerprinting.
Our suspicions at this stage of the investigation are probably quite clear and straightforward:
• “Drive-by” exploit It is certainly feasible that the “SHELLCODE x86 NOOP” alert was a true positive, and that the content of the connection contained a client-side exploit targeting the software that parses and renders JPEG images on internal host 192.168.1.169. If the internal host was compromised in this way, it would go a long way toward explaining the system’s subsequent behavior.
• Subsequent reconnaissance The packets that 192.168.1.169 began sending at 08:04:28 to other local systems are perhaps the single most positive indicator that something is amiss. The parameters of the packets recorded are simply inconsistent with any normal TCP transaction, violating both convention and RFC specifications in numerous ways.
If we assume that a process on 192.168.1.169 was conducting reconnaissance—which is quite consistent with the traffic patterns observed—this raises the obvious question of “Why?” It could well be that one of the system’s legitimate users conducted the scan. It could also be the case that the scanning process was launched by someone or something else, lending credence to the theory that the system was compromised.
7.9.9 Next Steps
Let’s explore some ways we can test each of the possible theories we’ve posited above:
• Malware Analysis Since we carved a suspicious JPEG image out of the Snort packet capture, we can provide this to REM investigators for a detailed analysis. This can help us create NIDS and/or antivirus signatures for identifying the malware elsewhere on our network, determine the purpose and function of the malware, and scope the extent of the potential breach.
• Hard Drive Analysis We would almost certainly want to track down and obtain the suspect internal host. This might entail reviewing DHCP leases, asset databases, CAM tables, and the like. If the system can be reliably obtained, it could be analyzed for signs of compromise. We should be able to determine whether any of its legitimate users had performed the scan, either intentionally or inadvertently. If the system’s browser cache contained the 635-byte JPEG (identifiable by MD5 sum), and their timestamps matched, that would provide excellent corroboration that the correct system had been apprehended.
• Firewall/Central Log Server Analysis Understanding the scope of the inside agent’s activities will be critical. Logs from both host-based and network-based firewalls should be consulted, as they may contain traces of activities in more volume or detail.
• Web Proxy Cache Analysis Recall that the HTTP server headers within the packet revealed the use of a proxy. If the proxy’s history and cache can be obtained, it might be possible to put the malicious JPEG into context, to see where it lives and how it came to be served to the internal host.
To Be Continued ...
Please see Section 10.8 for the conclusion of “InterOptic Saves the Planet.”