Chapter 1. Getting Started
It was mid-January 2003. Things were going well in my role as a network engineer supporting data center networks at Cisco. My team celebrated on January 21 when our site vice president powered off the last Avaya PBX; the Research Triangle Park (RTP) campus telephony was now 100% VoIP. We had just completed several WAN circuit and hardware upgrades and were beginning to see the highest availability numbers ever for our remote sites. Then, on January 25 (a Saturday at the RTP campus), the SQL Slammer worm wreaked havoc on networks around the world. Slammer, also known as Sapphire, targeted vulnerable MS-SQL servers using self-propagating malicious code. Security professionals surely remember the event well. The worm’s propagation technique created a potent denial-of-service (DoS) effect, bringing down many networks as it spread.
The only attribute distinguishing the Slammer worm from normal SQL traffic was a large number of 376-byte UDP packets destined for port 1434.[1]
ISPs used ingress/egress filtering to block traffic, but by then it was too late to prevent system compromise; rather, it was a mitigation measure to protect the Internet backbone:
The Sapphire Worm was the fastest computer worm in history. As it began spreading throughout the Internet, it doubled in size every 8.5 seconds. It infected more than 90 percent of vulnerable hosts within 10 minutes.[2]
The rate of replication and multitude of compromised systems on company networks began to saturate network links with propagation attempts. Network administrators saw this issue on some of the WAN links in the United States when their pagers began to light up like Christmas trees with utilization alerts, followed by link down Simple Network Management Protocol (SNMP) traps. Initially, the problem was thought to be related to a DS3 network card we had just replaced in one of our Southeast region WAN routers; however, as the issue appeared in other regional office WAN links, it became clear that this was not an isolated incident.
We had experienced the network problems caused by virus outbreaks such as Code Red (which attacked vulnerable Microsoft IIS web servers), but none approached the severity of network impact that Slammer did. A few Slammer hosts were able to generate enough traffic to take down WAN links, causing intermittent connectivity problems in our remote sites globally. Ultimately, a majority of the compromised systems were traced to unpatched lab servers. Identifying and mitigating these hosts was no easy task:
Too few network intrusion detection systems (NIDSs) were deployed, and no one was responsible to view or follow up on alerts for infected systems.
Network telemetry (such as NetFlow) and anomaly detection were insufficient to identify infected systems.
There was no way to prioritize the response; the only data we had were IP addresses and DNS names of affected machines. We didn’t have contextual information such as “data center host” versus “user LAN host” versus “lab host.”
Over the next 48 hours, global networking teams identified infected systems using a manual process that involved deploying the recommended access control lists (ACLs) on remote WAN routers[3] to block packets. Matches on the deny access control entries (ACEs) for UDP 1434 indicated an infected host at the site. We could not identify the source IP address that was creating the deny entries, as adding the “log” clause to the end of the deny ACE spiked the router’s CPU and drastically degraded network performance. The next step required network engineers to analyze switch port utilization in real time, searching for the infected host to disable its port. This manual process required substantial man-hours to address.
If we had implemented a few of the recommendations detailed in this book, our networking team could have contained the threat much more rapidly. A tuned NIDS deployment would have enabled us to locate the infected IP addresses immediately, prioritizing response based on their named network association (data center servers, lab hosts, or desktop systems, as you’ll see in Chapter 6). Even prior to the availability of the NIDS signature, we could have used NetFlow to identify infected hosts based on recognized traffic patterns, as we’ll discuss in Chapter 3. A prioritized, planned response would have occurred based on this information, with appropriate mitigation measures applied to the impacted systems. The IP information from NetFlow alone could have allowed for quick manual inspection of the router ARP tables and associated MAC-to-IP address mapping. Armed with that mapping, the network engineers could have quickly disabled ports on the access switches, shutting down worm propagation.
This book details infrastructure and frameworks that would have further helped when Nachi broke out several months later. Since we couldn’t see the future, however, Nachi created the same effect and was addressed with the same manual process as Slammer.
A Rapidly Changing Threat Landscape
We’ve heard it before: “gone are the days of script kiddies and teenagers out to wreak havoc just to show off.” The late 1990s and early 2000s produced a staggering number of DoS attacks. Malware, the engine for the DoS attack, has progressed from simple programs that attack a single vulnerability to complex software that attacks multiple OS and application vulnerabilities.
Let’s look at the description of the Nachi worm’s method of infection (circa 2003):
This worm spreads by exploiting a vulnerability in Microsoft Windows. (MS03-026)
Web servers (IIS 5) that are vulnerable to an MS03-007 attack (port 80), via WebDav, are also vulnerable to the virus propagating through this exploit.[4]
Here’s information on a very popular virus from 2006 called SDBot:
The worm propagates via accessible or poorly-secured network shares, and some variants are intended to take advantage of high profile exploits:
WEBDAV vulnerability (MS03-007)
LSASS vulnerability (MS04-011)
ASN.1 vulnerability (MS04-007)
Workstation Service vulnerability (MS03-049)
PNP vulnerability (MS05-039)
Imail IMAPD LOGIN username vulnerability
Cisco IOS HTTP Authorization vulnerability
Server service vulnerability (MS06-040)
When it attempts to spread through default administrative shares, for example:
PRINT$
E$
D$
C$
ADMIN$
IPC$
Some variants also carry a list of poor username/password combinations to gain access to these shares.
Weak Passwords and Configurations
Several variants are known to probe MS SQL servers for weak administrator passwords and configurations. When successful, the virus could execute remote system commands via the SQL server access.[5]
This more complex form of malware has components to make it persistent between reboots and to cloak itself from detection by antivirus programs. It even includes obfuscation techniques to prevent offline analysis! Many malware programs include a component to steal information from the infected system and relay it back to its creator, leveraging a remote control component (commonly called a botnet), which provides a vast array of capabilities to command the compromised system. Group all of these traits together—decentralized command and control structures (such as web-based or peer-to-peer [P2P] structures), and encryption and polymorphism (so that the malware can modify itself upon propagation to another system, evading detection by antivirus software)—and you can easily see why antivirus technology rarely lives up to its promise.
Failure of Antivirus Software
Hopefully, you no longer rely solely on antivirus software to detect and protect your end-user systems. Rather, a defense-in-depth strategy includes antivirus software, adding OS and application patch management, host-based intrusion detection, and appropriate access controls (we said “hopefully” ☺). If you are still relying exclusively on antivirus software for protection, you will be very disappointed. For example, in summer 2008, many of our employees received a well-crafted phishing campaign that contained a realistic-looking email regarding a missed shipment delivery from UPS:
-----Original Message----- From: United Parcel Service [mailto:[email protected]] Sent: Tuesday, August 12, 2008 10:55 AM To: [email protected] Subject: Tracking N_ 6741030653 Unfortunately we were not able to deliver postal package you sent on July the 21st in time because the recipient's address is not correct. Please print out the invoice copy attached and collect the package at our office Your UPS
Attached to this email was a trojan that more than 90% of the 37 antivirus software programs were unable to detect. Table 1-1 shows the test results yielded by analysis of the trojan binary.
Antivirus | Result | Antivirus | Result |
AhnLab-V3 | - | Kaspersky | - |
AntiVir | - | McAfee | - |
Authentium | W32/Downldr2.DIFZ | Microsoft | - |
Avast | - | NOD32v2 | - |
AVG | - | Norman | - |
BitDefender | - | Panda | - |
CAT-QuickHeal | - | PCTools | - |
ClamAV | - | Prevx1 | - |
DrWeb | - | Rising | - |
eSafe | - | Sophos | - |
eTrust-Vet | - | Sunbelt | Trojan-Spy.Win32.Zbot.gen (v) |
Ewido | - | Symantec | - |
F-Prot | - | TheHacker | - |
F-Secure | - | TrendMicro | - |
Fortinet | - | VBA32 | - |
GData | - | ViRobot | - |
Ikarus | Win32.Outbreak.UPSRechnung | VirusBuster | - |
K7AntiVirus | - | Webwasher-Gateway | - |
As you can see from the test results, these antivirus products, which detect malware via “known bad” signatures, failed to identify the trojan. Such technology fails primarily because an insignificant change to the virus will make it undetectable by existing signatures. Vendors are improving their techniques—by including heuristic/behavioral-based detection, for example—but they still fall far short of providing “complete” system security. An excellent source for more information regarding viruses, their capabilities, and why they are able to hide from detection is John Aycock’s book, Computer Viruses and Malware (Springer).
The prevalence and advanced capabilities of modern malware should be reason enough to closely monitor for its existence in your network. If it isn’t, perhaps its use by Mafia-like organizations of criminals for profit via identity theft, extortion, and espionage is more convincing.
Why Monitor?
Organized crime and insider threats are changing the security landscape, and provide ample rationale for proactive security monitoring.
The Miscreant Economy and Organized Crime
An enormous amount of money is being stolen every day—enough, in fact, to drive coordination and cooperation within groups of criminals. This illicit partnership has accelerated the development of sophisticated malware (used for this purpose, it’s often called crimeware). Most information security organizations, both government and private, are ill-equipped to handle such threats with their existing technology and processes.
A 2008 study by F-Secure Corporation predicted that the use of malware for criminal activity would increase in countries such as Brazil, China, the former Soviet Union, India, Africa, and Central America. This is due to an abundance of highly skilled people who lack opportunities to use those skills in a legal manner.[6]
Although most of this activity is not directed at corporations, we have seen incidents that exploit knowledge of names or team/management relationships, allowing the creation of very believable phishing emails. This technique is often referred to as spearphishing.
In contrast, the actions of malicious insiders with access to critical information and intellectual property make up what is referred to as an insider threat.
Insider Threats
Studies from the U.S. Secret Service and the U.S. Computer Emergency Response Team Coordination Center (CERT/CC) validate the existence of insider threats. Although many still debate the exact percentage, it appears that between 40% and 70% of all incidents are related to insider threats. This sizable amount, coupled with the insider’s access and knowledge, must be met with a proportionate amount of monitoring efforts toward insider activity. A few high-profile incidents should help to drive the insider threat message home:[7]
- Horizon Blue Cross Blue Shield
In January 2008, more than 300,000 names and Social Security numbers were exposed when a laptop was stolen. An employee who regularly works with member data was taking the laptop home.
- Hannaford Bros. Co.
In May 2008, 4.2 million credit and debit card numbers were compromised. Close to 1,800 cases of fraud were reported related to this security breach. It was found that the card numbers were harvested during the transaction process.
- Compass Bank
In March 2008, a database containing names, account numbers, and customer passwords was breached. A former employee stole a hard drive containing 1 million customer records and used that information to commit fraud. He used a credit card encoder and blank cards to create several new cards and withdraw money from multiple customer accounts.
- Countrywide Financial Corp.
In August 2008, the FBI arrested a former Countrywide Financial Corp. employee for stealing personal information, including Social Security numbers. The insider was a senior financial analyst at a subprime lending division. The alleged perpetrator of the theft sold account information weekly in groups of 20,000 for $500.
Not all of the aforementioned incidents were malicious in nature, but all of them began with a violation of security policy. Chapters 2 and 6 provide a framework for you to detect malware and insider threats. Chapters 4 and 5 will help you prioritize your limited monitoring resources and choose the event data that provides the “biggest bang for the buck.”
Challenges to Monitoring
Product limitations, the realities of operational monitoring, event volumes, and the necessity of privacy protection are challenges faced by security professionals when constructing a monitoring approach.
Vendor Promises
“Just plug it in; it will sort out everything for you!” This advice on setting up vendor XYZ’s Security Information Manager (SIM) system to “automagically” correlate security events may work in small, strict, well-maintained environments. However, utopian environments such as these are rare in our experience and in talking with our customers. Security monitoring is not like a Ron Popeil Showtime Rotisserie; you can’t “set it and forget it.”
Security technology cannot automatically provide the contextual information necessary for you to prioritize and focus your security monitoring. Every environment is unique, but the methods we discuss in Chapter 3 will enable you to build this critical contextual information into all of your security tools. “But wait, there’s more!”
Operational Realities
“Turn on auditing for all your database tables.” Database operations in a busy enterprise environment prioritize performance and stability, which gave us pause when considering such advice. What are the potential performance impacts? What risks does this introduce to business operations, change controls, stability, and uptime? We began to discuss these concepts through email with an IT database administrator. He stopped replying to our emails after we relayed the “turn on auditing for all your tables” advice! Indeed, such intensive database auditing in any but the most rarely used environment would reduce system performance to unacceptable levels. Our recommendations in this book are tested and proven by our own operational experience in IT, where we have both supported enterprise infrastructures. We won’t suggest methods that will negatively impact availability, thus harming your relationship with the support staff.
Volume
In the context of monitoring, logging data quickly degrades from essential lifeblood to useless swamp water when the volume is too high. An improperly tuned NIDS or syslog daemon can create so many messages that it literally crashes your collection systems. Even if your collector or SIM can handle the flood of event data, a huge volume of unsorted events will overwhelm your monitoring staff and drive them to ignore the data source. The guidelines in Chapters 5 and 6 will give you the right direction for making event volume manageable even in the most complex environments.
Privacy Concerns
You must take care to comply with local privacy laws and regulations, as they vary widely by country. The best advice we can give is to ensure that your human resources department and legal counsel are aware of your monitoring activities, formally documenting their approval for future reference. This is typically done in a monitoring statement, which should be included in your company’s acceptable use policy.
Outsourcing Your Security Monitoring
In many companies, security is little more than a checkbox on a compliance document. “Employees, check! IT Services, check! Security, check!” And so on....
If you’ve already outsourced the whole of your security monitoring to a managed security services vendor so that you can check your compliance boxes, stop reading and sell this copy on eBay. You probably haven’t even cracked the binding, so you can list it “like new.” In our experience and with talking to customers, it is extremely rare to find an outsourced security services vendor who really understands the network and security context of its clients, and as such restricts its effectiveness to the simplest of security problems. Imagine the following proposal: you want to know when someone copies customer data from your database systems to his desktop. How would an outsourced security provider do this? Rather, how much would such a provider charge to do this? The service supplied by most providers is limited to regular reports of selected NIDS alerts—the same NIDS alerts selected for every client—and affected IP addresses—not all that useful, in our opinion.
Monitoring to Minimize Risk
B2B, partner, outsource, extranet; words that make security professionals cringe with disdain. Sometimes directors must accept high risk, such as connecting a partner network before proper risk assessment can be completed, due to urgent business drivers. Often, however, such decisions are made by those without authority to assume such a high level of risk. Such decisions affect an entire corporation, and are often made with flawed or incomplete information. In response, those charged with information security are tempted to get frustrated and surrender to chance. Such capitulation is not necessary. If you follow the approach laid out in this book, you can tailor a monitoring strategy based on the “special” business situation, minimizing or even mitigating the additional risk. Require targeted security monitoring, funded by the risk-taking sponsors, by saying, “If you want to venture into this risky project, you will need to fund additional monitoring resources for hardware and headcount.”
Policy-Based Monitoring
We want to differentiate our framework for policy-based monitoring (sometimes we call it targeted monitoring) from malware monitoring, intrusion detection, extrusion detection, and popular monitoring frameworks. Policy-based monitoring prioritizes monitoring by enumerating and selecting critical systems, detecting policy deviations via their appropriate event logs. It requires analysis of generated events against defined security policies within the context of the environment. The methods we describe will help you to shift the focus of your monitoring resources to the most business-critical systems, bounding your alerts within the defined security policies for those systems.
Why Should This Work for You?
We strongly believe that the frameworks and methods presented here are effective and sound, based on our experience within one of the most complex and fluid enterprise networks in the world. We both have supported critical systems whose uptime directly impacted business revenue and employee productivity (and ultimately, our careers). This guidance is the result of iterative improvements, and should apply across all technologies in your existing security portfolio. The bottom line: if you implement just some of the recommendations made in this book, you should improve your monitoring and incident response capabilities greatly. If you implement all of the recommendations, you will create a world-class security monitoring capability.
Open Source Versus Commercial Products
Both of us are employees of Cisco Systems, and we use their security products. Because we are giving you advice based on our experience, you will find many references to Cisco products. We use open source tools when they meet a specific need, and reference them enthusiastically when they work well. Open source products are featured in Richard Bejtlich’s book, The Tao of Network Security Monitoring (Addison-Wesley Professional), which covers the use of security monitoring tools such as Snort, Bro, Argus, Sguil, and dozens of others. It is a great reference for those who have not already built, or are looking to enhance, their monitoring infrastructure. This book intends to help readers get the most out of their security monitoring tools, whichever products they choose.
Introducing Blanco Wireless
To illustrate our recommendations, we show their implementation within a fictitious company named Blanco Wireless. Blanco is a mobile telephony provider based in the United States. As a means of account management, Blanco collects and stores personal information, including names, addresses, phone numbers, Social Security numbers, credit ratings, and other important customer data. At the end of each chapter, we discuss how Blanco Wireless implements the frameworks and methods suggested in the chapter. Examples include diagrams and explanations of how Blanco will use the chapter’s guidance to develop security monitoring.