Detecting a Compromise

The easiest method of detecting an attack and a compromise of your system is by setting up alert capabilities of your firewall and IDS. Because firewalls and IDSs have similar functions, you can probably get by with one rather than having both in place. The several firewalls we reviewed have IDS-like capabilities, from the pop-up alert windows of ZoneAlarm Pro and Norton Personal Firewall to the flashing icon of BlackICE Defender.

Many commercial IDSs are available, but home users won't have much selection for Windows-based operating systems. The Unix environment offers many IDS products because Unix programs tend to be free and open source. Free solutions are more readily available to Unix users. One of the most popular is Snort (www.snort.org). Snort runs on both Unix and Windows systems, but it does require some in-depth knowledge. In addition, it requires a great deal of time in setting it up and understanding the reports.

Each IDS/firewall has different methods of displaying attack information and using filter capabilities to block and report attacks. Attacks and intrusions can be detected by the following methods:

• Anomaly detection— Statistical anomalies can be used to detect traffic that is out of the ordinary or does not follow a known pattern. If the traffic of the system is monitored and recorded, data such as CPU utilization, disk activity, user logins, file activity, transactions, and variation from the baseline can be seen as an anomaly and perhaps an attack. Although this type of detection cannot specifically tell you if a hacker is targeting your Web server with a file traversal attack, it can tell you that someone is launching an attack that goes beyond simple Web browsing of your Web site. All the traffic that is sent to the system is monitored. When invalid data or data that is not normally seen by the IDS is found, alerts are activated that notify you of an attack.

• Signature recognition— Sophisticated IDSs usually use a signature-based detection model. Because most attacks are known rather quickly and most attacks fall into known categories of attacks, such as Web attacks, DOS attacks, or buffer overflow attacks, looking for the known pattern of attack can be used to detect and stop the attack. Some of these systems have a database of signatures that they can use to detect attacks. When the attack from the database of signatures is seen, alerts notify you of an attack. This type of pattern matching is used for known attacks.

What to Detect

Intrusion detection looks for attacks against the system or network, but it also checks the system on which it is running for compromised data and programs. When an attacker compromises a system, he can insert Trojan horses, backdoors, and rootkits to cause further mischief.

Trojan horse utilities hide hacker activities from the system administrator. Trojan horse files can have the same size and time/date stamp of the file they are replacing to make detection difficult. The use of cryptographic checksums can help determine if a Trojan horse has replaced the real file on your system. The MD5 checksum can be used to compare your binary files to known checksums of good binary files. You can find it at ftp://ftp.cerias.purdue.edu/pub/tools/unix/crypto/md5/.

Backdoors allow the hacker to enter the system at a later time. Backdoors listen on a TCP or UDP port and allow an attacker remote access to the system so that he does not have to continuously hack up your system to gain entry. Backdoors can open TCP or UDP ports that can look like legitimate ports or can open on unused high-numbered ports. Popular backdoor Windows programs are BackOrifice (www.bo2k.com) and NetBus (http://packetstormsecurity.org/). After an attacker compromises a system and installs a program such as BackOrifice, he can use it to remotely control the compromised system. Figure 16.5 shows how the attacker can configure BackOrifice to control the victim's machine.

In Figure 16.6, we see some of the configuration options of NetBus. BackOrifice and NetBus allow an attacker who has compromised your system to take control of your desktop. Most virus scanners will detect these programs.

Rootkits are a combination of Trojan horses and backdoors. Rootkits can include log-wiping utilities that hide the steps taken by attackers from files such as utmp, wtmp, messages, and syslog in Unix and the event logs in Windows systems. Additionally, packet sniffers that can log all traffic on the compromised system's network and other utilities that can allow the attacker to perform other functions on your system can be part of a rootkit. With rootkits, you can't trust any binary file you have on your system. If you want to check for a backdoor that might be running on a high-numbered open port, you can use one of several third-party programs as well as the built-in netstat command. In Figure 16.7, we see the listening ports on our system using netstat -an in a DOS window. Many ports are open that are standard in our system, such as 135, 445, and some others; however, this example contains a port numbered 6666, which seems suspicious. This might be worth investigating to see what program is running.

IDSs and firewalls are good at detecting denial-of-service attacks, but stopping them is hard to do. Because DoS attacks can target everything from applications running to consuming network bandwidth to deny service, it's hard to stop all different forms of DoS attacks. Patches for operating systems and applications can stop many of these attacks, but they change so frequently that detecting them and attempting to block the attack is all that a consumer or a company can do.

Many ISPs implement filtering rules on their routers to help prevent DoS attacks.Consumers have no access to ISP routers, so this is out of your control. You can, however, disable any unused or unneeded network services, check your typical system utilization such as CPU and hard disk space for unidentified changes, back up your system in case a DoS attack destroys data, and adhere to good security measures. Your system can be compromised and used in distributed denial-of-service attacks. These attacks use multiple computers to attack a victim, thereby increasing the flood of data sent to the victim to easily overwhelm the computer and cause damage.

Traffic Monitoring

IDSs are somewhat synonymous with traffic monitoring. To detect an attack, an IDS looks at packets of data sent to the system or network. The computer that is running the IDS is put in promiscuous mode, which allows it to watch all traffic into and out of the system. Not all machines on the network need to be running an IDS. Ethereal, which we have already discussed, can be used to monitor all traffic and data, but it is not specifically designed as an IDS. IDS can capture every bit of data such as IDs and passwords that are used across the network. The firewall products we used can monitor traffic but cannot examine each packet of data and tell you exactly what is in them, such as what the ID and password that are being sent across the network contains.

Snort (www.snort.org), which is an IDS available for both Windows and Linux, provides traffic monitoring capabilities. It is a freeware program that is rather difficult to set up and understand. It does provide alerts via the screen or e-mails (you can choose the configuration). Unlike the firewall products that have IDS capabilities such as alerts, logging, and e-mail, Snort is not used to block traffic. It just detects attacks and reports on them.

Using WinRoute Pro, we can see how the firewall, along with blocking and filtering capability, can record to a log file and provide valuable attacker information. (Earlier, in the section “Problems with IDS,” you saw that all attack data that is blocked or allowed by your firewall filter rules can be sent to a file.) Figure 16.8 shows how the attacks that are blocked by the filter rules can be sent to the screen. WinRoute doesn't have an alert capability, so you must check the files and screen for attack information. This type of firewall cannot provide sophisticated IDS capability to block traffic based on anomaly detection or signature detection.

Another firewall product we used that has good alerting capabilities when the filter rules are violated is BlackICE Defender. Again, this product cannot check for any attack that does not specifically block by the filter rules. If you allow the Web server port to be open on your system, the firewall allows any Web attacks that are launched by an attacker that go over port 80 (http). Full-blown IDSs such as Snort do alert you to the attack. Many IDSs will not block traffic; in many cases, they just monitor and alert you to the traffic. A program such as Entercept blocks unknown attacks, but the average consumer cannot afford this corporate IDS product because it costs in the thousands of dollars.

Signs of a Compromise

Beyond reviewing your log files and looking for alerts from your IDS, you can check for other signs to see if your system has been compromised. Table 16.1 shows what you can look for.

Table 16.1. Signs of a Compromise

What to Check For	Description
Examine inherent log files.	Built-in logging capabilities exist in Windows- and Unix-based operating systems. Using Event Viewer in Windows NT/2000, you can see connections, errors, and security messages. Log files such as /var/log/ secure, /var/log/maillog, /var/log/messages, and var/log/httpd/access_log in Unix systems can provide valuable connection and security information.
Examine application logs.	Applications such as Web servers, FTP servers, and mail servers, among other,s have the capability to log access and errors in many instances. These can provide valuable intruder attempt information.
Evaluate your accounts.	You should know all the accounts that are valid on your system. If you see a new account that you did not create—especially one with privileged access—you should investigate it. You can use the User Manager tool in Windows NT/2000 to check the password file in Unix systems. You should also check accounts periodically that you do know about to make sure that they have not been given extra privileges. This could be indicative of an intruder gaining access to a valid account and escalating its privileges.
Check running processes and applications.	Intruders frequently start new services or install and run applications such as packet capture software. You should know what services you have running and if a new service or application is running that you have not started or don't know about. You can check services through the Services program in Windows NT/2000 (select Start, Control Panel, Services), use netstat (a built-in command in Windows that you can access by selecting Start, Run and typing cmd), or use Task Manager (by pressing Ctrl+Alt+Delete) to see what applications and processes are running. In Unix, you can check all the startup scripts and use the netstat command or top to check running processes. When systems start, programs can be run automatically through startup files such as the Startup folder in Windows or the rc* scripts in Unix. These should be checked to make sure nothing is started that you don't know about. If you see open ports that are listening that seem odd, a hacker program could be waiting for a connection or could be a backdoor.
Check your binaries.	Using MD5 checksums, as we have already mentioned, is a method of checking to ensure that the executable programs have not been replaced with Trojan horses. Although using MD5 checksums is difficult, it is a good measure to use periodically. Several programs are available that can do this automatically, including Tripwire (http://www.tripwire.com) for both Windows and Unix.
Watch your antivirus.	Your antivirus software can be used to help stop Trojan horses, worms, and viruses that are aimed at your system for malicious hacker purposes.
Check your network adapter.	If your network adapter has been placed in promiscuous mode to capture traffic and you weren't the one who did that, it's a sure sign that you have been compromised and someone is capturing traffic on your network. Programs such as Antisniff (http://www.securitysoftwaretech.com/antisniff/) for Windows NT/2000 and ifconfig in Unix will also tell you if the adapter is capturing traffic.
Check for shared data.	In both Windows and Unix, files and folders of data can be shared out to everyone. Check to make sure your system isn't sharing data that you do not know about.
Check for scheduled tasks.	An attacker frequently sets up a process or program to run at some future date to give him access or perform some function when the victim will least expect it. You can check what jobs might be scheduled for later execution with the at command in Windows NT/2000 and the cron command in Unix.
Check for network connectivity disruptions.	If you notice that your connection keeps going down or is sluggish for no reason, you might be the target of a denial of service attack. You can check your log files for failed connections or check your IDS for repeated connections.
Watch for obvious signs.	If your mouse is moving by itself or applications are opening and closing by themselves, someone might have taken over your system with a remote control program such as BackOrifice or VNC.