In this chapter, we will focus on the parts of the forensic investigation that are specific to the operating systems. We chose the most widely used operating systems on the desktop and server systems—Microsoft Windows and Linux.
For both operating systems, we selected examples of interesting evidence and how to automate its analysis using Python. Consequently, in this chapter, you will learn the following:
- Analyzing the foundations of the Windows event log, selecting interesting parts, and automatically parsing them
- Organizing the Windows Registry and efficiently searching for Indicators of Compromise (IOC)
- Searching Linux local account information for IOC
- Understanding, using, and parsing Linux file metadata with POSIX ACL and file based capabilities as the most prominent extensions to the standard metadata
Windows includes many monitoring and logging capabilities and traces data and events for a large amount and variety of activities occurring in the operating system. The vast number of events, which can be logged, does neither make it easy for an administrator to identify the specific important events nor helps a forensic investigator to find Indicators of Compromise. Therefore, we will start this section with a small introduction to the Windows Event Log and the changes in its format over time, followed by a description of the important event types that should help an investigator to quickly find suspicious actions in the large amount of other events. In the last section of this chapter, we will demonstrate how to parse the Event Log and automatically find the potential IOCs (e.g., user logons, service creation, and so on).
According to Microsoft, Windows Event Log files are special files that record significant events, such as when a user logs on to the computer or when a program encounters an error, (refer to http://windows.microsoft.com/en-us/windows/what-information-event-logs-event-viewer#1TC=windows-7). Whenever these types of events occur, Windows records the event in an event log that can be read using Event Viewer or similar tools.
With the release of Windows 7 and Windows Server 2008, Microsoft has performed a major change in their Event Log technique. They changed from the classical Windows Event Log (EVT) to the newer Windows XML Event Log (EVTX). In the following paragraphs, we will explain some of the main differences between these two log file types.
Due to the fact that Microsoft no longer supports Windows XP and Server 2003 is in the extended support stage at present (meaning that it will go out of support very soon), there are XP and 2003 systems still out there. Thus, some investigators are still going to need to know the difference between the older EVT and the new EVTX and the possible problems arising during analysis of these files.
Besides the binary differences in the records and the Event Log files themselves, the amount of these log files differs too. On a Windows XP/2003 system, there were three main Event Log files: System, Application, and Security. They are stored in the C:Windowssystem32config directory. The server versions of the OS may maintain additional Event Logs (DNS Server, Directory Service, File Replication Service, and so on) depending upon the functionality of the server. On a current Windows 7 system, you can find more than 143 files full of event logs. This gets even more if you compare it to the newer server versions of Microsoft Windows.
The EVT log records only contain a very small amount of human-readable content and are made human readable through tools such as the event viewer at analysis time. These tools combine the predefined log templates that are commonly stored in the system's DLL or EXE files with the data stored in the EVT file itself. When one of the various log viewing tools displays log records, it has to determine which DLL files will store the message templates. This meta-information is stored in the Windows Registry and is specific to each type of the previously mentioned three main Event Log files (System, Application, and Security).
All of the earlier mentioned details follow the fact that the EVT files aren't really useful without their corresponding metafiles, which store the core meaning of the log. This creates two major analysis problems:
- First, an attacker could modify DLL files or the Windows Registry in order to change the meaning of event logs without having to touch the EVT file.
- Second, when the software is uninstalled on a system, it could result in the EVT records losing their context.
As an investigator, one must carefully keep these issues in mind when analyzing EVT logs and also when writing those logs to remote systems for later analysis. An even more detailed analysis of the EVT records can be found in the ForensicsWiki, http://forensicswiki.org/wiki/Windows_Event_Log_(EVT).
In comparison to EVT, the EVTX files are stored as a binary XML file format. On the newer Windows systems, the event logs can be viewed and analyzed with either the Event Viewer or a vast number of other programs and tools (in the following sections, we will describe some Python scripts that can be used too). When using the Event Viewer, one has to bear in mind that this program can represent the EVTX files in two different formats: general and detailed. The general (sometimes called formatted) view can hide significant event data that is stored in the event record and can only be seen in the detailed view. Thus, if you are planning to use the Event Viewer for analyzing the EVTX files, always use the detailed option to display the files.
If you are interested in a more detailed analysis of the EVTX file format, you should take a look at the ForensicsWiki, http://forensicswiki.org/wiki/Windows_XML_Event_Log_(EVTX). Another great explanation of the deeper EVTX file format details has been presented by Andreas Schuster at DFRWS 2007, refer to http://www.dfrws.org/2007/proceedings/p65-schuster_pres.pdf. This presentation can be very helpful if you want to understand the details of the binary XML format or write your own parsers of EVTX files.
If you need to open the EVT files on a Windows 7 or newer system, it's best to convert the older EVT file to the EVTX syntax before opening it. This can be done in several ways as described in a technet.com blog post, http://blogs.technet.com/b/askperf/archive/2007/10/12/windows-vista-and-exported-event-log-files.aspx.
A full list of Windows events on the newer system can be found in a knowledge base article of Microsoft at, https://support.microsoft.com/en-us/kb/947226. As the number of these events is getting bigger with every new version of the system and every newly installed application, you can easily find more than several hundreds of different event types on a single Windows system. Due to this fact, we tried to sort out some interesting event types that can be helpful when analyzing a system or reconstructing user events (a more detailed explanation of which Event Logs can be helpful under what conditions can also be found in TSA-13-1004-SG, https://www.nsa.gov/ia/_files/app/spotting_the_adversary_with_windows_event_log_monitoring.pdf):
- EMET (1, 2): If the organization is actively using the Microsoft Enhanced Mitigation Experience Toolkit (EMET), then these logs can be very helpful during investigation.
- Windows-Update-Failure (20, 24, 25, 31, 34, 35): The failure to update issues should be addressed to avoid prolonging the existence of an application issue or vulnerability in the operating system or an application. Sometimes, this also helps in identifying infections of a system.
- Microsoft-Windows-Eventlog (104, 1102): It is unlikely that event log data would be cleared during normal operations and it is more likely that a malicious attacker may try to cover their tracks by clearing an event log. When an event log gets cleared, it is suspicious.
- Microsoft-Windows-TaskScheduler (106): It displays newly registered Scheduled Tasks. This can be very helpful if you are searching for signs of malware infections.
- McAfee-Log-Event (257): McAfee malware detection—McAfee AntiVirus may detect malware behaviors without actually detecting the EXE file itself. This can be very valuable in determining how the malware got into a system. In general, the event logs of the installed AV solution are very valuable logs when starting an analysis of a potentially compromised system. Therefore, you should remind yourself where to find those logs in the Event Log.
- Microsoft-Windows-DNS-Client (1014): DNS name resolution timeout; this event type can also be very helpful when searching for malware or when trying to find out whether a user has tried to connect to a specific website or service.
- Firewall-Rule-Add/Change/Delete (2004, 2005, 2006, 2033): If the client workstations are taking advantage of the built-in host-based Windows Firewall, then there is value in collecting events to track the firewall status. Normal users should not be modifying the firewall rules of their local machine.
- Microsoft-Windows-Windows Defender (3004): Windows Defender malware detection logs.
- Microsoft-Windows-Security-Auditing (4720, 4724, 4725, 4728, 4732, 4635, 4740, 4748, 4756): In these logs, you can find information such as remote desktop logins and users that have been added to privileged groups, and account lockouts can also be tracked. User accounts that are being promoted to the privileged groups should be audited very closely to ensure that the users are, in fact, supposed to be in a privileged group. Unauthorized membership of the privileged groups is a strong indicator that a malicious activity has occurred.
- Service-Control-Manager (7030, 7045): It monitors whether a service is configured to interact with the desktop or has been installed on the system in general.
- App-Locker-Block/Warning (8003, 8004, 8006, 8007): Application whitelisting events should be collected to look for the applications that have been blocked from execution. Any blocked application could be malware or the users trying to run an unapproved software.
Harlan Carvey stated in one of his blog posts (http://windowsir.blogspot.de/2014/10/windows-event-logs.html) that beyond individual event records (source/ID pairs), one of the aspects of the newer versions of Windows (in particular, Windows 7) is that there are a lot of events that are being recorded by default across multiple Event Log files. Thus, when some events occur, multiple event records are stored in different Event Log types and often across different Event Log files. For example, when a user logs in to a system on the console, there will be an event recorded in the security event log, a couple of events will be recorded in the Microsoft-Windows-TerminalServices-LocalSessionManager/Operational log, and a couple of events will also be recorded in the Microsoft-Windows-TaskScheduler/Operational log.
The Event Log can also be used to detect whether an attacker has used some kind of anti-forensic techniques. One of those techniques would be to change the system time in order to mislead an investigator. To detect this kind of modification, an investigator has to list all the available Event Log records by the sequence number and generated time. If the system time has been rolled back, there would be a point where the time an event has been generated was before the previous event. Some more examples of detecting anti-forensic techniques with the help of Windows Event Log can be found in a blog post by Harlan Carvey, at http://windowsir.blogspot.de/2013/07/howto-determinedetect-use-of-anti.html.
When talking about Event Logs and analyzing these logs with Python, there is no way to get around python-evtx. These scripts (https://github.com/williballenthin/python-evtx) have been developed using the 2.7+ tags of the Python programming language. As it is purely Python, the module works equally well across the platforms. The code does not depend on any modules that require separate compilation and operates on the event log files from the Windows operating systems that are newer than Windows Vista that is EVTX.
The second tool that we want to bring to your attention is plaso, (refer to http://plaso.kiddaland.net/). This tool set has evolved from log2timeline and is now build in Python. With the help of this tool set, you can create meaningful timelines of the system events and other log files (for example, Apache). There is also a very good cheat sheet, http://digital-forensics.sans.org/media/log2timeline_cheatsheet.pdf, for log2timeline that demonstrates the real power of this tool. One of the big advantages of this tool set is the fact that you can even run it on a full image of a system to generate a timeline of all actions that the users performed on that system before creating the image.
In the following sections, we will show some examples of how to use python-evtx to find IOC in the Windows Event Log and how plaso will help you identify more IOCs and display them in a nicely formatted timeline.
First of all, we want to start with a basic conversion of the binary XML format of EVTX files to the readable XML files. This can be done using evtxdump.py, https://github.com/williballenthin/python-evtx, which will also be the basis of our following scripts:
#!/usr/bin/env python
import mmap
import contextlib
import argparse
from Evtx.Evtx import FileHeader
from Evtx.Views import evtx_file_xml_view
def main():
parser = argparse.ArgumentParser(description="Dump a binary EVTX file into XML.")
parser.add_argument("--cleanup", action="store_true", help="Cleanup unused XML entities (slower)"),
parser.add_argument("evtx", type=str, help="Path to the Windows EVTX event log file")
args = parser.parse_args()
with open(args.evtx, 'r') as f:
with contextlib.closing(mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)) as buf:
fh = FileHeader(buf, 0x0)
print "<?xml version="1.0" encoding="utf-8" standalone="yes" ?>"
print "<Events>"
for xml, record in evtx_file_xml_view(fh):
print xml
print "</Events>"
if __name__ == "__main__":
main()When dumping a logon event (event id 4724) with the help of the previously mentioned script, the result will look similar to the following:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"><System><Provider Name="Microsoft-Windows-Security-Auditing" Guid="54849625-5478-4994-a5ba-3e3b0328c30d"></Provider> <EventID Qualifiers="">4724</EventID> <Version>0</Version> <Level>0</Level> <Task>13824</Task> <Opcode>0</Opcode> <Keywords>0x8020000000000000</Keywords> <TimeCreated SystemTime="2013-11-21 10:40:51.552799"></TimeCreated> <EventRecordID>115</EventRecordID> <Correlation ActivityID="" RelatedActivityID=""></Correlation> <Execution ProcessID="452" ThreadID="1776"></Execution> <Channel>Security</Channel> <Computer>windows</Computer> <Security UserID=""></Security> </System> <EventData><Data Name="TargetUserName">mspreitz</Data> <Data Name="TargetDomainName">windows</Data> <Data Name="TargetSid">S-1-5-21-3147386740-1191307685-1965871575-1000</Data> <Data Name="SubjectUserSid">S-1-5-18</Data> <Data Name="SubjectUserName">WIN-PC9VCSAQB0H$</Data> <Data Name="SubjectDomainName">WORKGROUP</Data> <Data Name="SubjectLogonId">0x00000000000003e7</Data> </EventData> </Event>
When using evtxdump.py, https://github.com/williballenthin/python-evtx, with a large Windows Event Log file, the output will be very large as you will find all the recorded logs in the generated XML file. For an analyst, it is often important to perform a fast triage or search for specific events quickly. Due to this, we modify the script in a way that it is possible to extract only specific events, as shown in the following:
#!/usr/bin/env python
import mmap
import contextlib
import argparse
from xml.dom import minidom
from Evtx.Evtx import FileHeader
from Evtx.Views import evtx_file_xml_view
def main():
parser = argparse.ArgumentParser(description="Dump specific event ids from a binary EVTX file into XML.")
parser.add_argument("--cleanup", action="store_true", help="Cleanup unused XML entities (slower)"),
parser.add_argument("evtx", type=str, help="Path to the Windows EVTX event log file")
parser.add_argument("out", type=str, help="Path and name of the output file")
parser.add_argument("--eventID", type=int, help="Event id that should be extracted")
args = parser.parse_args()
outFile = open(args.out, 'a+')
with open(args.evtx, 'r') as f:
with contextlib.closing(mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)) as buf:
fh = FileHeader(buf, 0x0)
outFile.write("<?xml version="1.0" encoding="utf-8" standalone="yes" ?>")
outFile.write("<Events>")
for xml, record in evtx_file_xml_view(fh):
xmldoc = minidom.parseString(xml)
event_id = xmldoc.getElementsByTagName('EventID')[0].childNodes[0].nodeValue
if event_id == str(args.eventID):
outFile.write(xml)
else:
continue
outFile.write("</Events>")
if __name__ == "__main__":
main()If you now want to extract all logon events from the security event log of a Windows system in a given XML file, you just have to execute the script as follows:
user@lab:~$ ./evtxdump.py security.evtx logon_events.xml –eventID 4724
In this section, we will demonstrate how to find logon and logoff events on a Terminal Server. The Terminal Services logon and logoff events can be tagged using plasm and filtered using psort to get a quick overview of which users have been logging in to a machine and when and where from. This information can be very helpful when searching for compromises. To start with plaso, you first need to tag all your data. Tagging with plaso is as easy as shown in the following:
user@lab:~$ ./plasm.py tag --tagfile=tag_windows.txt storage_file
After successful tagging, you can search the storage file for tags with the following command:
user@lab:~$ ./psort.py storage_file "tag contains 'Session logon succeeded'"
The result of this command execution will show you all the successful logon events on a given system. Similar commands can be executed when searching for the services that are started or EMET failures.
Now, that you have seen the kind of data that you are able to extract from Windows Event Log, we will show you a second component of Microsoft Windows that is really helpful when searching for IOC or when trying to reconstruct the user behavior.