Malware uses a variety of techniques to scan for indications that a debugger is attached, including using the Windows API, manually checking memory structure for debugging artifacts, and searching the system for residue left by a debugger. Debugger detection is the most common way that malware performs anti-debugging.
The use of Windows API functions is the most obvious of the anti-debugging techniques. The Windows API provides several functions that can be used by a program to determine if it is being debugged. Some of these functions were designed for debugger detection; others were designed for different purposes but can be repurposed to detect a debugger. A few of these functions use functionality not documented in the API.
Typically, the easiest way to overcome a call to an anti-debugging API function is to manually modify the malware during execution to not call these functions or to modify the flag’s post call to ensure that the proper path is taken. A more difficult option would be to hook these functions, as with a rootkit.
The following Windows API functions can be used for anti-debugging:
IsDebuggerPresent
The simplest API function for detecting a debugger is IsDebuggerPresent
. This function searches the Process Environment Block (PEB) structure
for the field IsDebugged
, which will return zero if you are not
running in the context of a debugger or a nonzero value if a debugger is attached. We’ll
discuss the PEB structure in more detail in the next section.
CheckRemoteDebuggerPresent
This API function is nearly identical to IsDebuggerPresent
.
The name is misleading though, as it does not check for a debugger on a remote machine, but rather
for a process on the local machine. It also checks the PEB structure for the IsDebugged
field; however, it can do so for itself or another process on
the local machine. This function takes a process handle as a parameter and will check if that
process has a debugger attached. CheckRemoteDebuggerPresent
can
be used to check your own process by simply passing a handle to your process.
NtQueryInformationProcess
This is a native API function in Ntdll.dll that retrieves information
about a given process. The first parameter to this function is a process handle; the second is used
to tell the function the type of process information to be retrieved. For example, using the value
ProcessDebugPort
(value 0x7
)
for this parameter will tell you if the process in question is currently being debugged. If the
process is not being debugged, a zero will be returned; otherwise, a port number will be
returned.
This function is used to send a string to a debugger for display. It can be used to detect the
presence of a debugger. For example, Example 16-1 uses
SetLastError
to set the current error code to an arbitrary value.
If OutputDebugString
is called and there is no debugger attached,
GetLastError
should no longer contain our arbitrary value,
because an error code will be set by the OutputDebugString
function if it fails. If OutputDebugString
is called and there is
a debugger attached, the call to OutputDebugString
should
succeed, and the value in GetLastError
should not be
changed.
Using the Windows API may be the most obvious method for detecting the presence of a debugger, but manually checking structures is the most common method used by malware authors. There are many reasons why malware authors are discouraged from using the Windows API for anti-debugging. For example, the API calls could be hooked by a rootkit to return false information. Therefore, malware authors often choose to perform the functional equivalent of the API call manually, rather than rely on the Windows API.
In performing manual checks, several flags within the PEB structure provide information about the presence of a debugger. Here, we’ll look at some of the commonly used flags for checking for a debugger.
A Windows PEB structure is maintained by the OS for each running process, as shown in the example in Example 16-2. It contains all user-mode parameters associated with a process. These parameters include the process’s environment data, which itself includes environment variables, the loaded modules list, addresses in memory, and debugger status.
Example 16-2. Documented Process Environment Block (PEB) structure
typedef struct _PEB {
BYTE Reserved1[2];
BYTE BeingDebugged
;
BYTE Reserved2[1];
PVOID Reserved3[2];
PPEB_LDR_DATA Ldr;
PRTL_USER_PROCESS_PARAMETERS ProcessParameters;
BYTE Reserved4[104];
PVOID Reserved5[52];
PPS_POST_PROCESS_INIT_ROUTINE PostProcessInitRoutine;
BYTE Reserved6[128];
PVOID Reserved7[1];
ULONG SessionId;
} PEB, *PPEB;
While a process is running, the location of the PEB can be referenced by the location
fs:[30h]
. For anti-debugging, malware will use that location to
check the BeingDebugged
flag, which indicates whether the
specified process is being debugged. Table 16-1
shows two examples of this type of check.
In the code on the left in Table 16-1, the
location of the PEB is moved into EAX. Next, this offset plus 2 is moved into EBX, which corresponds
to the offset into the PEB of the location of the BeingDebugged
flag. Finally, EBX is checked to see if it is zero. If so, a debugger is not attached, and the jump
will be taken.
Another example is shown on the right side of Table 16-1. The location of the PEB is moved into EDX
using a push/pop combination of instructions, and then the BeingDebugged
flag at offset 2 is directly compared to 1.
This check can take many forms, and, ultimately, the conditional jump determines the code path. You can take one of the following approaches to surmount this problem:
Force the jump to be taken (or not) by manually modifying the zero flag immediately before the jump instruction is executed. This is the easiest approach.
Manually change the BeingDebugged
flag to zero.
Both options are generally effective against all of the techniques described in this section.
An undocumented location within the Reserved4
array
(shown in Example 16-2), known as ProcessHeap
, is set to the location of a process’s first heap
allocated by the loader. ProcessHeap
is located at 0x18 in the
PEB structure. This first heap contains a header with fields used to tell the kernel whether the
heap was created within a debugger. These are known as the ForceFlags
and Flags
fields.
Offset 0x10 in the heap header is the ForceFlags
field on
Windows XP, but for Windows 7, it is at offset 0x44 for 32-bit applications. Malware may also look
at offset 0x0C on Windows XP or offset 0x40 on Windows 7 for the Flags
field. This field is almost always equal to the ForceFlags
field, but is usually ORed with the value 2.
Example 16-3 shows the assembly code for this technique. (Note that two separate dereferences must occur.)
Example 16-3. Manual ProcessHeap
flag check
mov eax, large fs:30h mov eax, dword ptr [eax+18h] cmp dword ptr ds:[eax+10h], 0 jne DebuggerDetected
The best way to overcome this technique is to change the ProcessHeap
flag manually or to use a hidedebug plug-in for your debugger. If you are
using WinDbg, you can start the program with the debug heap disabled. For example, the command
windbg –hd notepad.exe
will start the heap in normal mode
as opposed to debug mode, and the flags we’ve discussed won’t be set.
Since processes run slightly differently when started with a debugger, they create memory heaps differently. The information that the system uses to determine how to create heap structures is stored at an undocumented location in the PEB at offset 0x68. If the value at this location is 0x70, we know that we are running in a debugger.
The value of 0x70 is a combination of the following flags when a heap is created by a debugger. These flags are set for the process if it is started from within a debugger.
(FLG_HEAP_ENABLE_TAIL_CHECK | FLG_HEAP_ENABLE_FREE_CHECK | FLG_HEAP_VALIDATE_PARAMETERS)
Example 16-4 shows the assembly code for performing this check.
Example 16-4. NTGlobalFlag
check
mov eax, large fs:30h cmp dword ptr ds:[eax+68h], 70h jz DebuggerDetected
The easiest way to overcome this technique is to change the flags manually or with a hidedebug plug-in for your debugger. If you are using WinDbg, you can start the program with the debug heap option disabled, as mentioned in the previous section.
When analyzing malware, we typically use debugging tools, which leave residue on the system. Malware can search for this residue in order to determine when you are attempting to analyze it, such as by searching registry keys for references to debuggers. The following is a common location for a debugger:
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionAeDebug
This registry key specifies the debugger that activates when an application error occurs. By default, this is set to Dr. Watson, so if it is changed to something like OllyDbg, malware may determine that it is under a microscope.
Malware can also search the system for files and directories, such as common debugger program
executables, which are typically present during malware analysis. (Many backdoors already have code
in place to traverse filesystems.) Or the malware can detect residue in live memory, by viewing the
current process listing or, more commonly, by performing a FindWindow
in search of a debugger, as shown in Example 16-5.
Example 16-5. C code for FindWindow
detection
if(FindWindow
("OLLYDBG", 0) == NULL)
{
//Debugger Not Found
}
else
{
//Debugger Detected
}
In this example, the code simply looks for a window named OLLYDBG
.