Unloaded Module

One of the frequent problems is an access violation at an address that belongs to an unloaded module. Here's an example that recently happened on our machine during an auto-update of the popular software package so we immediately attached a debugger after seeing a WER dialog box:

0:000> ~*k

. 0 Id: bc8.bcc Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0035f1c4 771a0bdd ntdll!ZwWaitForMultipleObjects+0x15
0035f260 75771a2c KERNELBASE!WaitForMultipleObjectsEx+0x100
0035f2a8 75774208 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0035f2c4 757980a4 kernel32!WaitForMultipleObjects+0x18
0035f330 75797f63 kernel32!WerpReportFaultInternal+0x186
0035f344 75797858 kernel32!WerpReportFault+0x70
0035f354 757977d7 kernel32!BasepReportFault+0x20
0035f3e0 77ec74df kernel32!UnhandledExceptionFilter+0x1af
0035f3e8 77ec73bc ntdll!__RtlUserThreadStart+0x62
0035f3fc 77ec7261 ntdll!_EH4_CallFilterFunc+0x12
0035f424 77eab459 ntdll!_except_handler4+0x8e
0035f448 77eab42b ntdll!ExecuteHandler2+0x26
0035f46c 77eab3ce ntdll!ExecuteHandler+0x24
0035f4f8 77e60133 ntdll!RtlDispatchException+0x127
0035f4f8 73eb2200 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0035f844 76e462fa <Unloaded_fpb.tmp>+0×12200
0035f870 76e46d3a USER32!InternalCallWinProc+0×23
0035f8e8 76e4965e USER32!UserCallWinProcCheckWow+0×109
0035f92c 76e496c5 USER32!SendMessageWorker+0×581
0035f950 7269c05c USER32!SendMessageW+0×7f
0035f9ec 7270be62 comctl32!CCSendNotify+0xc19
0035fa28 75f6f52a comctl32!SendNotify+0×36
0035fa4c 75f61d66 SHELL32!SetAppStartingCursor+0×6d
0035fa64 75f61ee2 SHELL32!CShellExecute::ExecuteNormal+0×16
0035fa78 75f61e70 SHELL32!ShellExecuteNormal+0×33
0035fa90 75f53cd0 SHELL32!ShellExecuteExW+0×62
0035fae4 003e2211 SHELL32!ShellExecuteW+0×77
0035fbc4 77e838be InstallFlashPlayer+0×2211
0035fcb4 77e83492 ntdll!RtlpFreeHeap+0xbb1
0035fcd4 757714dd ntdll!RtlFreeHeap+0×142
0035fce8 003f0324 kernel32!HeapFree+0×14
0035fd80 003f0241 InstallFlashPlayer+0×10324
0035fe10 7577339a InstallFlashPlayer+0×10241
0035fe1c 77e89ef2 kernel32!BaseThreadInitThunk+0xe
0035fe5c 77e89ec5 ntdll!__RtlUserThreadStart+0×70
0035fe74 00000000 ntdll!_RtlUserThreadStart+0×1b

1 Id: bc8.6b0 Suspend: 2 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
03e1f9e0 77ea2f51 ntdll!ZwWaitForMultipleObjects+0x15
03e1fb74 7577339a ntdll!TppWaiterpThread+0x33d
03e1fb80 77e89ef2 kernel32!BaseThreadInitThunk+0xe
03e1fbc0 77e89ec5 ntdll!__RtlUserThreadStart+0x70
03e1fbd8 00000000 ntdll!_RtlUserThreadStart+0x1b

2 Id: bc8.8dc Suspend: 2 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
03f5fd50 77ea3352 ntdll!NtWaitForWorkViaWorkerFactory+0x12
03f5feb0 7577339a ntdll!TppWorkerThread+0x216
03f5febc 77e89ef2 kernel32!BaseThreadInitThunk+0xe
03f5fefc 77e89ec5 ntdll!__RtlUserThreadStart+0x70
03f5ff14 00000000 ntdll!_RtlUserThreadStart+0x1b

3 Id: bc8.944 Suspend: 2 Teb: 7efaf000 Unfrozen
ChildEBP RetAddr
0416f8b4 77ea3352 ntdll!NtWaitForWorkViaWorkerFactory+0x12
0416fa14 7577339a ntdll!TppWorkerThread+0x216
0416fa20 77e89ef2 kernel32!BaseThreadInitThunk+0xe
0416fa60 77e89ec5 ntdll!__RtlUserThreadStart+0x70
0416fa78 00000000 ntdll!_RtlUserThreadStart+0x1b

The exception thread shows fpb.tmp module as unloaded:

0:000> lmv m fpb.tmp
start    end        module name

Unloaded modules:
00cb0000 00d5a000    fpb.tmp
Timestamp: Fri Jun 01 02:56:00 2012 (4FC82130)
Checksum:  000B0CD5
ImageSize:  000AA000
73ea0000 73f15000    fpb.tmp
Timestamp: Fri Jun 01 02:49:25 2012 (4FC81FA5)
Checksum:  0007A7CE
ImageSize:  00075000

We change the exception thread context to get registers at the time of the exception:

0:000> kv
ChildEBP RetAddr Args to Child
0035f1c4 771a0bdd 00000002 0035f214 00000001 ntdll!ZwWaitForMultipleObjects+0x15
0035f260 75771a2c 0035f214 0035f288 00000000
KERNELBASE!WaitForMultipleObjectsEx+0x100
0035f2a8 75774208 00000002 7efde000 00000000
kernel32!WaitForMultipleObjectsExImplementation+0xe0
0035f2c4 757980a4 00000002 0035f2f8 00000000 kernel32!WaitForMultipleObjects+0x18
0035f330 75797f63 0035f410 00000001 00000001 kernel32!WerpReportFaultInternal+0x186
0035f344 75797858 0035f410 00000001 0035f3e0 kernel32!WerpReportFault+0x70
0035f354 757977d7 0035f410 00000001 658587c7 kernel32!BasepReportFault+0x20
0035f3e0 77ec74df 00000000 77ec73bc 00000000 kernel32!UnhandledExceptionFilter+0x1af
0035f3e8 77ec73bc 00000000 0035fe5c 77e7c530 ntdll!__RtlUserThreadStart+0x62
0035f3fc 77ec7261 00000000 00000000 00000000 ntdll!_EH4_CallFilterFunc+0x12
0035f424 77eab459 fffffffe 0035fe4c 0035f560 ntdll!_except_handler4+0x8e
0035f448 77eab42b 0035f510 0035fe4c 0035f560 ntdll!ExecuteHandler2+0x26
0035f46c 77eab3ce 0035f510 0035fe4c 0035f560 ntdll!ExecuteHandler+0x24
0035f4f8 77e60133 0135f510 0035f560 0035f510 ntdll!RtlDispatchException+0x127
0035f4f8 73eb2200 0135f510 0035f560 0035f510 ntdll!KiUserExceptionDispatcher+0xf
(CONTEXT @ 0035f560)
WARNING: Frame IP not in any known module. Following frames may be wrong.
0035f844 76e462fa 000201ce 0000004e 00000000 <Unloaded_fpb.tmp>+0×12200
0035f870 76e46d3a 73eb2200 000201ce 0000004e USER32!InternalCallWinProc+0×23
0035f8e8 76e4965e 00000000 73eb2200 000201ce USER32!UserCallWinProcCheckWow+0×109
0035f92c 76e496c5 013907f0 00000000 73eb2200 USER32!SendMessageWorker+0×581
0035f950 7269c05c 000201ce 0000004e 00000000 USER32!SendMessageW+0×7f
0035f9ec 7270be62 0035fa00 fffffff7 00000000 comctl32!CCSendNotify+0xc19
0035fa28 75f6f52a 000201ce 00000000 fffffff7 comctl32!SendNotify+0×36
0035fa4c 75f61d66 000201ce 00000001 00001500 SHELL32!SetAppStartingCursor+0×6d
0035fa64 75f61ee2 0035faa4 00001500 0035faa4
SHELL32!CShellExecute::ExecuteNormal+0×16
0035fa78 75f61e70 0035faa4 00001500 00000200 SHELL32!ShellExecuteNormal+0×33
0035fa90 75f53cd0 0035faa4 003fb654 003fa554 SHELL32!ShellExecuteExW+0×62
0035fae4 003e2211 000201ce 003fa554 0035fb14 SHELL32!ShellExecuteW+0×77
0035fbc4 77e838be 00da0138 77e8389a 77c467ad InstallFlashPlayer+0×2211
0035fcb4 77e83492 00000000 00da2320 00da2320 ntdll!RtlpFreeHeap+0xbb1
0035fcd4 757714dd 00da0000 00000000 00da2320 ntdll!RtlFreeHeap+0×142
0035fce8 003f0324 00da0000 00000000 003f0343 kernel32!HeapFree+0×14
0035fd80 003f0241 003e0000 00000000 010d3135 InstallFlashPlayer+0×10324
0035fe10 7577339a 7efde000 0035fe5c 77e89ef2 InstallFlashPlayer+0×10241
0035fe1c 77e89ef2 7efde000 77c46545 00000000 kernel32!BaseThreadInitThunk+0xe
0035fe5c 77e89ec5 003f02ac 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
0035fe74 00000000 003f02ac 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

0:000> .cxr 0035f560
eax=73eb2200 ebx=00000000 ecx=01080d68 edx=00000000 esi=73eb2200
edi=00000000
eip=73eb2200 esp=0035f848 ebp=0035f870 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00210206
<Unloaded_fpb.tmp>+0×12200:
73eb2200 ?? ???

Then we double check that a window procedure was indeed called from that module range:

0:000> kv
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
0035f844 76e462fa 000201ce 0000004e 00000000 <Unloaded_fpb.tmp>+0×12200
0035f870 76e46d3a 73eb2200 000201ce 0000004e USER32!InternalCallWinProc+0×23
0035f8e8 76e4965e 00000000 73eb2200 000201ce USER32!UserCallWinProcCheckWow+0×109
0035f92c 76e496c5 013907f0 00000000 73eb2200 USER32!SendMessageWorker+0×581
0035f950 7269c05c 000201ce 0000004e 00000000 USER32!SendMessageW+0×7f
0035f9ec 7270be62 0035fa00 fffffff7 00000000 comctl32!CCSendNotify+0xc19
0035fa28 75f6f52a 000201ce 00000000 fffffff7 comctl32!SendNotify+0×36
0035fa4c 75f61d66 000201ce 00000001 00001500 SHELL32!SetAppStartingCursor+0×6d
0035fa64 75f61ee2 0035faa4 00001500 0035faa4
SHELL32!CShellExecute::ExecuteNormal+0×16
0035fa78 75f61e70 0035faa4 00001500 00000200 SHELL32!ShellExecuteNormal+0×33
0035fa90 75f53cd0 0035faa4 003fb654 003fa554 SHELL32!ShellExecuteExW+0×62
0035fae4 003e2211 000201ce 003fa554 0035fb14 SHELL32!ShellExecuteW+0×77
0035fbc4 77e838be 00da0138 77e8389a 77c467ad InstallFlashPlayer+0×2211
0035fcb4 77e83492 00000000 00da2320 00da2320 ntdll!RtlpFreeHeap+0xbb1
00da15a0 00000000 00da1780 02971450 003e1000 ntdll!RtlFreeHeap+0×142

0:000> ub 76e462fa
USER32!InternalCallWinProc+0×6:
76e462dd 68cdabbadc      push    0DCBAABCDh
76e462e2 56              push    esi
76e462e3 ff7518          push    dword ptr [ebp+18h]
76e462e6 ff7514          push    dword ptr [ebp+14h]
76e462e9 ff7510          push    dword ptr [ebp+10h]
76e462ec ff750c          push    dword ptr [ebp+0Ch]
76e462ef 64800dca0f000001 or      byte ptr fs:[0FCAh],1
76e462f7 ff5508          call    dword ptr [ebp+8]

We now get a memory value pointed to by an EBP+8 address:

0:000> r
Last set context:
eax=73eb2200 ebx=00000000 ecx=01080d68 edx=00000000 esi=73eb2200
edi=00000000
eip=73eb2200 esp=0035f848 ebp=0035f870 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00210206
<Unloaded_fpb.tmp>+0×12200:
73eb2200 ?? ???

0:000> dp 0035f870+8 l1
0035f878 73eb2200

0:000> dd 73eb2200
73eb2200 ???????? ???????? ???????? ????????
73eb2210 ???????? ???????? ???????? ????????
73eb2220 ???????? ???????? ???????? ????????
73eb2230 ???????? ???????? ???????? ????????
73eb2240 ???????? ???????? ???????? ????????
73eb2250 ???????? ???????? ???????? ????????
73eb2260 ???????? ???????? ???????? ????????
73eb2270 ???????? ???????? ???????? ????????

The value is indeed belongs to unloaded fpb.tmp module address range:

0:000> ln 73eb2200
(73eb2200) <Unloaded_fpb.tmp>+0×12200