PART 2: Crash Dump Analysis Patterns

Divide by Zero (Kernel Mode)

This is a kernel mode counterpart of Divide by Zero pattern in user mode (Volume 3, page 96). It manifests under different bugchecks, for example:

1: kd> !analyze -v

[...]

UNEXPECTED_KERNEL_MODE_TRAP (7f)
This means a trap occurred in kernel mode, and it's a trap of a kind that the kernel
isn't allowed to have/catch (bound trap) or that is always instant death (double
fault).  The first number in the bugcheck params is the number of the trap (8 =
double fault, etc) Consult an Intel x86 family manual to learn more about what these
traps are. Here is a *portion* of those codes:
If kv shows a taskGate
         use .tss on the part before the colon, then kv.
Else if kv shows a trapframe
         use .trap on that value
Else
         .trap on the appropriate frame will show where the trap was taken
         (on x86, this will be the ebp that goes with the procedure KiTrap)
Endif
kb will then show the corrected stack.
Arguments:
Arg1: 00000000, EXCEPTION_DIVIDED_BY_ZERO
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

[...]

TRAP_FRAME: a8954c8c -- (.trap 0xffffffffa8954c8c)
ErrCode = 00000000
eax=ffffffff ebx=00000000 ecx=00000005 edx=00000000 esi=00000000 edi=00000000
eip=975c42cd esp=a8954d00 ebp=a8954d4c iopl=0 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246
win32k!NtGdiEnumObjects+0xc6:
975c42cd f7f6 div eax,esi
Resetting default scope

PROCESS_NAME: Application.EXE

[...]
STACK_TEXT:
a8954c2c 81ac2b76 0000007f 5317512a 975c42cd nt!KeBugCheck+0x14
a8954c80 81899808 a8954c8c a8954d4c 975c42cd nt!Ki386CheckDivideByZeroTrap+0×44
a8954c80 975c42cd a8954c8c a8954d4c 975c42cd nt!KiTrap00+0×88
a8954d4c 81898a7a 062102ce 00000001 00000000 Driver!EnumObjects+0xc6
a8954d4c 77a59a94 062102ce 00000001 00000000 nt!KiFastCallEntry+0×12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012ca70 00000000 00000000 00000000 00000000 0×77a59a94

0: kd> !analyze -v

[...]

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000094, Exception code that caused the bugcheck
Arg2: fffff9600025ba6d, Address of the exception record for the exception that caused
the bugcheck
Arg3: fffff8800ac361d0, Address of the context record for the exception that caused
the bugcheck
Arg4: 0000000000000000, zero.

[...]

EXCEPTION_CODE: (NTSTATUS) 0xc0000094 - {EXCEPTION} Integer division by zero.

FAULTING_IP:
Driver!EnumObjects+e9
fffff960`0025ba6d f7f6 div eax,esi


CONTEXT: fffff8800ac361d0 -- (.cxr 0xfffff8800ac361d0)
rax=00000000ffffffff rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff9600025ba6d rsp=fffff8800ac36ba0 rbp=fffff8800ac36ca0
r8=0000000000000000 r9=0000000000000000 r10=0000000005892f18
r11=fffff900c28379e0 r12=0000000000000000 r13=0000000000000002
r14=0000000000000001 r15=0000000000000000
iopl=0 nv up ei ng nz na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010286
Driver!EnumObjects+0xe9:
fffff960`0025ba6d f7f6 div eax,esi
Resetting default scope

[...]

STACK_TEXT:
fffff880`0ac36ba0 fffff800`01682993 Driver!EnumObjects+0xe9
fffff880`0ac36c20 00000000`748a1b3a nt!KiSystemServiceCopyEnd+0x13
00000000`001cdf08 00000000`00000000 0x748a1b3a

Fat Process Dump

During repeated execution either on one computer or in parallel on many computers with a uniform software and hardware the given process VM size tends to cluster around some value range, for example, 40 - 60 MB. If we get a collection of user process memory dumps taken from several production servers, say 20 files, we can either employ scripts to process all of them (Volume 1, page 227) or compare their file size and look for a bigger ones for a starter, for example, 85 or 110 Mb. For certain processes, for example, a print spooler, after a software problem the process size tends to increase compared to normal execution. For other processes, certain error processing modules might be loaded increasing VM size or in case of incoming requests for a hang process certain memory regions like heap could increase as well contributing to a dump file size increase. If we have fat and thin clients we should also have thin and fat process dumps as well.

Blocked Queue

We provide an example of an ALPC port here. If we see an LPC/ALPC wait chain (Volume 3, page 97) endpoint or just have a message address (and optionally a port address) we can check the port queue length. For example, in a frozen system we have this WinDbg output:

THREAD fffffa8009db7160  Cid 03b0.2ec0  Teb: 000007fffffd5000 Win32Thread:
0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
      fffffa8009db7520  Semaphore Limit 0x1
Waiting for reply to ALPC Message fffff8a00dbc6650 : queued at port
fffffa800577ee60 : owned by process fffffa80056ddb30
Not impersonating
DeviceMap                 fffff8a000008b30
Owning Process            fffffa8005691b30       Image:
ServiceA.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      39742808       Ticks: 3469954 (0:15:02:11.629)
Context Switch Count      9
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0×0000000076cd8e70
Stack Init fffff8800bf60db0 Current fffff8800bf60620
Base fffff8800bf61000 Limit fffff8800bf5b000 Call 0
Priority 10 BasePriority 9 UnusualBoost 0 ForegroundBoost 0 IoPriority 2
PagePriority 5
Kernel stack not resident.
Child-SP RetAddr Call Site
fffff880`0bf60660 fffff800`016de992 nt!KiSwapContext+0×7a
fffff880`0bf607a0 fffff800`016e0cff nt!KiCommitThreadWait+0×1d2
fffff880`0bf60830 fffff800`016f5d1f nt!KeWaitForSingleObject+0×19f
fffff880`0bf608d0 fffff800`019ddac6 nt!AlpcpSignalAndWait+0×8f
fffff880`0bf60980 fffff800`019dba50 nt!AlpcpReceiveSynchronousReply+0×46
fffff880`0bf609e0 fffff800`019d8fcb
nt!AlpcpProcessSynchronousRequest+0×33d
fffff880`0bf60b00 fffff800`016d6993 nt!NtAlpcSendWaitReceivePort+0×1ab
fffff880`0bf60bb0 00000000`76d105aa nt!KiSystemServiceCopyEnd+0×13
(TrapFrame @ fffff880`0bf60c20)
00000000`01efe638 000007fe`fec0aa76 ntdll!ZwAlpcSendWaitReceivePort+0xa
00000000`01efe640 000007fe`fecacb64 RPCRT4!LRPC_CCALL::SendReceive+0×156
00000000`01efe700 000007fe`fecacd55 RPCRT4!NdrpClientCall3+0×244
00000000`01efe9c0 000007fe`fcbf18a1 RPCRT4!NdrClientCall3+0xf2
[...]
0: kd> !alpc /m fffff8a00dbc6650
 Message @ fffff8a00dbc6650
   MessageID             : 0x0720 (1824)
   CallbackID            : 0x257C575 (39306613)
   SequenceNumber        : 0x00000002 (2)
   Type                  : LPC_REQUEST
   DataLength            : 0x0044 (68)
   TotalLength           : 0x006C (108)
   Canceled              : No
   Release               : No
   ReplyWaitReply        : No
   Continuation          : Yes
   OwnerPort             : fffffa8006a4bb10
[ALPC_CLIENT_COMMUNICATION_PORT]
   WaitingThread         : fffffa8009db7160
   QueueType             : ALPC_MSGQUEUE_PENDING
   QueuePort             : fffffa800577ee60 [ALPC_CONNECTION_PORT]
   QueuePortOwnerProcess : fffffa80056ddb30 (ServiceB.exe)
   ServerThread          : fffffa8007ead4d0
   QuotaCharged          : No
   CancelQueuePort       : 0000000000000000
   CancelSequencePort    : 0000000000000000
    CancelSequenceNumber : 0×00000000 (0)
   ClientContext         : 0000000002a60f40
   ServerContext         : 0000000000000000
   PortContext           : 000000000227a370
   CancelPortContext     : 0000000000000000
   SecurityData          : 0000000000000000
   View                  : 0000000000000000

0: kd> !alpc /p fffffa800577ee60
 Port @ fffffa800577ee60
   Type                      : ALPC_CONNECTION_PORT
   CommunicationInfo         : fffff8a0022435d0
     ConnectionPort          : fffffa800577ee60
     ClientCommunicationPort : 0000000000000000
     ServerCommunicationPort : 0000000000000000
   OwnerProcess              : fffffa80056ddb30 (ServiceB.exe)
   SequenceNo                : 0×0000481A (18458)
   CompletionPort            : fffffa8005728e80
   CompletionList            : 0000000000000000
   MessageZone               : 0000000000000000
   ConnectionPending         : No
   ConnectionRefused         : No
   Disconnected              : No
   Closed                    : No
   FlushOnClose              : Yes
   ReturnExtendedInfo        : No
   Waitable                  : No
   Security                  : Static
   Wow64CompletionList       : No
Main queue is empty.

Large message queue is empty.
Pending queue has 698 message(s)

fffff8a002355aa0 00000404 0000000000001344:0000000000001358 0000000000000000 fffffa8004c0cb60
LPC_REQUEST
fffff8a00a52f030 00000644 0000000000001078:00000000000024c0 0000000000000000 fffffa80072f1b60
LPC_REQUEST
fffff8a00abb5030 000007a8 000000000000103c:000000000000050c 0000000000000000 fffffa800725b580
LPC_REQUEST
fffff8a00239cab0 000000b8 0000000000000480:00000000000015f8 0000000000000000 fffffa80077f0b60
LPC_REQUEST
fffff8a00ac81a90 00000a18 00000000000028ac:0000000000001e54 0000000000000000 fffffa8007fba060
LPC_CANCELED
fffff8a005879140 00000f80 0000000000001260:0000000000000730 fffffa8006432060 fffffa8006b18060
LPC_REQUEST
fffff8a013720d00 00000c6c 0000000000003764:00000000000032a8 0000000000000000 fffffa8006b00a60
LPC_CANCELED
fffff8a00ac82660 00000810 0000000000003af4:0000000000002a98 0000000000000000 fffffa80068c0b60
LPC_CANCELED
fffff8a00bdeca50 00000ec8 000000000000233c:00000000000013f8 0000000000000000 fffffa80079455b0
LPC_CANCELED
fffff8a00b662830 000005cc 00000000000005e4:0000000000000e0c fffffa800791a7a0 fffffa8007376580
LPC_REQUEST
fffff8a003d57150 00000f08 0000000000002678:0000000000003e0c 0000000000000000 fffffa8007e4a870
LPC_CANCELED
fffff8a00cd08830 00000750 0000000000003408:0000000000003adc 0000000000000000 fffffa8008631b60
LPC_CANCELED
fffff8a01855b2f0 000004f4 0000000000002c74:0000000000002d00 0000000000000000 fffffa800746b890
LPC_CANCELED
fffff8a00da0d0b0 00000db0 0000000000001a34:0000000000002d80 0000000000000000 fffffa800aff4b60
LPC_CANCELED
fffff8a00eddb030 0000059c 0000000000003f34:0000000000003c8c 0000000000000000 fffffa8008f96060
LPC_CANCELED
fffff8a017a14d00 00000920 0000000000003850:0000000000002588 0000000000000000 fffffa8009f66060
LPC_CANCELED
fffff8a01792d030 000007f8 0000000000003844:00000000000028d0 0000000000000000 fffffa800ad56260
LPC_CANCELED
fffff8a00f8d6ae0 00000f30 000000000000239c:0000000000001694 0000000000000000 fffffa8008b86060
LPC_CANCELED
fffff8a01395ab80 00000cdc 0000000000003630:00000000000018f8 0000000000000000 fffffa8005bc0770
LPC_CANCELED
fffff8a0166ff800 00000984 00000000000005e4:00000000000025f4 fffffa8009718910 fffffa8008cbfb60
LPC_REQUEST
fffff8a012b9f5a0 00000ac8 0000000000002d34:0000000000001b24 0000000000000000 fffffa8009cd8410
LPC_CANCELED
fffff8a014313830 00000afc 00000000000005e4:00000000000023bc fffffa80073f0230 fffffa80054d7060
LPC_REQUEST
fffff8a00a34a6b0 00000ca8 0000000000002534:0000000000002dd0 0000000000000000 fffffa80064c3980
LPC_CANCELED
[...]
fffff8a00ad8f610 00000e64 0000000000003714:00000000000030b8 0000000000000000 fffffa800aeea9f0
LPC_REQUEST
fffff8a015720710 00001594 0000000000003638:00000000000029b8 0000000000000000 fffffa800b5359a0
LPC_REQUEST
fffff8a009bac560 00001508 0000000000003994:0000000000001aac 0000000000000000 fffffa800b5359a0
LPC_REQUEST
fffff8a00b6e78f0 00001574 0000000000002938:0000000000001998 0000000000000000 fffffa800aeea9f0
LPC_REQUEST
fffff8a00b5716b0 00001570 0000000000002938:0000000000001698 0000000000000000 fffffa800a3b8620
LPC_REQUEST
fffff8a018531d00 00000db8 00000000000016d8:00000000000031c4 0000000000000000 fffffa800b5359a0
LPC_REQUEST
fffff8a01112f410 000014b0 0000000000001b6c:0000000000001618 0000000000000000 fffffa800a3b8620
LPC_CANCELED


Canceled queue is empty.

Crash Signature

This is one of the most common patterns. It consists of a set of attributes derivable from saved execution context for exceptions, faults and traps. For example, on x64 Windows it is usually RIP and RSP addresses. For x86 it is usually EIP, ESP and EBP. It can also include the application module name.

0:009> !analyze -v

[...]

FAILURE_BUCKET_ID: SOFTWARE_NX_FAULT_c0000005_ApplicationA.exe!Unknown

BUCKET_ID:
APPLICATION_FAULT_SOFTWARE_NX_FAULT_STACK_CORRUPTION_BAD_IP_ApplicationA+2
d560

[...]

0:009> kL
ChildEBP RetAddr
0354f270 75bc0962 ntdll!NtWaitForMultipleObjects+0x15
0354f30c 7651162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0354f354 76511921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0354f370 76539b0d kernel32!WaitForMultipleObjects+0x18
0354f3dc 76539baa kernel32!WerpReportFaultInternal+0x186
0354f3f0 765398d8 kernel32!WerpReportFault+0x70
0354f400 76539855 kernel32!BasepReportFault+0x20
0354f48c 77750727 kernel32!UnhandledExceptionFilter+0x1af
0354f494 77750604 ntdll!__RtlUserThreadStart+0x62
0354f4a8 777504a9 ntdll!_EH4_CallFilterFunc+0x12
0354f4d0 777387b9 ntdll!_except_handler4+0x8e
0354f4f4 7773878b ntdll!ExecuteHandler2+0x26
0354f5a4 776f010f ntdll!ExecuteHandler+0x24
0354f5a4 0354f958 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0354f908 02ff0340 0×354f958
00000000 00000000 0×2ff0340

0:009> kv
ChildEBP RetAddr Args to Child
[...]
0354f5a4 0354f958 0154f5bc 0354f60c 0354f5bc
ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 0354f60c)
WARNING: Frame IP not in any known module. Following frames may be wrong.
0354f908 02ff0340 00000000 00000000 00000000 0×354f958
00000000 00000000 00000000 00000000 00000000 0×2ff0340
0:009> .cxr 0354f60c
eax=80010105 ebx=0354f924 ecx=00000003 edx=0000ffff esi=00d7dce0
edi=00d7e0c8
eip=0354f958 esp=0354f8f4 ebp=0354f908 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
0354f958 64f9 stc

0:009> !address 0354f958
TEB 7efdd000 in range 7efdb000 7efde000
TEB 7efda000 in range 7efd8000 7efdb000
TEB 7efd7000 in range 7efd5000 7efd8000
TEB 7efaf000 in range 7efad000 7efb0000
TEB 7efac000 in range 7efaa000 7efad000
TEB 7efa9000 in range 7efa7000 7efaa000
TEB 7efa6000 in range 7efa4000 7efa7000
TEB 7efa3000 in range 7efa1000 7efa4000
TEB 7ef9f000 in range 7ef9d000 7efa0000
TEB 7ef9c000 in range 7ef9a000 7ef9d000
TEB 7ef99000 in range 7ef97000 7ef9a000
ProcessParametrs       007714b0 in range 00770000 00870000
Environment            007707f0 in range 00770000 00870000
03450000 : 0354d000 - 00003000
Type                   00020000 MEM_PRIVATE
Protect                00000004 PAGE_READWRITE
State                  00001000 MEM_COMMIT
Usage                  RegionUsageStack
Pid.Tid                1ea0.12dc

0:009> !address 02ff0340
TEB 7efdd000 in range 7efdb000 7efde000
TEB 7efda000 in range 7efd8000 7efdb000
TEB 7efd7000 in range 7efd5000 7efd8000
TEB 7efaf000 in range 7efad000 7efb0000
TEB 7efac000 in range 7efaa000 7efad000
TEB 7efa9000 in range 7efa7000 7efaa000
TEB 7efa6000 in range 7efa4000 7efa7000
TEB 7efa3000 in range 7efa1000 7efa4000
TEB 7ef9f000 in range 7ef9d000 7efa0000
TEB 7ef9c000 in range 7ef9a000 7ef9d000
TEB 7ef99000 in range 7ef97000 7ef9a000
ProcessParametrs       007714b0 in range 00770000 00870000
Environment            007707f0 in range 00770000 00870000
02fc0000 : 02fc0000 - 00043000
Type                   00020000 MEM_PRIVATE
Protect                00000004 PAGE_READWRITE
State                  00001000 MEM_COMMIT
Usage                  RegionUsageHeap
Handle 00d70000

Stack trace (Volume 1, page 395) may or may not be included here and it might be incorrect (Volume 1, page 288), heuristic (Volume 2, page 43) and not fully discernible automatically (requires raw stack semantic analysis) like in the example above. In some cases exception information might not be valid (Volume 5, page 116) though, for example, in the case of laterally damaged (Volume 1, page 264) or truncated (Volume 1, page 340) memory dump files.

Invalid Parameter (Process Heap)

This is a general pattern of passing unexpected values to functions. Here we look at invalid heap block parameter pattern specialization. It is different from heap corruption (Volume 1, page 257) or double free (Volume 1, page 378) pattern because no corruption happens in heap structures before detection and the parameter value has never been correct before its use. For example, we have this stack trace:

0:003> kL 100
ChildEBP RetAddr
01b2e6f0 77f27d0c ntdll!ZwWaitForSingleObject+0x15
01b2e774 77f27e3a ntdll!RtlReportExceptionEx+0x14b
01b2e7cc 77f4dc2e ntdll!RtlReportException+0x86
01b2e7e0 77f4dcab ntdll!RtlpTerminateFailureFilter+0x14
01b2e7ec 77ef05c4 ntdll!RtlReportCriticalFailure+0x67
01b2e800 77ef0469 ntdll!_EH4_CallFilterFunc+0x12
01b2e828 77ed8799 ntdll!_except_handler4+0x8e
01b2e84c 77ed876b ntdll!ExecuteHandler2+0x26
01b2e8fc 77e9010f ntdll!ExecuteHandler+0x24
01b2e8fc 77f4dc9b ntdll!KiUserExceptionDispatcher+0xf
01b2ecc4 77f4eba1 ntdll!RtlReportCriticalFailure+0x57
01b2ecd4 77f4ec81 ntdll!RtlpReportHeapFailure+0x21
01b2ed08 77efdda0 ntdll!RtlpLogHeapFailure+0xa1
01b2ed38 76bc14d1 ntdll!RtlFreeHeap+0×64
01b2ed4c 75694c39 kernel32!HeapFree+0×14
01b2ed98 726f167d msvcr80!free+0xcd
01b2eda4 7270613d DllA!FreeData+0xd
[...]
01b2fe38 77eb9d42 kernel32!BaseThreadInitThunk+0xe
01b2fe78 77eb9d15 ntdll!__RtlUserThreadStart+0×70
01b2fe90 00000000 ntdll!_RtlUserThreadStart+0×1b

We see that the failure was detected and logged immediately without any instrumentation information (Volume 5, page 108):

0:003> !gflag
Current NtGlobalFlag contents: 0x00000000

If we enable full page heap we get this default analysis output and the following stack trace:

0:003> !gflag
Current NtGlobalFlag contents: 0x02000000
hpa - Place heap allocations at ends of pages
0:003> !analyze -v

[...]

APPLICATION_VERIFIER_HEAPS_CORRUPTED_HEAP_BLOCK_EXCEPTION_RAISED_FOR_PROBI
NG (c)
Exception raised while verifying the heap block.
This situation happens if we really cannot determine any particular type
of corruption for the block. For instance you will get this if during a
heap free operation you pass an address that points to a non-accessible
memory area.
This can also happen for double free situations if we do not find the
block among full page heap blocks and we probe it as a light page heap
block.
Arguments:
Arg1: 05eb1000, Heap handle used in the call.
Arg2: 00720071, Heap block involved in the operation.
Arg3: 00000000, Size of the heap block.
Arg4: c0000005, Reserved.

[...]

0:003> kL 100
ChildEBP RetAddr
0818dca4 75fa0962 ntdll!ZwWaitForMultipleObjects+0x15
0818dd40 76bc162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0818dd88 76bc1921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0818dda4 76be9b0d kernel32!WaitForMultipleObjects+0x18
0818de10 76be9baa kernel32!WerpReportFaultInternal+0x186
0818de24 76be98d8 kernel32!WerpReportFault+0x70
0818de34 76be9855 kernel32!BasepReportFault+0x20
0818dec0 77ef06e7 kernel32!UnhandledExceptionFilter+0x1af
0818dec8 77ef05c4 ntdll!__RtlUserThreadStart+0x62
0818dedc 77ef0469 ntdll!_EH4_CallFilterFunc+0x12
0818df04 77ed8799 ntdll!_except_handler4+0x8e
0818df28 77ed876b ntdll!ExecuteHandler2+0x26
0818dfd8 77e9010f ntdll!ExecuteHandler+0x24
0818dfd8 71a6ba58 ntdll!KiUserExceptionDispatcher+0xf
0818e344 71a69ee0 verifier!VerifierStopMessage+0x1f8
0818e3a8 71a66f11 verifier!AVrfpDphReportCorruptedBlock+0x2b0
0818e3bc 71a819ec verifier!AVrfpDphFindBusyMemoryNoCheck+0x141
0818e3d0 71a8174e verifier!_EH4_CallFilterFunc+0x12
0818e3f8 77ed8799 verifier!_except_handler4+0x8e
0818e41c 77ed876b ntdll!ExecuteHandler2+0x26
0818e4cc 77e9010f ntdll!ExecuteHandler+0x24
0818e4cc 71a66e88 ntdll!KiUserExceptionDispatcher+0xf
0818e868 71a66f95 verifier!AVrfpDphFindBusyMemoryNoCheck+0xb8
0818e88c 71a67240 verifier!AVrfpDphFindBusyMemory+0x15
0818e8a8 71a69080
verifier!AVrfpDphFindBusyMemoryAndRemoveFromBusyList+0x20
0818e8c4 77f50aac verifier!AVrfDebugPageHeapFree+0x90
0818e90c 77f0a8ff ntdll!RtlDebugFreeHeap+0x2f
0818ea00 77eb2a32 ntdll!RtlpFreeHeap+0x5d
0818ea20 76bc14d1 ntdll!RtlFreeHeap+0x142
0818ea34 75694c39 kernel32!HeapFree+0x14
0818ea80 726f167d msvcr80!free+0xcd
0818ea8c 7270613d DllA!FreeData+0xd
[...]
0818fb20 77eb9d42 kernel32!BaseThreadInitThunk+0xe
0818fb60 77eb9d15 ntdll!__RtlUserThreadStart+0x70
0818fb78 00000000 ntdll!_RtlUserThreadStart+0x1b

In both examples above we see that 00720071 value was passed to free function (we also verify from the code using ub command that there was no parameter optimization (Volume 1, page 265)):

0:003> kv
ChildEBP RetAddr Args to Child
[...]
01b2ed98 726f167d 00720071 01b2edb0 7270613d msvcr80!free+0xcd
[...]

We recognize that value as Unicode as an example of a wild pointer (Volume 2, page 202) but parameters need not be pointers in a general case. We can also consider Invalid Handle (Volume 2, page 269) pattern as another specialization of Invalid Parameter pattern.

Hooking Level

In addition to hooked functions (Volume 1, page 469) pattern we should also pay attention a number of patched functions. Often value-added hooksware (Volume 2, page 63) has configuration options that fine-tune hooking behavior. For example, an application with the less number of patched functions behaved incorrectly and two process user dumps were saved from the working and non-working environment:

0:000> * problem behavior

0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:mssuser32.dll49E0380E9d000user32.dll
No range specified

Scanning section: .text
Size: 422527
Range to scan: 76e31000-76e9827f
76e3d6f8-76e3d6fc 5 bytes - user32!NtUserSetThreadDesktop
[ b8 30 12 00 00:e9 03 29 13 09 ]
76e3dc2a-76e3dc2e 5 bytes - user32!CreateWindowExA (+0x532)
[ 8b ff 55 8b ec:e9 d1 23 15 09 ]
76e3f8f8-76e3f8fc 5 bytes - user32!PostMessageA (+0x1cce)
[ 8b ff 55 8b ec:e9 03 07 fa 08 ]
76e41305-76e41309 5 bytes - user32!CreateWindowExW (+0x1a0d)
[ 8b ff 55 8b ec:e9 f6 ec 13 09 ]
76e435e3-76e435e7 5 bytes - user32!NtUserSetWindowPos (+0x22de)
[ b8 38 12 00 00:e9 18 ca 11 09 ]
76e48343-76e48347 5 bytes - user32!PeekMessageA (+0x4d60)
[ 8b ff 55 8b ec:e9 b8 7c fb 08 ]
76e48ab3-76e48ab7 5 bytes - user32!GetMessageA (+0x770)
[ 8b ff 55 8b ec:e9 48 75 fd 08 ]
76e4a175-76e4a179 5 bytes - user32!PostMessageW (+0x16c2)
[ 8b ff 55 8b ec:e9 86 5e f8 08 ]
76e4fef7-76e4fefb 5 bytes - user32!GetMessageW (+0x5d82)
[ 8b ff 55 8b ec:e9 04 01 fc 08 ]
76e5045a-76e5045e 5 bytes - user32!PeekMessageW (+0x563)
[ 8b ff 55 8b ec:e9 a1 fb f9 08 ]
76e8d37d-76e8d381 5 bytes - user32!MessageBoxTimeoutW (+0x3cf23)
[ 8b ff 55 8b ec:e9 7e 2c fd 08 ]
76e8d4d9-76e8d4dd 5 bytes - user32!MessageBoxIndirectA (+0x15c)
[ 8b ff 55 8b ec:e9 22 2b ff 08 ]
76e8d5d3-76e8d5d7 5 bytes - user32!MessageBoxIndirectW (+0xfa)
[ 8b ff 55 8b ec:e9 28 2a fe 08 ]
76e8d65d-76e8d661 5 bytes - user32!MessageBoxExW (+0x8a)
[ 8b ff 55 8b ec:e9 9e 29 00 09 ]

Total bytes compared: 422527(100%)
Number of errors: 70
70 errors : !user32 (76e3d6f8-76e8d661)

0:000> u EnumDisplayDevicesW
user32!EnumDisplayDevicesW:
76e3ba5b 8bff          mov edi,edi
76e3ba5d 55            push ebp
76e3ba5e 8bec          mov ebp,esp
76e3ba60 81ec54030000  sub esp,354h
76e3ba66 a1c090e976    mov eax,dword ptr [user32!__security_cookie
(76e990c0)]
76e3ba6b 33c5          xor eax,ebp
76e3ba6d 8945fc        mov dword ptr [ebp-4],eax
76e3ba70 53            push ebx

0:000> * expected behavior

0:000> !chkimg -lo 50 -d !user32 -v
Searching for module with expression: !user32
Will apply relocation fixups to file used for comparison
Will ignore NOP/LOCK errors
Will ignore patched instructions
Image specific ignores will be applied
Comparison image path: c:mssuser32.dll49E0380E9d000user32.dll
No range specified

Scanning section: .text
Size: 422527
Range to scan: 76e31000-76e9827f
76e39c11-76e39c15 5 bytes - user32!MonitorFromPoint
[ 6a 08 68 50 9c:e9 ea 63 10 09 ]
76e3b8ea-76e3b8ee 5 bytes - user32!GetMonitorInfoA (+0x1cd9)
[ 8b ff 55 8b ec:e9 11 47 12 09 ]
76e3ba5b-76e3ba5f 5 bytes - user32!EnumDisplayDevicesW (+0×171)
[ 8b ff 55 8b ec:e9 a0 45 0b 09 ]
76e3d6f8-76e3d6fa 3 bytes - user32!NtUserSetThreadDesktop (+0×1c9d)
[ b8 30 12:e9 03 29 ]
76e3d6fc - user32!NtUserSetThreadDesktop+4 (+0×04)
[ 00:09 ]
76e3dc2a-76e3dc2e 5 bytes - user32!CreateWindowExA (+0×52e)
[ 8b ff 55 8b ec:e9 d1 23 15 09 ]
76e3e7cd-76e3e7d1 5 bytes - user32!SetWindowLongA (+0xba3)
[ 8b ff 55 8b ec:e9 2e 18 03 09 ]
76e3f8f8-76e3f8fc 5 bytes - user32!PostMessageA (+0×112b)
[ 8b ff 55 8b ec:e9 03 07 e7 08 ]
76e41305-76e41309 5 bytes - user32!CreateWindowExW (+0×1a0d)
[ 8b ff 55 8b ec:e9 f6 ec 13 09 ]
76e413b4-76e413b8 5 bytes - user32!SetWindowLongW (+0xaf)
[ 8b ff 55 8b ec:e9 47 ec 03 09 ]
76e41709-76e4170d 5 bytes - user32!MonitorFromRect (+0×355)
[ 6a 08 68 48 17:e9 f2 e8 0e 09 ]
76e435e3-76e435e7 5 bytes - user32!NtUserSetWindowPos (+0×1eda)
[ b8 38 12 00 00:e9 18 ca fe 08 ]
76e440c5-76e440c9 5 bytes - user32!EnumDisplaySettingsExW (+0xae2)
[ 8b ff 55 8b ec:e9 36 bf 06 09 ]
76e441a1-76e441a5 5 bytes - user32!EnumDisplaySettingsW (+0xdc)
[ 8b ff 55 8b ec:e9 5a be 08 09 ]
76e46d4a-76e46d4e 5 bytes - user32!EnumDisplayDevicesA (+0×2ba9)
[ 8b ff 55 8b ec:e9 b1 92 0b 09 ]
76e46fe6-76e46fea 5 bytes - user32!EnumDisplaySettingsA (+0×29c)
[ 8b ff 55 8b ec:e9 15 90 09 09 ]
76e47010-76e47014 5 bytes - user32!EnumDisplaySettingsExA (+0×2a)
[ 8b ff 55 8b ec:e9 eb 8f 07 09 ]
76e47d12-76e47d16 5 bytes - user32!GetMonitorInfoW (+0xd02)
[ 8b ff 55 8b ec:e9 e9 82 10 09 ]
76e48343-76e48347 5 bytes - user32!PeekMessageA (+0×631)
[ 8b ff 55 8b ec:e9 b8 7c e8 08 ]
76e4844c-76e48450 5 bytes - user32!NtUserEnumDisplayMonitors (+0×109)
[ b8 81 11 00 00:e9 af 7b 0c 09 ]
76e488d4-76e488d8 5 bytes - user32!MonitorFromWindow (+0×488)
[ 6a 08 68 28 89:e9 27 77 0d 09 ]
76e48ab3-76e48ab7 5 bytes - user32!GetMessageA (+0×1df)
[ 8b ff 55 8b ec:e9 48 75 ea 08 ]
76e49994-76e49998 5 bytes - user32!GetWindowLongA (+0xee1)
[ 6a 08 68 d0 99:e9 67 66 00 09 ]
76e49af1-76e49af5 5 bytes - user32!GetSystemMetrics (+0×15d)
[ 6a 0c 68 58 9b:e9 0a 65 12 09 ]
76e4a175-76e4a179 5 bytes - user32!PostMessageW (+0×684)
[ 8b ff 55 8b ec:e9 86 5e e5 08 ]
76e4f8bf-76e4f8c3 5 bytes - user32!GetWindowLongW (+0×574a)
[ 6a 08 68 00 f9:e9 3c 07 01 09 ]
76e4fef7-76e4fefb 5 bytes - user32!GetMessageW (+0×638)
[ 8b ff 55 8b ec:e9 04 01 e9 08 ]
76e5045a-76e5045e 5 bytes - user32!PeekMessageW (+0×563)
[ 8b ff 55 8b ec:e9 a1 fb e6 08 ]
76e8d37d-76e8d381 5 bytes - user32!MessageBoxTimeoutW (+0×3cf23)
[ 8b ff 55 8b ec:e9 7e 2c ea 08 ]
76e8d4d9-76e8d4dd 5 bytes - user32!MessageBoxIndirectA (+0×15c)
[ 8b ff 55 8b ec:e9 22 2b ec 08 ]
76e8d5d3-76e8d5d7 5 bytes - user32!MessageBoxIndirectW (+0xfa)
[ 8b ff 55 8b ec:e9 28 2a eb 08 ]
76e8d65d-76e8d661 5 bytes - user32!MessageBoxExW (+0×8a)
[ 8b ff 55 8b ec:e9 9e 29 ed 08 ]
Total bytes compared: 422527(100%)
Number of errors: 154
154 errors : !user32 (76e39c11-76e8d661)
0:000> u EnumDisplayDevicesW
user32!EnumDisplayDevicesW:
76e3ba5b e9a0450b09    jmp 7fef0000
76e3ba60 81ec54030000  sub esp,354h
76e3ba66 a1c090e976    mov eax,dword ptr [user32!__security_cookie
(76e990c0)]
76e3ba6b 33c5          xor eax,ebp
76e3ba6d 8945fc        mov dword ptr [ebp-4],eax
76e3ba70 53 push       ebx
76e3ba71 56 push       esi
76e3ba72 8b7510        mov esi,dword ptr [ebp+10h]

Embedded Comments

Such comments in dump files are useful to record external information like the reason for saving a memory dump, a tool used to do that, and some pre-analysis and monitoring data that might help or guide in the future analysis. Comments are not widely used but some examples include Manual Process Dump (Volume 1, page 487), False Positive Dump (Volume 1, page 259) patterns, and process and thread CPU consumption comments in dump files saved by Sysinternals ProcDump⁶ tool. Such comments may not be necessarily saved by IDebugClient2 :: WriteDumpFile2 function but any buffer saved in memory that is accessible later from a dump file will do as can be easily demonstrated by the Citrix SystemDump (Volume 1, page 646) tool that allows embedding comments of arbitrary complexity.

Well-Tested Module

We introduce this pattern by analogy with Well-Tested Function (Volume 4, page 144). This is a module we usually skip when analyzing a stack trace because we suspect it the least. WinDbg can also be customized to skip such modules for the default analysis command as shown in component identification minidump analysis example (Vplume 1, page 46).

String Parameter

This analysis pattern is frequently useful when found on a function call stack. The trivial case is when a function parameter is a pointer to an ASCII or a Unicode string (da and du WinDbg commands). More interesting case is when we have a function that takes pointers to a structure that has string fields (dpa and dpu commands), for example:

0:018> kv 100
ChildEBP RetAddr Args to Child
00de8c7c 7739bf53 7739610a 07750056 00000000 ntdll!KiFastSystemCallRet
00de8cb4 7738965e 00080126 07750056 00000001 user32!NtUserWaitMessage+0xc
00de8cdc 7739f762 77380000 0012b238 07750056 user32!InternalDialogBox+0xd0
00de8f9c 7739f047 00de90f8 00000000 ffffffff user32!SoftModalMessageBox+0x94b
00de90ec 7739eec9 00de90f8 00000028 07750056 user32!MessageBoxWorker+0x2ba
00de9144 773d7d0d 07750056 0015cd68 00132a60 user32!MessageBoxTimeoutW+0x7a
00de9178 773c42c8 07750056 00de923f 00de91ec user32!MessageBoxTimeoutA+0x9c
00de9198 773c42a4 07750056 00de923f 00de91ec user32!MessageBoxExA+0x1b
00de91b4 6dfcf8c2 07750056 00de923f 00de91ec user32!MessageBoxA+0×45
00de99f0 6dfcfad2 00de9285 00de9a1c 77bc6cd5 compstui!FilterException+0×174
00dead94 7739b6e3 0038010e 00000110 00000000 compstui!CPSUIPageDlgProc+0xf3
00deadc0 77395f82 6dfcf9df 0038010e 00000110 user32!InternalCallWinProc+0×28
00deae3c 77395e22 0015d384 6dfcf9df 0038010e user32!UserCallDlgProcCheckWow+0×147
00deae84 7738aaa4 00000000 00000110 00000000 user32!DefDlgProcWorker+0xa8
00deaeb4 77388c01 004673d0 00461130 00000000 user32!SendMessageWorker+0×43e
00deaf6c 77387910 6dfc0000 004673d0 00000404 user32!InternalCreateDialog+0×9cf
00deaf90 7739fb5b 6dfc0000 001621d0 07750056 user32!CreateDialogIndirectParamAorW+0×33
00deafb0 774279a5 6dfc0000 001621d0 07750056 user32!CreateDialogIndirectParamW+0×1b
00deb000 77427abc 02192c78 000ddd08 07750056 comctl32!_CreatePageDialog+0×79
00deb028 77429d12 02192c78 6dff5c30 07750056 comctl32!_CreatePage+0xb1
00deb244 7742b8b6 02192c78 00000001 00290110 comctl32!PageChange+0xcc
00deb604 7742c446 07750056 02192c78 00deb6ec comctl32!InitPropSheetDlg+0xbb8
00deb674 7739b6e3 07750056 00000110 00290110 comctl32!PropSheetDlgProc+0×4cb
00deb6a0 77395f82 7742bf7b 07750056 00000110 user32!InternalCallWinProc+0×28
00deb71c 77395e22 0008c33c 7742bf7b 07750056 user32!UserCallDlgProcCheckWow+0×147
00deb764 7738aaa4 00000000 00000110 00290110 user32!DefDlgProcWorker+0xa8
00deb794 77388c01 004652e0 00461130 00290110 user32!SendMessageWorker+0×43e
00deb84c 77387910 77420000 004652e0 00000100 user32!InternalCreateDialog+0×9cf
00deb870 7739fb5b 77420000 02184be8 00000000 user32!CreateDialogIndirectParamAorW+0×33
00deb890 774ab1c5 77420000 02184be8 00000000 user32!CreateDialogIndirectParamW+0×1b
00deb8d8 7742ca78 77420000 02184be8 00000000 comctl32!SHFusionCreateDialogIndirectParam+0×36
00deb93c 7742ccea 00000000 000000a0 00000000 comctl32!_RealPropertySheet+0×242
00deb954 7742cd05 00deb9b4 00000000 00deb99c comctl32!_PropertySheet+0×146
00deb964 6dfd1178 00deb9b4 000000a0 00deba30 comctl32!PropertySheetW+0xf
00deb99c 6dfcf49b 00deb9b4 0256b3f8 0013fbe0 compstui!PropertySheetW+0×4b
00deba14 6dfd0718 00000000 00134da4 00debae8 compstui!DoComPropSheet+0×2ef
00deba44 6dfd0799 00000000 7307c8da 00debad0 compstui!DoCommonPropertySheetUI+0xe9
00deba5c 730801c5 00000000 7307c8da 00debad0 compstui!CommonPropertySheetUIW+0×17
00debaa4 73080f5d 00000000 7307c8da 00debad0 winspool!CallCommonPropertySheetUI+0×43
00debeec 4f49cdfe 00000000 0218bd84 02277fe8 winspool!PrinterPropertiesNative+0×10c
WARNING: Stack unwind information not available. Following frames may be wrong.
00debf2c 4f4950a5 00deea08 00000002 02277fe8 PrintDriverA!DllGetClassObject+0xdb7e
00deee18 4f4904fb 00ca6ee0 00000003 00000001 PrintDriverA!DllGetClassObject+0×5e25
00deee30 18f60282 02277fe8 00ca6ee0 00000003 PrintDriverA!DllGetClassObject+0×127b
00deee58 18f5abce 001042e4 00ca6ee0 00000003 ps5ui!HComOEMPrinterEvent+0×33
00deee9c 7308218c 00ca6ee0 00000003 00000001 ps5ui!DrvPrinterEvent+0×22e
00deeee8 761543c8 00ca6ee0 00000003 00000001 winspool!SpoolerPrinterEventNative+0×57
00deef04 761560d2 00ca6ee0 00000003 00000000 localspl!SplDriverEvent+0×21
00deef28 761447f9 00cb2160 00000003 00000000 localspl!PrinterDriverEvent+0×46
00def3f0 76144b12 00000000 00000002 00d12020 localspl!SplAddPrinter+0×5f3
00def41c 74070193 00000000 00000002 00d12020 localspl!LocalAddPrinterEx+0×2e
00def86c 7407025c 00000000 00000002 00d12020 spoolss!AddPrinterExW+0×151
00def888 01007a93 00000000 00000002 00d12020 spoolss!AddPrinterW+0×17
00def8a4 01006772 00000000 00ce74b0 021b6278 spoolsv!YAddPrinter+0×75
00def8c8 77c80355 00000000 00ce74b0 021b6278 spoolsv!RpcAddPrinter+0×37
00def8f0 77ce43e1 0100673b 00defae0 00000005 rpcrt4!Invoke+0×30
00defcf8 77ce45c4 00000000 00000000 000e8584 rpcrt4!NdrStubCall2+0×299
00defd14 77c8013a 000e8584 000d63d8 000e8584 rpcrt4!NdrServerCall2+0×19
00defd48 77c805ef 01002c57 000e8584 00defdec rpcrt4!DispatchToStubInCNoAvrf+0×38
00defd9c 77c80515 00000005 00000000 0100d228 rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0×11f
00defdc0 77c8139e 000e8584 00000000 0100d228 rpcrt4!RPC_INTERFACE::DispatchToStub+0xa3
00defdfc 77c814b2 000e1c48 000d85b8 02154180 rpcrt4!LRPC_SCALL::DealWithRequestMessage+0×42c
00defe20 77c88848 000d85f0 00defe38 000e1c48 rpcrt4!LRPC_ADDRESS::DealWithLRPCRequest+0×127
00deff84 77c88962 00deffac 77c888fd 000d85b8 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0×430
00deff8c 77c888fd 000d85b8 00000000 00000000 rpcrt4!RecvLotsaCallsWrapper+0xd
00deffac 77c7b293 0008b038 00deffec 77e6482f rpcrt4!BaseCachedThreadRoutine+0×9d
00deffb8 77e6482f 000bdba8 00000000 00000000 rpcrt4!ThreadStartRoutine+0×1b
00deffec 00000000 77c7b278 000bdba8 00000000 kernel32!BaseThreadStart+0×34


0:018> da 00de923f
00de923f “Function address 0×77481456 caus”
00de925f “ed a protection fault. (exceptio”
00de927f “n code 0xc0000005).The applicati”
00de929f “on property sheet page(s) may no”
00de92bf “t function properly.”

0:018> dpu 00d12020
00d12020 00000000
00d12024 021b6088 “Printer A User B Server C”
00d12028 00000000
00d1202c 021b6124 “Remote Printer Address for User C”
00d12030 021b6190 “Printer Name and Family”
00d12034 021b61c4 “Printer Client Name”
00d12038 021b6228 “Printer Location”
00d1203c 00000000
00d12040 00000000
00d12044 021b6264 “Printer Module Name”
00d12048 00000000
00d1204c 00000000
00d12050 021b628c
00d12054 00008841
00d12058 00000000
00d1205c 00000000
00d12060 00000000
00d12064 00000000
00d12068 00000000
00d1206c 00000000
00d12070 00000000
00d12074 00000000
00d12078 00000000
00d1207c 00000000
00d12080 00000000
00d12084 00000000
00d12088 00000000
00d1208c 00000000
00d12090 00000000
00d12094 00000000
00d12098 00000000
00d1209c 00000000

Environment Hint

This pattern is useful for inconsistent dumps or incomplete supporting information. It provides environment variable information for troubleshooting suggestions such as product elimination for testing purposes and / or necessary upgrade, for example:

0: kd> !peb
PEB at 7ffd7000
InheritedAddressSpace:           No
ReadImageFileExecOptions:        Yes
BeingDebugged:                   No
ImageBaseAddress:                01000000
Ldr 7c8897e0
Ldr.Initialized:                 Yes
Ldr.InInitializationOrderModuleList:     00081f18 . 000f9e88
Ldr.InLoadOrderModuleList:               00081eb0 . 000f9e78
Ldr.InMemoryOrderModuleList:             00081eb8 . 000f9e80
Base       TimeStamp                       Module
1000000  45d6a03c Feb 17 06:27:08 2007 C:WINNTsystem32svchost.exe
7c800000 49900d60 Feb 09 11:02:56 2009 C:WINNTsystem32
tdll.dll
[...]
SubSystemData:                   00000000
ProcessHeap:                     00080000
ProcessParameters:               00020000
WindowTitle: ‘C:WINNTsystem32svchost.exe’
ImageFile: ‘C:WINNTsystem32svchost.exe’
CommandLine: ‘C:WINNTsystem32svchost.exe -k rpcss’
DllPath: [...]
Environment:                     00010000
ALLUSERSPROFILE=C:Documents and SettingsAll Users
[...]
PROTECTIONDIR=C:Documents and SettingsAll UsersApplication
Data3rdPartyAntivirusProtection
[...]
Path= [...]

Dual Stack Trace

This is the kernel mode and space counterpart to a user mode and space stack trace and vice versa, for example:

25 Id: e8c.f20 Suspend: 1 Teb: 7ff9c000 Unfrozen
ChildEBP RetAddr
086acac4 7c90df5a ntdll!KiFastSystemCallRet
086acac8 7c8025db ntdll!ZwWaitForSingleObject+0xc
086acb2c 7c802542 kernel32!WaitForSingleObjectEx+0xa8
086acb40 00fbba3a kernel32!WaitForSingleObject+0×12
WARNING: Stack unwind information not available. Following frames may be
wrong.
086ad3c8 00fbc139 ModuleA!DllCanUnloadNow+0×638b4a
086adc38 00faba75 ModuleA!DllCanUnloadNow+0×639249
086ae4c8 00fa0da8 ModuleA!DllCanUnloadNow+0×629b25
086aed60 00a45331 ModuleA!DllCanUnloadNow+0×61ee48
086af6c4 00a44b10 ModuleA!DllCanUnloadNow+0xc6de1
086affb4 7c80b729 ModuleA!DllCanUnloadNow+0xc65c0
086affec 00000000 kernel32!BaseThreadStart+0×37

0: kd> !thread 88ec9020 1f
THREAD 88ec9020 Cid 17a0.2034 Teb: 7ffad000 Win32Thread: bc28c6e8 WAIT:
(Unknown) UserMode Non-Alertable
89095f48 Semaphore Limit 0x10000
IRP List:
    89a5a370: (0006,0094) Flags: 00000900 Mdl: 00000000
Not impersonating
DeviceMap              d6c30c48
Owning Process         88fffd88        Image: iexplore.exe
Attached Process       N/A             Image: N/A
Wait Start TickCount   5632994         Ticks: 2980 (0:00:00:46.562)
Context Switch Count   2269            LargeStack
UserTime               00:00:00.000
KernelTime             00:00:00.000
Win32 Start Address 0x00a262d0
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init b204c000 Current b204bc60 Base b204c000 Limit b2048000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
b204bc78 80833ec5 nt!KiSwapContext+0×26
b204bca4 80829c14 nt!KiSwapThread+0×2e5
b204bcec 8093b174 nt!KeWaitForSingleObject+0×346
b204bd50 8088b41c nt!NtWaitForSingleObject+0×9a
b204bd50 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ b204bd64)
058fcabc 7c827d29 ntdll!KiFastSystemCallRet
058fcac0 77e61d1e ntdll!ZwWaitForSingleObject+0xc
058fcb30 77e61c8d kernel32!WaitForSingleObjectEx+0xac
058fcb44 00f98b4a kernel32!WaitForSingleObject+0×12
WARNING: Stack unwind information not available. Following frames may be
wrong.
058fd3cc 00f99249 ModuleA+0×638b4a
058fdc3c 00f89b25 ModuleA+0×639249
058fe4cc 00f7ee48 ModuleA+0×629b25
058fed64 00a26de1 ModuleA+0×61ee48
058ff6c8 00a265c0 ModuleA+0xc6de1
058fffb8 77e6482f ModuleA+0xc65c0
058fffec 00000000 kernel32!BaseThreadStart+0×34

This pattern is helpful when we have both process user space memory dumps and kernel and complete memory dumps and want to match stack traces of interest between them. See also patterns Stack Trace (Volume 1, page 395) and Stack Trace Collection (Volume 1, page 409).

Blocking Module

We would like to add this pattern in addition to Blocked Thread (Volume 2, page 184) and endpoint threads of Wait Chain (Volume 1, page 482) patterns to account for modules calling waiting or delaying functions, for example:

0:017> kL
ChildEBP RetAddr
02c34100 7c90df5a ntdll!KiFastSystemCallRet
02c34104 7c8025db ntdll!ZwWaitForSingleObject+0xc
02c34168 7c802542 kernel32!WaitForSingleObjectEx+0xa8
02c3417c 009f0ed9 kernel32!WaitForSingleObject+0×12
02c34a08 00bc2c9a ModuleA!DllCanUnloadNow+0×6db39
02c3526c 00bc2fa4 ModuleA!DllCanUnloadNow+0×23f8fa
02c35ae0 00f6413c ModuleA!DllCanUnloadNow+0×23fc04
02c363e8 00c761ab ModuleA!DllCanUnloadNow+0×5e0d9c
02c36c74 00c74daa ModuleA!DllCanUnloadNow+0×2f2e0b
02c374e4 3d1a9eb4 ModuleA!DllCanUnloadNow+0×2f1a0a
02c3753c 3d0ed032 mshtml!CView::SetObjectRectsHelper+0×98
02c37578 3cf7e43b mshtml!CView::EndDeferSetObjectRects+0×75
02c375bc 3cf2542d mshtml!CView::EnsureView+0×39f
02c375d8 3cf4072c mshtml!CElement::EnsureRecalcNotify+0×17c
02c37614 3cf406ce mshtml!CElement::get_clientHeight_Logical+0×54
02c37628 3d0822a1 mshtml!CElement::get_clientHeight+0×27
02c37648 3cf8ad53 mshtml!G_LONG+0×7b
02c376bc 3cf96e21 mshtml!CBase::ContextInvokeEx+0×5d1
02c3770c 3cfa2baf mshtml!CElement::ContextInvokeEx+0×9d
02c37738 3cf8a751 mshtml!CElement::VersionedInvokeEx+0×2d
[...]

Wait Chain (Window Messaging)

This is another variant of the general Wait Chain (Volume 1, page 482) pattern where blocked threads are waiting for synchronous window message calls (sent messages). For example, here three threads from different processes are blocked in such a chain (hWnd parameters for SendMessage calls and associated window procedures are marked with bold italics, bold and underline):

0:000> ~*kbL

. 0 Id: 116c.1174 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr Args to Child
0034f83c 76261c01 000a0e54 00000111 00000068 USER32!NtUserMessageCall+0x15
0034f87c 7625cd81 011114d0 00000000 00d41190 USER32!SendMessageWorker+0x5e9
0034f8a0 00fa1256 000a0e54 00000111 00000068 USER32!SendMessageW+0×7f
0034f90c 76256238 00040eb0 00000111 00000068 WCM_A!WndProc+0xc6
0034f938 762568ea 00fa1190 00040eb0 00000111 USER32!InternalCallWinProc+0×23
0034f9b0 76257d31 00000000 00fa1190 00040eb0 USER32!UserCallWinProcCheckWow+0×109
0034fa10 76257dfa 00fa1190 00000000 76257d79 USER32!DispatchMessageWorker+0×3bc
0034fa20 00fa10d3 0034fa3c 0034fae8 00000000 USER32!DispatchMessageW+0xf
0034fa54 00fa14b6 00fa0000 00000000 00571bee WCM_A!wWinMain+0xd3
0034fae8 76493677 7efde000 0034fb34 77399d72 WCM_A!__tmainCRTStartup+0×150
0034faf4 77399d72 7efde000 72afcb2e 00000000 kernel32!BaseThreadInitThunk+0xe
0034fb34 77399d45 00fa1625 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
0034fb4c 00000000 00fa1625 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

0:000> ~*kbL

. 0 Id: 10dc.e14 Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr Args to Child
0017f7c4 76261c01 000c0ea4 00000111 00000068 USER32!NtUserMessageCall+0x15
0017f804 7625cd81 00ec3ec0 00000000 012e1190 USER32!SendMessageWorker+0x5e9
0017f828 00d41256 000c0ea4 00000111 00000068 USER32!SendMessageW+0×7f
0017f890 76256238 000a0e54 00000111 00000068 WCM_B!WndProc+0xc6
0017f8bc 762568ea 00d41190 000a0e54 00000111 USER32!InternalCallWinProc+0×23
0017f934 76257177 00000000 00d41190 000a0e54 USER32!UserCallWinProcCheckWow+0×109
0017f990 762572f1 00eb14d0 00000000 00000111 USER32!DispatchClientMessage+0xe0
0017f9cc 773700e6 0017f9e4 00000000 0017fae4 USER32!__fnDWORD+0×2b
0017f9e0 00eb14d0 00000000 00000111 00000068 ntdll!KiUserCallbackDispatcher+0×2e
WARNING: Frame IP not in any known module. Following frames may be wrong.
0017fa20 00d410e0 0017fa48 00000000 00000000 0xeb14d0
0017fa60 00d414b6 00d40000 00000000 00601bee WCM_B!wWinMain+0xe0
0017faf4 76493677 7efde000 0017fb40 77399d72 WCM_B!__tmainCRTStartup+0×150
0017fb00 77399d72 7efde000 728cf6de 00000000 kernel32!BaseThreadInitThunk+0xe
0017fb40 77399d45 00d41625 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
0017fb58 00000000 00d41625 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b
0:000> ~*kbL

. 0 Id: e68.fbc Suspend: 1 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr Args to Child
0017f4c8 76272674 000c0ea4 00000000 00000000 USER32!NtUserWaitMessage+0x15
0017f504 7627288a 00070ee6 000c0ea4 00000000 USER32!DialogBox2+0x222
0017f530 762727b8 012e0000 012efc54 000c0ea4 USER32!InternalDialogBox+0xe5
0017f550 76272aa1 012e0000 012efc54 000c0ea4 USER32!DialogBoxIndirectParamAorW+0x37
0017f574 012e124d 012e0000 00000067 000c0ea4 USER32!DialogBoxParamW+0x3f
0017f5e4 76256238 000c0ea4 00000111 00000068 WCM_C!WndProc+0xbd
0017f610 762568ea 012e1190 000c0ea4 00000111 USER32!InternalCallWinProc+0×23
0017f688 76257177 00000000 012e1190 000c0ea4 USER32!UserCallWinProcCheckWow+0×109
0017f6e4 762572f1 01463ec0 00000000 00000111 USER32!DispatchClientMessage+0xe0
0017f720 773700e6 0017f738 00000000 0017f838 USER32!__fnDWORD+0×2b
0017f734 01463ec0 00000000 00000111 00000068 ntdll!KiUserCallbackDispatcher+0×2e
WARNING: Frame IP not in any known module. Following frames may be wrong.
0017f774 012e10e0 0017f79c 00000000 00000000 0×1463ec0
0017f7b4 012e1496 012e0000 00000000 00471bee WCM_C!wWinMain+0xe0
0017f848 76493677 7efde000 0017f894 77399d72 WCM_C!__tmainCRTStartup+0×150
0017f854 77399d72 7efde000 728ca9cf 00000000 kernel32!BaseThreadInitThunk+0xe
0017f894 77399d45 012e1605 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
0017f8ac 00000000 012e1605 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

Send message calls can also be directed to the same thread:

0: kd> kv 100
ChildEBP RetAddr Args to Child
aa839180 80833ed1 8c446b60 8c446c08 00000002 nt!KiSwapContext+0x26
aa8391ac 80829c14 8c446d4c 8c446d20 8c446b60 nt!KiSwapThread+0x2e5
aa8391f4 80921102 8c446d4c 00000011 8c4a8c01 nt!KeWaitForSingleObject+0x346
aa8392b0 8088b41c 000006a8 00172e58 00172e58 nt!NtRequestWaitReplyPort+0x776
aa8392b0 7c82860c 000006a8 00172e58 00172e58 nt!KiFastCallEntry+0xfc
0012f194 7c827899 77c80a6e 000006a8 00172e58 ntdll!KiFastSystemCallRet
0012f198 77c80a6e 000006a8 00172e58 00172e58 ntdll!ZwRequestWaitReplyPort+0xc
0012f1e4 77c7fcf0 0012f220 0012f204 77c80673 RPCRT4!LRPC_CCALL::SendReceive+0x230
0012f1f0 77c80673 0012f220 771f2918 0012f60c RPCRT4!I_RpcSendReceive+0x24
0012f204 77ce315a 0012f24c 00172ea8 77e63e5f RPCRT4!NdrSendReceive+0x2b
0012f5ec 771f4fbd 771f2918 771f1858 0012f60c RPCRT4!NdrClientCall2+0x22e
[...]
0012f698 7739b6e3 0004001a 00000016 00000001 ApplicationA!WndProc+0xcc
0012f6c4 7739b874 00407440 0004001a 00000016 USER32!InternalCallWinProc+0×28
0012f73c 7739c8b8 00000000 00407440 0004001a USER32!UserCallWinProcCheckWow+0×151
0012f798 7739c9c6 00607890 00000016 00000001 USER32!DispatchClientMessage+0xd9
0012f7c0 7c828556 0012f7d8 00000018 0012f894 USER32!__fnDWORD+0×24
0012f7c0 80831378 0012f7d8 00000018 0012f894 ntdll!KiUserCallbackDispatcher+0×2e
aa83957c 8091fbbb aa839634 aa839638 aa839608 nt!KiCallUserMode+0×4
aa8395d4 bf8a2492 00000002 aa839618 00000018 nt!KeUserModeCallback+0×8f
aa839658 bf8a229d be487890 00000016 00000001 win32k!SfnDWORD+0xb4
aa8396a0 bf8a1249 02487890 00000016 00000001 win32k!xxxSendMessageToClient+0×176
aa8396ec bf8a115e be487890 00000016 00000001 win32k!xxxSendMessageTimeout+0×1a6
aa839710 bf926e0d be487890 00000016 00000001 win32k!xxxSendMessage+0×1b
aa83974c bf926eb5 bc18cbc8 00000016 00000001 win32k!xxxClientShutdown2+0×87
aa839768 bf8ad9fa be487890 80000009 0000029e win32k!xxxClientShutdown+0×47
aa8397c4 bf8845d4 be487890 0000003b 80000009 win32k!xxxRealDefWindowProc+0×364
aa8397dc bf884604 be487890 0000003b 80000009 win32k!xxxWrapRealDefWindowProc+0×16
aa8397f8 bf8c1259 be487890 0000003b 80000009 win32k!NtUserfnNCDESTROY+0×27
aa839830 8088b41c 0004001a 0000003b 80000009 win32k!NtUserMessageCall+0xc0
aa839830 7c82860c 0004001a 0000003b 80000009 nt!KiFastCallEntry+0xfc (TrapFrame @
aa839854)
0012f7c0 7c828556 0012f7d8 00000018 0012f894 ntdll!KiFastSystemCallRet
0012f7c0 80831378 0012f7d8 00000018 0012f894 ntdll!KiUserCallbackDispatcher+0×2e
aa839b08 8091fbbb aa839bc0 aa839bc4 aa839b94 nt!KiCallUserMode+0×4
aa839b60 bf8a2492 00000002 aa839ba4 00000018 nt!KeUserModeCallback+0×8f
aa839be4 bf8a229d be487890 0000003b 80000009 win32k!SfnDWORD+0xb4
aa839c2c bf8c3f77 02487890 0000003b 80000009 win32k!xxxSendMessageToClient+0×176
aa839c9c bf89b88e bc18e838 aa839d64 0012fa38 win32k!xxxReceiveMessage+0×2b5
aa839cec bf89d201 aa839d18 0004001a 00000000 win32k!xxxRealInternalGetMessage+0×2d7
aa839d4c 8088b41c 0012fa5c 0004001a 00000000 win32k!NtUserGetMessage+0×3f
aa839d4c 7c82860c 0012fa5c 0004001a 00000000 nt!KiFastCallEntry+0xfc (TrapFrame @
aa839d64)
0012f9f0 7c828556 0012fa08 00000018 0012ffb0 ntdll!KiFastSystemCallRet
0012fa1c 7739c811 7739c844 0012fa5c 0004001a ntdll!KiUserCallbackDispatcher+0×2e
0012fa3c 0040634e 0012fa5c 0004001a 00000000 USER32!NtUserGetMessage+0xc
0012ff18 00408d9d 00000032 00000000 00142546 ApplicationA!WinMain+0×80f
0012ffc0 77e6f22b 00000000 00000000 7ffdf000 ApplicationA!WinMainCRTStartup+0×185
0012fff0 00000000 00408c18 00000000 78746341 kernel32!BaseProcessStart+0×23

Blocked sent message calls can also be manifested in kernel space and mixed with patterns like Message Box (Volume 2, page 177) and Main Thread (), for example:

1: kd> k250
ChildEBP RetAddr
8d5d2808 82a7eb15 nt!KiSwapContext+0x26
8d5d2840 82a7d403 nt!KiSwapThread+0x266
8d5d2868 82a772cf nt!KiCommitThreadWait+0x1df
8d5d28e0 82550d75 nt!KeWaitForSingleObject+0x393
8d5d293c 82550e10 win32k!xxxRealSleepThread+0x1d7
8d5d2958 824ff4b0 win32k!xxxSleepThread+0x2d
8d5d29cc 825547e8 win32k!xxxInterSendMsgEx+0xb1c
8d5d2a1c 825546a4 win32k!xxxSendMessageTimeout+0x13b
8d5d2a44 82533843 win32k!xxxSendMessage+0×28
8d5d2b08 824fd865 win32k!xxxCalcValidRects+0xf7
8d5d2b64 82502c98 win32k!xxxEndDeferWindowPosEx+0×100
8d5d2b84 825170c9 win32k!xxxSetWindowPos+0xf6
8d5d2c08 82517701 win32k!xxxActivateThisWindow+0×2b1
8d5d2c38 82517537 win32k!xxxActivateWindow+0×144
8d5d2c4c 824fd9dd win32k!xxxSwpActivate+0×44
8d5d2ca4 82502c98 win32k!xxxEndDeferWindowPosEx+0×278
8d5d2cc4 824fff82 win32k!xxxSetWindowPos+0xf6
8d5d2d10 82a5342a win32k!NtUserSetWindowPos+0×140
8d5d2d10 76ee64f4 nt!KiFastCallEntry+0×12a (TrapFrame @ 8d5d2d34)
01e2cea0 7621358d ntdll!KiFastSystemCallRet
01e2cea4 6a8fa0eb USER32!NtUserSetWindowPos+0xc
01e2cf14 6a894b13 IEFRAME!SHToggleDialogExpando+0×15a
01e2cf28 6a894d5d IEFRAME!EleDlg::ToggleExpando+0×20
01e2d74c 6a895254 IEFRAME!EleDlg::OnInitDlg+0×229
01e2d7b8 762186ef IEFRAME!EleDlg::DlgProcEx+0×189
01e2d7e4 76209eb2 USER32!InternalCallWinProc+0×23
01e2d860 7620b98b USER32!UserCallDlgProcCheckWow+0xd6
01e2d8a8 7620bb7b USER32!DefDlgProcWorker+0xa8
01e2d8c4 762186ef USER32!DefDlgProcW+0×22
01e2d8f0 76218876 USER32!InternalCallWinProc+0×23
01e2d968 76217631 USER32!UserCallWinProcCheckWow+0×14b
01e2d9a8 76209b1d USER32!SendMessageWorker+0×4d0
01e2da64 76235500 USER32!InternalCreateDialog+0xb0d
01e2da94 76235553 USER32!InternalDialogBox+0xa7
01e2dab4 76235689 USER32!DialogBoxIndirectParamAorW+0×37
01e2dad8 6a5d4952 USER32!DialogBoxParamW+0×3f
01e2db00 6a5d5024 IEFRAME!Detour_DialogBoxParamW+0×47
01e2db24 6a8956df IEFRAME!SHFusionDialogBoxParam+0×32
01e2db58 6a8957bb IEFRAME!EleDlg::ShowDialog+0×398
01e2e638 6a8959d3 IEFRAME!ShowDialogBox+0xb6
01e2eb9c 6a9013ed IEFRAME!ShowElevationPrompt+0×1dd
01e2f010 7669fc8f IEFRAME!CIEUserBrokerObject::BrokerCoCreateInstance+0×202
01e2f040 76704c53 RPCRT4!Invoke+0×2a
01e2f448 76d9d936 RPCRT4!NdrStubCall2+0×2d6
01e2f490 76d9d9c6 ole32!CStdStubBuffer_Invoke+0xb6
01e2f4d8 76d9df1f ole32!SyncStubInvoke+0×3c
01e2f524 76cb213c ole32!StubInvoke+0xb9
01e2f600 76cb2031 ole32!CCtxComChnl::ContextInvoke+0xfa
01e2f61c 76d9a754 ole32!MTAInvoke+0×1a
01e2f64c 76d9dcbb ole32!AppInvoke+0xab
01e2f72c 76d9a773 ole32!ComInvokeWithLockAndIPID+0×372
01e2f778 7669f34a ole32!ThreadInvoke+0×302
01e2f7b4 7669f4da RPCRT4!DispatchToStubInCNoAvrf+0×4a
01e2f80c 7669f3c6 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×16c
01e2f834 766a0cef RPCRT4!RPC_INTERFACE::DispatchToStub+0×8b
01e2f86c 7669f882 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xb2
01e2f8b8 7669f7a4 RPCRT4!LRPC_SCALL::DispatchRequest+0×23b
01e2f8d8 7669f763 RPCRT4!LRPC_SCALL::QueueOrDispatchCall+0xbd
01e2f8f4 7669f5ff RPCRT4!LRPC_SCALL::HandleRequest+0×34f
01e2f928 7669f573 RPCRT4!LRPC_SASSOCIATION::HandleRequest+0×144
01e2f960 7669ee4f RPCRT4!LRPC_ADDRESS::HandleRequest+0xbd
01e2f9dc 7669ece7 RPCRT4!LRPC_ADDRESS::ProcessIO+0×50a
01e2f9e8 766a1357 RPCRT4!LrpcServerIoHandler+0×16
01e2f9f8 76ecd3a3 RPCRT4!LrpcIoComplete+0×16
01e2fa20 76ed0748 ntdll!TppAlpcpExecuteCallback+0×1c5
01e2fb88 76e11174 ntdll!TppWorkerThread+0×5a4
01e2fb94 76efb3f5 kernel32!BaseThreadInitThunk+0xe
01e2fbd4 76efb3c8 ntdll!__RtlUserThreadStart+0×70
01e2fbec 00000000 ntdll!_RtlUserThreadStart+0×1b

2: kd> !process 890ff430 1f
PROCESS 890ff430 SessionId: 1 Cid: 18a4 Peb: 7ffdc000 ParentCid: 1fdc
DirBase: 7fbf04e0 ObjectTable: da89fb80 HandleCount: 852.
Image: iexplore.exe

THREAD 89141db0 Cid 18a4.19c8 Teb: 7ffdf000 Win32Thread: bc373d18 WAIT: (Unknown)
UserMode Non-Alertable
8915b020 SynchronizationEvent
Not impersonating
DeviceMap da7f9680
Owning Process 890ff430 Image: iexplore.exe
Attached Process N/A Image: N/A
Wait Start TickCount 56879 Ticks: 1634 (0:00:00:25.531)
Context Switch Count 12410 NoStackSwap LargeStack
UserTime 00:00:00.078
KernelTime 00:00:01.234
Win32 Start Address iexplore!wWinMainCRTStartup (0x004031b9)
Start Address kernel32!BaseProcessStartThunk (0x77e617f8)
Stack Init b5672000 Current b56717c4 Base b5672000 Limit b566c000 Call 0
Priority 4 BasePriority 4 PriorityDecrement 0
ChildEBP RetAddr
b56717dc 80833ec5 nt!KiSwapContext+0x26
b5671808 80829c14 nt!KiSwapThread+0x2e5
b5671850 bf89ab73 nt!KeWaitForSingleObject+0x346
b56718ac bf8c4ba6 win32k!xxxSleepThread+0x1be
b5671948 bf8a13e0 win32k!xxxInterSendMsgEx+0x798
b5671994 bf8a132f win32k!xxxSendMessageTimeout+0x1f3
b56719b8 bf85ca01 win32k!xxxSendMessage+0×1b
b5671a7c bf85da04 win32k!xxxCalcValidRects+0xea
b5671ad8 bf85de2e win32k!xxxEndDeferWindowPosEx+0xf2
b5671af4 bf861cf2 win32k!xxxSetWindowPos+0xb1
b5671b3c bf882098 win32k!xxxProcessEventMessage+0×232
b5671c7c bf89b89e win32k!xxxScanSysQueue+0×21e
b5671ce4 bf89c529 win32k!xxxRealInternalGetMessage+0×2aa
b5671d48 8088b41c win32k!NtUserPeekMessage+0×42
b5671d48 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ b5671d64)
0012e6e8 7739bde5 ntdll!KiFastSystemCallRet
0012e714 7739be5e USER32!NtUserPeekMessage+0xc
0012e740 02935f8c USER32!PeekMessageW+0xab
0012e7b4 02936150 IEUI!DUserRegisterSuper+0×920
0012e7d4 40d2ee98 IEUI!PeekMessageExW+0×42
0012e818 40d2abf4 IEFRAME!CBrowserFrame::FrameMessagePump+0×23
0012e824 40d2bc63 IEFRAME!BrowserThreadProc+0×3f
0012e848 40d2bbb1 IEFRAME!BrowserNewThreadProc+0×7b
0012f8b8 40d2ba61 IEFRAME!SHOpenFolderWindow+0×188
0012fae8 00401484 IEFRAME!IEWinMain+0×2d9
0012ff2c 0040131f iexplore!wWinMain+0×2c6
0012ffc0 77e6f23b iexplore!_initterm_e+0×1b1
0012fff0 00000000 kernel32!BaseProcessStart+0×23

Wait Chain (Named Pipes)

This is a variant of the general Wait Chain pattern (Volume 1, page 482) where threads are waiting for named pipes. This is visible when we examine a pending IRP from a blocked thread:

THREAD 88ec9020 Cid 17a0.2034 Teb: 7ffad000 Win32Thread: bc28c6e8 WAIT:
(Unknown) UserMode Non-Alertable
89095f48 Semaphore Limit 0x10000
IRP List:
89a5a370: (0006,0094) Flags: 00000900 Mdl: 00000000
Not impersonating
DeviceMap d6c30c48
Owning Process 88fffd88      Image: ApplicationA.exe
Attached Process N/A         Image: N/A
Wait Start TickCount 5632994 Ticks: 2980 (0:00:00:46.562)
Context Switch Count 2269    LargeStack
UserTime       00:00:00.000
KernelTime     00:00:00.000
Win32 Start Address 0×00a262d0
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init b204c000 Current b204bc60 Base b204c000 Limit b2048000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr
b204bc78 80833ec5 nt!KiSwapContext+0×26
b204bca4 80829c14 nt!KiSwapThread+0×2e5
b204bcec 8093b174 nt!KeWaitForSingleObject+0×346
b204bd50 8088b41c nt!NtWaitForSingleObject+0×9a
b204bd50 7c82860c nt!KiFastCallEntry+0xfc (TrapFrame @ b204bd64)
058fcabc 7c827d29 ntdll!KiFastSystemCallRet
058fcac0 77e61d1e ntdll!ZwWaitForSingleObject+0xc
058fcb30 77e61c8d kernel32!WaitForSingleObjectEx+0xac
058fcb44 00f98b4a kernel32!WaitForSingleObject+0×12
[...]
058fffec 00000000 kernel32!BaseThreadStart+0×34

0: kd> !irp 89a5a370
Irp is active with 1 stacks 1 is current (= 0×89a5a3e0)
No Mdl: No System Buffer: Thread 88ec9020: Irp stack trace.
cmd     flg cl Device  File      Completion-Context
>[ 3, 0] 0 1   89ebee90 891d4f90 00000000-00000000 pending
FileSystemNpfs
Args: 00000100 00000000 00000000 00000000
0: kd> !fileobj 891d4f90

ServiceBSVC

Device Object: 0x89ebee90 FileSystemNpfs
Vpb is NULL

Flags: 0x40080
Named Pipe
Handle Created

FsContext: 0xdaeca230 FsContext2: 0x8949bdb0
Private Cache Map: 0x00000001
CurrentByteOffset: 0

The pipe chain can also extend from a thread to a thread and even cross machine boundary.

Top Module

This pattern is similar to Blocking Module pattern (page 54) with the difference in stack trace syntax only. A top module is any module we choose that is simply on top of a stack trace. Most of the time, it is likely to be a non-OS vendor module. Whether the stack trace is well-formed and semantically sound or incorrect (Volume 1, page 288) is irrelevant:

0:005> kL
ChildEBP RetAddr
01abc4d8 6efba23d ntdll!KiFastSystemCallRet
WARNING: Stack unwind information not available. Following frames may be
wrong.
01abc988 7c820833 ModuleB+0×2a23d
01abcbe4 7c8207f6 kernel32!GetVolumeNameForRoot+0×26
01abcc0c 7c82e6de kernel32!BasepGetVolumeNameForVolumeMountPoint+0×75
01abcc54 6efaf70b kernel32!GetVolumePathNameW+0×18a
01abccdc 6efbd1a6 ModuleB+0×1f70b
01abcce0 00000000 ModuleB+0×2d1a6

Here we can also check the validity of ModuleB code by backwards disassembly of 6efba23d return address (ub command). If we have an abridged dump (Volume 5, page 88) file (minidump) we need to specify the image file path in WinDbg.

Why a top module is important? In various troubleshooting scenarios we can check the module timestamp (Not My Version pattern, Volume 2, page 299) and other useful information (lmv and !lmi WinDbg commands). If we suspect the module as hooksware we can also recommend removing it or its software vendor package for testing purposes.

Dialog Box

Similar to Message Box (Volume 2, page 177) and String Parameter (page 49) patterns we also have Dialog Box pattern where we can see dialog window caption and contents when we examine function parameters. Although in the examples below we know the dialog purpose from the friendly call stack function names, for many 3rd-party applications we either don't have symbols or such helper functions but we want to know what was on the screen when screenshots were not collected.

The first 2 examples are from Notepad application and the 3rd is from IE:

0:000> kv
ChildEBP RetAddr Args to Child
0017f5c4 777b073f 777c3c9f 000d023c 00000001 ntdll!KiFastSystemCallRet
0017f5c8 777c3c9f 000d023c 00000001 00000000 user32!NtUserWaitMessage+0xc
0017f5fc 777c2dc0 00310778 000d023c 00000001 user32!DialogBox2+0x202
0017f624 777c2eec 76460000 02a6bc60 000d023c user32!InternalDialogBox+0xd0
0017f644 76489a65 76460000 02a6bc60 000d023c user32!DialogBoxIndirectParamAorW+0×37
0017f680 76489ccf 0017f68c 00000001 0017f6d4 comdlg32!ChooseFontX+0×1ba
0017f6bc 006741c7 0017f6d4 00000111 00000000 comdlg32!ChooseFontW+0×2e
0017f734 0067164a 000d023c 00000021 00000000 notepad!NPCommand+0×4c7
0017f758 777afd72 000d023c 00000111 00000021 notepad!NPWndProc+0×4cf
0017f784 777afe4a 0067146c 000d023c 00000111 user32!InternalCallWinProc+0×23
0017f7fc 777b018d 00000000 0067146c 000d023c user32!UserCallWinProcCheckWow+0×14b
0017f860 777b022b 0067146c 00000000 0017f8a4 user32!DispatchMessageWorker+0×322
0017f870 00671465 0017f888 00000000 0067a21c user32!DispatchMessageW+0xf
0017f8a4 0067195d 00670000 00000000 00231cfa notepad!WinMain+0xe3
0017f934 7652d0e9 7ffd9000 0017f980 77b019bb notepad!_initterm_e+0×1a1
0017f940 77b019bb 7ffd9000 78f7b908 00000000 kernel32!BaseThreadInitThunk+0xe
0017f980 77b0198e 006731ed 7ffd9000 00000000 ntdll!__RtlUserThreadStart+0×23
0017f998 00000000 006731ed 7ffd9000 00000000 ntdll!_RtlUserThreadStart+0×1b


0:000> dc 02a6bc60 l50
02a6bc60 80c800c4 00000000 000d0014 011f0036 ............6...
02a6bc70 000000c4 00460000 006e006f 00000074 ......F.o.n.t...
02a6bc80 004d0008 00200053 00680053 006c0065 ..M.S. .S.h.e.l.
02a6bc90 0020006c 006c0044 00000067 50020000 l. .D.l.g......P
02a6bca0 00000000 00070007 00090028 ffff0440 ........(...@...
02a6bcb0 00260082 006f0046 0074006e 0000003a ..&.F.o.n.t.:...
02a6bcc0 00000000 50210b51 00000000 00100007 ....Q.!P........
02a6bcd0 004c0062 ffff0470 00000085 00000000 b.L.p...........
02a6bce0 50020000 00000000 0007006e 0009002c ...P....n...,...
02a6bcf0 ffff0441 00460082 006e006f 00200074 A.....F.o.n.t. .
02a6bd00 00740073 00790026 0065006c 0000003a s.t.&.y.l.e.:...
02a6bd10 00000000 50210041 00000000 0010006e ....A.!P....n...
02a6bd20 004c004a ffff0471 00000085 00000000 J.L.q...........
02a6bd30 50020000 00000000 000700bd 0009001e ...P............
02a6bd40 ffff0442 00260082 00690053 0065007a B.....&.S.i.z.e.
02a6bd50 0000003a 00000000 50210b51 00000000 :.......Q.!P....
02a6bd60 001000be 004c0024 ffff0472 00000085 ....$.L.r.......
02a6bd70 00000000 50020007 00000000 00610007 .......P......a.
02a6bd80 00480062 ffff0430 00450080 00660066 b.H.0.....E.f.f.
02a6bd90 00630065 00730074 00000000 50010003 e.c.t.s........P

0:000> kv
ChildEBP RetAddr Args to Child
0017f5a8 777b073f 777c3c9f 000d023c 00000001 ntdll!KiFastSystemCallRet
0017f5ac 777c3c9f 000d023c 00000001 00000000 user32!NtUserWaitMessage+0xc
0017f5e0 777c2dc0 0044034a 000d023c 00000001 user32!DialogBox2+0x202
0017f608 777c2eec 768a0000 029030bc 000d023c user32!InternalDialogBox+0xd0
0017f628 777c10ef 768a0000 029030bc 000d023c user32!DialogBoxIndirectParamAorW+0×37
0017f64c 7695d877 768a0000 00003810 000d023c user32!DialogBoxParamW+0×3f
0017f670 76a744dc 768a0000 00003810 000d023c shell32!SHFusionDialogBoxParam+0×32
0017f6b0 00674416 000d023c 002530dc 00672fc4 shell32!ShellAboutW+0×4d
0017f734 0067164a 000d023c 00000041 00000000 notepad!NPCommand+0×718
0017f758 777afd72 000d023c 00000111 00000041 notepad!NPWndProc+0×4cf
0017f784 777afe4a 0067146c 000d023c 00000111 user32!InternalCallWinProc+0×23
0017f7fc 777b018d 00000000 0067146c 000d023c user32!UserCallWinProcCheckWow+0×14b
0017f860 777b022b 0067146c 00000000 0017f8a4 user32!DispatchMessageWorker+0×322
0017f870 00671465 0017f888 00000000 0067a21c user32!DispatchMessageW+0xf
0017f8a4 0067195d 00670000 00000000 00231cfa notepad!WinMain+0xe3
0017f934 7652d0e9 7ffd9000 0017f980 77b019bb notepad!_initterm_e+0×1a1
0017f940 77b019bb 7ffd9000 78f7b908 00000000 kernel32!BaseThreadInitThunk+0xe
0017f980 77b0198e 006731ed 7ffd9000 00000000 ntdll!__RtlUserThreadStart+0×23
0017f998 00000000 006731ed 7ffd9000 00000000 ntdll!_RtlUserThreadStart+0×1b

0:000> dc 029030bc l50
029030bc ffff0001 00000000 00000000 80c800cc ................
029030cc 0014000c 01130014 000000ee 00410000 ..............A.
029030dc 006f0062 00740075 00250020 00000073 b.o.u.t. .%.s...
029030ec 00000008 004d0000 00200053 00680053 ......M.S. .S.h.
029030fc 006c0065 0020006c 006c0044 00000067 e.l.l. .D.l.g...
0290310c 00000000 00000000 50000043 00370007 ........C..P..7.
0290311c 00140015 00003009 0082ffff 0000ffff .....0..........
0290312c 00000000 00000000 00000000 5000008c ...............P
0290313c 00370023 000a00c8 00003500 0082ffff #.7......5......
0290314c 00000000 00000000 00000000 5000008c ...............P
0290315c 00410023 000a00eb 0000350b 0082ffff #.A......5......
0290316c 00000000 00000000 00000000 50000080 ...............P
0290317c 004b0023 000a00d2 0000350a 0082ffff #.K......5......
0290318c 00000000 00000000 00000000 50000080 ...............P
0290319c 00550023 002800d2 00003513 0082ffff #.U...(..5......
029031ac 00680054 00200065 00570025 004e0049 T.h.e. .%.W.I.N.
029031bc 004f0044 00530057 004c005f 004e004f D.O.W.S._.L.O.N.
029031cc 00250047 006f0020 00650070 00610072 G.%. .o.p.e.r.a.
029031dc 00690074 0067006e 00730020 00730079 t.i.n.g. .s.y.s.
029031ec 00650074 0020006d 006e0061 00200064 t.e.m. .a.n.d. .
16 Id: 10fc.124c Suspend: 0 Teb: 7ffd7000 Unfrozen
ChildEBP RetAddr Args to Child
053f8098 777b073f 777c3c9f 003d0650 00000001 ntdll!KiFastSystemCallRet
053f809c 777c3c9f 003d0650 00000001 00000000 user32!NtUserWaitMessage+0xc
053f80d0 777c2dc0 002e0378 003d0650 00000001 user32!DialogBox2+0x202
053f80f8 777c2eec 6f270000 03387bd4 003d0650 user32!InternalDialogBox+0xd0
053f8118 777c10ef 6f270000 03387bd4 003d0650 user32!DialogBoxIndirectParamAorW+0×37
053f813c 6f2c5548 6f270000 00005398 003d0650 user32!DialogBoxParamW+0×3f
053f8164 6f2c5743 6f270000 00005398 003d0650 ieframe!Detour_DialogBoxParamW+0×47
053f8188 6f2c56f5 6f270000 00005398 001905ea ieframe!SHFusionDialogBoxParam+0×32
053f9228 6f2c5378 001905ea 053fb540 00000104 ieframe!DoAddToFavDlgEx+0xcf
053fbb5c 6f2c58f9 001905ea 0e69a0c0 053fbff0 ieframe!AddToFavoritesEx+0×349
053fbdb8 6f2c57ee 00000000 053fbff0 00000000 ieframe!CBaseBrowser2::_AddToFavorites+0xe9
053fc0f4 6f2c3e5e 00000000 00000000 00000001 ieframe!CBaseBrowser2::_ExecAddToFavorites+0×123
053fc124 6f39ca4e 6f39c524 00000008 00000001 ieframe!CBaseBrowser2::_ExecExplorer+0xbe
053fc14c 6f39cee8 114ea39c 6f39c524 00000008 ieframe!CBaseBrowser2::Exec+0×12d
053fc17c 6f39cf17 6f39c524 00000008 00000001 ieframe!CShellBrowser2::_Exec_CCommonBrowser+0×80
053fc414 6f498284 114ea39c 6f39c524 00000008 ieframe!CShellBrowser2::Exec+0×626
053fc43c 6f49e5cd 0000a173 00000000 ffffff71 ieframe!CShellBrowser2::_FavoriteOnCommand+0×75
053fc458 6f3c5ea8 0000a173 00000000 00000111 ieframe!CShellBrowser2::_OnDefault+0×3e
053fd6f0 6f394194 0000a173 00000000 0000031a ieframe!CShellBrowser2::v_OnCommand+0xa7b
053fd70c 6f39898d 001905ea 00000111 0000a173 ieframe!CBaseBrowser2::v_WndProc+0×247
053fd770 6f3988db 001905ea 00000111 0000a173 ieframe!CShellBrowser2::v_WndProc+0×3fe
053fd794 777afd72 001905ea 00000111 0000a173 ieframe!CShellBrowser2::s_WndProc+0xfb
053fd7c0 777afe4a 6f39887a 001905ea 00000111 user32!InternalCallWinProc+0×23
053fd838 777b0943 00000000 6f39887a 001905ea user32!UserCallWinProcCheckWow+0×14b
053fd878 777b0b36 00252838 01223dc0 0000a173 user32!SendMessageWorker+0×4b7
053fd898 6f3cf032 001905ea 00000111 0000a173 user32!SendMessageW+0×7c
053fd8d0 6f396ead 0056049c 00000111 0000a173 ieframe!CInternetToolbarHost::v_WndProc+0xf8
053fd8f4 777afd72 0056049c 00000111 0000a173 ieframe!CImpWndProc::s_WndProc+0×65
053fd920 777afe4a 6f396e6e 0056049c 00000111 user32!InternalCallWinProc+0×23
053fd998 777b018d 00000000 6f396e6e 0056049c user32!UserCallWinProcCheckWow+0×14b
053fd9fc 777b022b 6f396e6e 00000000 053ffb14 user32!DispatchMessageWorker+0×322
053fda0c 6f39c1f5 053fda30 00000000 10eec4c0 user32!DispatchMessageW+0xf
053ffb14 6f34337f 0e7c3708 00000000 11bd8dc8 ieframe!CTabWindow::_TabWindowThreadProc+0×54c
053ffbcc 77525179 10eec4c0 00000000 053ffbe8 ieframe!LCIETab_ThreadProc+0×2c1
053ffbdc 7652d0e9 11bd8dc8 053ffc28 77b019bb iertutil!CIsoScope::RegisterThread+0xab
053ffbe8 77b019bb 11bd8dc8 7dd62326 00000000 kernel32!BaseThreadInitThunk+0xe
053ffc28 77b0198e 7752516b 11bd8dc8 00000000 ntdll!__RtlUserThreadStart+0×23
053ffc40 00000000 7752516b 11bd8dc8 00000000 ntdll!_RtlUserThreadStart+0×1b


0:000> dc 03387bd4 l50
03387bd4 ffff0001 00000000 00000000 80c808c0 ................
03387be4 0000000a 011f0000 00000064 00410000 ........d.....A.
03387bf4 00640064 00610020 00460020 00760061 d.d. .a. .F.a.v.
03387c04 0072006f 00740069 00000065 00000008 o.r.i.t.e.......
03387c14 004d0000 00200053 00680053 006c0065 ..M.S. .S.h.e.l.
03387c24 0020006c 006c0044 00000067 00000000 l. .D.l.g.......
03387c34 00000000 50000003 0007000f 00140015 .......P........
03387c44 00009760 0082ffff 00bfffff 00000000 `...............
03387c54 00000000 00000000 50020000 00070035 ...........P5...
03387c64 000800db 000003f4 0082ffff 00640041 ............A.d.
03387c74 00200064 00200061 00610046 006f0076 d. .a. .F.a.v.o.
03387c84 00690072 00650074 00000000 00000000 r.i.t.e.........
03387c94 00000000 50020000 00110035 001000db .......P5.......
03387ca4 000003f5 0082ffff 00640041 00200064 ........A.d.d. .
03387cb4 00680074 00730069 00770020 00620065 t.h.i.s. .w.e.b.
03387cc4 00610070 00650067 00610020 00200073 p.a.g.e. .a.s. .
03387cd4 00200061 00610066 006f0076 00690072 a. .f.a.v.o.r.i.
03387ce4 00650074 0020002e 006f0054 00610020 t.e... .T.o. .a.
03387cf4 00630063 00730065 00200073 006f0079 c.c.e.s.s. .y.o.
03387d04 00720075 00660020 00760061 0072006f u.r. .f.a.v.o.r.

Stack traces with DialogBoxIndirectParam call and x64 platform complicates the picture a bit. Please also note that a user might not see the dialog box you see on a stack trace due to many reasons like terminal session problems or a process running in a non-interactive session.

Technology-Specific Subtrace (COM Interface Invocation)

Stack Trace (Volume 1, page 395) is a general pattern and there can always be found fine-grained patterns in stack traces as well. Here we discuss the general category of such stack trace patterns (TSST) and give examples related to COM technology.

Consider this trace:

1: kd> k250
ChildEBP RetAddr
8d5d2808 82a7eb15 nt!KiSwapContext+0x26
8d5d2840 82a7d403 nt!KiSwapThread+0x266
8d5d2868 82a772cf nt!KiCommitThreadWait+0x1df
8d5d28e0 82550d75 nt!KeWaitForSingleObject+0x393
8d5d293c 82550e10 win32k!xxxRealSleepThread+0x1d7
8d5d2958 824ff4b0 win32k!xxxSleepThread+0x2d
8d5d29cc 825547e8 win32k!xxxInterSendMsgEx+0xb1c
8d5d2a1c 825546a4 win32k!xxxSendMessageTimeout+0x13b
8d5d2a44 82533843 win32k!xxxSendMessage+0×28
8d5d2b08 824fd865 win32k!xxxCalcValidRects+0xf7
8d5d2b64 82502c98 win32k!xxxEndDeferWindowPosEx+0×100
8d5d2b84 825170c9 win32k!xxxSetWindowPos+0xf6
8d5d2c08 82517701 win32k!xxxActivateThisWindow+0×2b1
8d5d2c38 82517537 win32k!xxxActivateWindow+0×144
8d5d2c4c 824fd9dd win32k!xxxSwpActivate+0×44
8d5d2ca4 82502c98 win32k!xxxEndDeferWindowPosEx+0×278
8d5d2cc4 824fff82 win32k!xxxSetWindowPos+0xf6
8d5d2d10 82a5342a win32k!NtUserSetWindowPos+0×140
8d5d2d10 76ee64f4 nt!KiFastCallEntry+0×12a (TrapFrame @ 8d5d2d34)
01e2cea0 7621358d ntdll!KiFastSystemCallRet
01e2cea4 6a8fa0eb USER32!NtUserSetWindowPos+0xc
01e2cf14 6a894b13 IEFRAME!SHToggleDialogExpando+0×15a
01e2cf28 6a894d5d IEFRAME!EleDlg::ToggleExpando+0×20
01e2d74c 6a895254 IEFRAME!EleDlg::OnInitDlg+0×229
01e2d7b8 762186ef IEFRAME!EleDlg::DlgProcEx+0×189
01e2d7e4 76209eb2 USER32!InternalCallWinProc+0×23
01e2d860 7620b98b USER32!UserCallDlgProcCheckWow+0xd6
01e2d8a8 7620bb7b USER32!DefDlgProcWorker+0xa8
01e2d8c4 762186ef USER32!DefDlgProcW+0×22
01e2d8f0 76218876 USER32!InternalCallWinProc+0×23
01e2d968 76217631 USER32!UserCallWinProcCheckWow+0×14b
01e2d9a8 76209b1d USER32!SendMessageWorker+0×4d0
01e2da64 76235500 USER32!InternalCreateDialog+0xb0d
01e2da94 76235553 USER32!InternalDialogBox+0xa7
01e2dab4 76235689 USER32!DialogBoxIndirectParamAorW+0×37
01e2dad8 6a5d4952 USER32!DialogBoxParamW+0×3f
01e2db00 6a5d5024 IEFRAME!Detour_DialogBoxParamW+0×47
01e2db24 6a8956df IEFRAME!SHFusionDialogBoxParam+0×32
01e2db58 6a8957bb IEFRAME!EleDlg::ShowDialog+0×398
01e2e638 6a8959d3 IEFRAME!ShowDialogBox+0xb6
01e2eb9c 6a9013ed IEFRAME!ShowElevationPrompt+0×1dd
01e2f010 7669fc8f
IEFRAME!CIEUserBrokerObject::BrokerCoCreateInstance+0×202
01e2f040 76704c53 RPCRT4!Invoke+0×2a
01e2f448 76d9d936 RPCRT4!NdrStubCall2+0×2d6
01e2f490 76d9d9c6 ole32!CStdStubBuffer_Invoke+0xb6
01e2f4d8 76d9df1f ole32!SyncStubInvoke+0×3c
01e2f524 76cb213c ole32!StubInvoke+0xb9
01e2f600 76cb2031 ole32!CCtxComChnl::ContextInvoke+0xfa
01e2f61c 76d9a754 ole32!MTAInvoke+0×1a
01e2f64c 76d9dcbb ole32!AppInvoke+0xab
01e2f72c 76d9a773 ole32!ComInvokeWithLockAndIPID+0×372
01e2f778 7669f34a ole32!ThreadInvoke+0×302
01e2f7b4 7669f4da RPCRT4!DispatchToStubInCNoAvrf+0×4a
01e2f80c 7669f3c6 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×16c
01e2f834 766a0cef RPCRT4!RPC_INTERFACE::DispatchToStub+0×8b
01e2f86c 7669f882 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xb2
01e2f8b8 7669f7a4 RPCRT4!LRPC_SCALL::DispatchRequest+0×23b
01e2f8d8 7669f763 RPCRT4!LRPC_SCALL::QueueOrDispatchCall+0xbd
01e2f8f4 7669f5ff RPCRT4!LRPC_SCALL::HandleRequest+0×34f
01e2f928 7669f573 RPCRT4!LRPC_SASSOCIATION::HandleRequest+0×144
01e2f960 7669ee4f RPCRT4!LRPC_ADDRESS::HandleRequest+0xbd
01e2f9dc 7669ece7 RPCRT4!LRPC_ADDRESS::ProcessIO+0×50a
01e2f9e8 766a1357 RPCRT4!LrpcServerIoHandler+0×16
01e2f9f8 76ecd3a3 RPCRT4!LrpcIoComplete+0×16
01e2fa20 76ed0748 ntdll!TppAlpcpExecuteCallback+0×1c5
01e2fb88 76e11174 ntdll!TppWorkerThread+0×5a4
01e2fb94 76efb3f5 kernel32!BaseThreadInitThunk+0xe
01e2fbd4 76efb3c8 ntdll!__RtlUserThreadStart+0×70
01e2fbec 00000000 ntdll!_RtlUserThreadStart+0×1b

In the middle of the stack trace we see COM interface invocation in IEFRAME module. The similar stack trace fragment can be found in the following stack trace where COM IRemUnknown interface implementation resides in .NET CLR mscorwks module:

0:000> kL
ChildEBP RetAddr
0018a924 68b5f8f0 mscorwks!SafeReleaseHelper+0x77
0018a958 68b04a99 mscorwks!SafeRelease+0x2f
0018a98c 68b04860 mscorwks!IUnkEntry::Free+0x68
0018a9a0 68b049b5 mscorwks!RCW::ReleaseAllInterfaces+0x18
0018a9d0 68b049e1 mscorwks!RCW::ReleaseAllInterfacesCallBack+0xbd
0018aa00 68c0a108 mscorwks!RCW::Cleanup+0x22
0018aa0c 68c0a570 mscorwks!RCWCleanupList::ReleaseRCWListRaw+0x16
0018aa3c 68bd4b3d mscorwks!RCWCleanupList::ReleaseRCWListInCorrectCtx+0xdf
0018aa4c 75dd8c2e mscorwks!CtxEntry::EnterContextCallback+0×89
0018aa68 763c586c ole32!CRemoteUnknown::DoCallback+0×7a
0018aa84 764405f1 rpcrt4!Invoke+0×2a
0018ae88 75efd936 rpcrt4!NdrStubCall2+0×2ea
0018aed0 75efd9c6 ole32!CStdStubBuffer_Invoke+0xb6
0018af18 75efdf1f ole32!SyncStubInvoke+0×3c
0018af64 75e1223c ole32!StubInvoke+0xb9
0018b040 75e12131 ole32!CCtxComChnl::ContextInvoke+0xfa
0018b05c 75e130fa ole32!MTAInvoke+0×1a
0018b088 75efde47 ole32!STAInvoke+0×46
0018b0bc 75efdcbb ole32!AppInvoke+0xab
0018b19c 75efe34c ole32!ComInvokeWithLockAndIPID+0×372
0018b1c4 75e12ed2 ole32!ComInvoke+0xc5
0018b1d8 75e12e91 ole32!ThreadDispatch+0×23
0018b21c 75a06238 ole32!ThreadWndProc+0×161
0018b248 75a068ea user32!InternalCallWinProc+0×23
0018b2c0 75a07d31 user32!UserCallWinProcCheckWow+0×109
0018b320 75a07dfa user32!DispatchMessageWorker+0×3bc
0018b330 75ddd6be user32!DispatchMessageW+0xf
0018b360 75ddd66d ole32!CCliModalLoop::PeekRPCAndDDEMessage+0×4c
0018b390 75ddd57e ole32!CCliModalLoop::FindMessage+0×30
0018b3f0 75ddd633 ole32!CCliModalLoop::HandleWakeForMsg+0×41
0018b408 75dd1117 ole32!CCliModalLoop::BlockFn+0xc3
0018b488 68a6c905 ole32!CoWaitForMultipleHandles+0xcd
0018b4a8 68a6c866 mscorwks!NT5WaitRoutine+0×51
0018b514 68a6c7ca mscorwks!MsgWaitHelper+0xa5
0018b534 68b5fbe4 mscorwks!Thread::DoAppropriateAptStateWait+0×28
0018b5b8 68b5fc79 mscorwks!Thread::DoAppropriateWaitWorker+0×13c
0018b608 68b5fdf9 mscorwks!Thread::DoAppropriateWait+0×40
0018b664 68a1c5b6 mscorwks!CLREvent::WaitEx+0xf7
0018b678 68b1adb4 mscorwks!CLREvent::Wait+0×17
0018b6c8 68b1ab2a mscorwks!WKS::GCHeap::FinalizerThreadWait+0xfb
0018b764 08fa12c1 mscorwks!GCInterface::RunFinalizers+0×99
[...]

A TSST usually spans several modules. In any stack trace we can also find several TSST that may be overlapping. For example, in the first stack trace above we can discern fragments of COM, RPC, LPC, GUI Dialog, Window Management, and Window Messaging subtraces. In the second trace we can also see GC, Modal Loop, COM Wrapper, and Interface Management stack frames.

The closest software trace analysis pattern here is Implementation Discourse (page 245).

Livelock

Finally I got a few good crash dumps illustrating Livelock pattern when 2 threads are looping while acquiring and releasing a resource but not progressing. We have these signs in selected WinDbg output below:

• high contention patterns⁷ or context switch counts

• increased CPU time in user and / or kernel mode (Volume 1, page 305)

• at least one livelocked thread is scheduled for execution

• one of livelocked threads has unusual priority boost

• the same thread stack trace (Volume 1, page 395) for both live-locked threads having similar stats like spent time and context switch counts

• zero waiting ticks

1: kd> !locks

Resource @ 0xfffffa8008464528 Exclusively owned
Contention Count = 43743004
NumberOfExclusiveWaiters = 1
Threads: fffffa8008315b60-01<*>
Threads Waiting On Exclusive Access:
fffffa8005769660

41080 total locks, 1 locks currently held

1: kd> !running

Prcbs Current Next
1 fffff88001e68180 fffff88001e72fc0 fffffa8008315b60 ................

We have these stack traces from stack trace collection (Volume 1, page 409):

THREAD fffffa8008315b60 Cid 0724.2a28 Teb: 000007fffff9c000 Win32Thread:
0000000000000000 ????
IRP List:
fffffa80082e5990: (0006,0118) Flags: 00060000 Mdl: 00000000
Not impersonating
DeviceMap fffff8a009f434e0
Owning Process fffffa8005726360    Image:  ProcessA.exe
Attached Process         N/A       Image:  N/A
Wait Start TickCount     522197    Ticks:  0
Context Switch Count 21665144
UserTime         00:00:40.373
KernelTime       00:02:42.791
Win32 Start Address 0×000007fef6939430
Stack Init fffff88007321db0 Current fffff88007321520
Base fffff88007322000 Limit fffff8800731c000 Call 0
Priority 8 BasePriority 6 UnusualBoost 1 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
fffff880`07321560 fffff800`0168a992 nt!KiSwapContext+0×7a
fffff880`073216a0 fffff800`0168ccff nt!KiCommitThreadWait+0×1d2
fffff880`07321730 fffff800`0164c242 nt!KeWaitForSingleObject+0×19f
fffff880`073217d0 fffff800`0168b5ac nt!ExpWaitForResource+0xae
fffff880`07321840 fffff880`04608d91 nt!ExAcquireResourceExclusiveLite+0×14f
fffff880`073218b0 fffff880`04609e4e DriverA!foo+0×42
[...]
fffff880`07321a10 fffff800`0199ef66 nt!IopXxxControlFile+0×607
fffff880`07321b40 fffff800`01682993 nt!NtDeviceIoControlFile+0×56
fffff880`07321bb0 00000000`76ffff2a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @
fffff880`07321c20)
00000000`03a1f488 000007fe`fd06b399 ntdll!NtDeviceIoControlFile+0xa
00000000`03a1f490 00000000`76ea610f KERNELBASE!DeviceIoControl+0×75
00000000`03a1f500 000007fe`f74f3d7c kernel32!DeviceIoControlImplementation+0×7f
[...]

THREAD fffffa8005769660 Cid 0724.10b0 Teb: 000007fffffa6000 Win32Thread:
0000000000000000 WAIT: (WrResource) KernelMode Non-Alertable
fffffa8006661f20 SynchronizationEvent
IRP List:
fffffa8006b1ac10: (0006,0118) Flags: 00060000 Mdl: 00000000
Not impersonating
DeviceMap fffff8a009f434e0
Owning Process fffffa8005726360     Image:   ProcessA.exe
Attached Process          N/A       Image:   N/A
Wait Start TickCount      522197    Ticks:   0
Context Switch Count 21601988
UserTime 00:00:30.147
KernelTime 00:02:30.972
Win32 Start Address 0×000007fef6939430
Stack Init fffff880071bbdb0 Current fffff880071bb520
Base fffff880071bc000 Limit fffff880071b6000 Call 0
Priority 7 BasePriority 6 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
fffff880`071bb560 fffff800`0168a992 nt!KiSwapContext+0×7a
fffff880`071bb6a0 fffff800`0168ccff nt!KiCommitThreadWait+0×1d2
fffff880`071bb730 fffff800`0164c242 nt!KeWaitForSingleObject+0×19f
fffff880`071bb7d0 fffff800`0168b5ac nt!ExpWaitForResource+0xae
fffff880`071bb840 fffff880`04608d91 nt!ExAcquireResourceExclusiveLite+0×14f
fffff880`071bb8b0 fffff880`04609e4e DriverA!foo+0×42
[...]
fffff880`071bba10 fffff800`0199ef66 nt!IopXxxControlFile+0×607
fffff880`071bbb40 fffff800`01682993 nt!NtDeviceIoControlFile+0×56
fffff880`071bbbb0 00000000`76ffff2a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @
fffff880`071bbc20)
00000000`033bf708 000007fe`fd06b399 ntdll!NtDeviceIoControlFile+0xa
00000000`033bf710 00000000`76ea610f KERNELBASE!DeviceIoControl+0×75
00000000`033bf780 000007fe`f74f3d7c kernel32!DeviceIoControlImplementation+0×7f
[...]

In both traces we have DriverA as Blocking Module (page 54).

Semantic Structure (PID.TID)

This part starts the block of patterns called Semantic Structures. These structures are fragments of memory which have meaning helping us in troubleshooting and debugging. The first pattern in this block deals with PID.TID structures of the form DWORD : DWORD or QWORD : QWORD. Such memory fragments are useful for wait chain analysis⁸, for example, by looking at the execution residue (Volume 2, page 239) left on a raw stack to find a target or an origin of RPC or (A)LPC calls. RPC target example can be found in the article: In Search of Lost CID (Volume 2, page 136). Here we look at another example, this time to find the originator of an ALPC call.

A ServiceA process was executing some undesired functionality and a breakpoint was set on ModuleA code to trigger it under irreproducible conditions. Then a complete memory dump was saved for offline analysis. There we see an ALPC server thread that triggered the breakpoint but we don't see the message information in the output of WinDbg !thread command that can help us finding a corresponding ALPC client thread easily:

THREAD fffffa8005e6b060 Cid 0cc0.1838 Teb: 000007fffff8e000 Win32Thread:
0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
SuspendCount 1
fffff880094ad0a0 SynchronizationEvent
Not impersonating
DeviceMap        fffff8a001aba3c0
Owning Process   fffffa8004803b30 Image: ServiceA.exe
Attached Process N/A              Image: N/A
Wait Start TickCount 1441562      Ticks: 106618 (0:00:27:43.251)
Context Switch Count 414
UserTime           00:00:00.000
KernelTime         00:00:00.031
Win32 Start Address ntdll!TppWorkerThread (0×0000000077c88f00)
Stack Init fffff880094addb0 Current fffff880094acdb0
Base fffff880094ae000 Limit fffff880094a8000 Call 0
Priority 12 BasePriority 10 UnusualBoost 0 ForegroundBoost 0 IoPriority 2
PagePriority 5
Child-SP RetAddr Call Site
fffff880`094acdf0 fffff800`01678992 nt!KiSwapContext+0×7a
fffff880`094acf30 fffff800`0167acff nt!KiCommitThreadWait+0×1d2
fffff880`094acfc0 fffff800`01a150e8 nt!KeWaitForSingleObject+0×19f
fffff880`094ad060 fffff800`01a1546c nt!DbgkpQueueMessage+0×2a8
fffff880`094ad230 fffff800`019b9116 nt!DbgkpSendApiMessage+0×5c
fffff880`094ad270 fffff800`016abb96 nt! ?? ::NNGAKEGL::`stringrsquo;+0×3463d
fffff880`094ad3b0 fffff800`01670d82 nt!KiDispatchException+0×316
fffff880`094ada40 fffff800`0166ebb4 nt!KiExceptionDispatch+0xc2
fffff880`094adc20 000007fe`f79365d1 nt!KiBreakpointTrap+0xf4 (TrapFrame @
fffff880`094adc20)
00000000`035ee568 000007fe`f80670b5 ModuleA+0×38611
[...]
00000000`035ee5d0 000007fe`ff4bc7f5 ModuleB!Start+0×6e1
00000000`035ee770 000007fe`ff56b62e RPCRT4!Invoke+0×65
00000000`035ee7c0 000007fe`ff4bf1f6 RPCRT4!Ndr64StubWorker+0×61b
00000000`035eed80 000007fe`ffedf223 RPCRT4!NdrStubCall3+0xb5
00000000`035eede0 000007fe`ffedfc0d ole32!CStdStubBuffer_Invoke+0×5b
00000000`035eee10 000007fe`ffedfb83 ole32!SyncStubInvoke+0×5d
00000000`035eee80 000007fe`ffd7fd60 ole32!StubInvoke+0xdb
00000000`035eef30 000007fe`ffedfa22 ole32!CCtxComChnl::ContextInvoke+0×190
00000000`035ef0c0 000007fe`ffedf76b ole32!AppInvoke+0xc2
00000000`035ef130 000007fe`ffeded6d ole32!ComInvokeWithLockAndIPID+0×52b
00000000`035ef2c0 000007fe`ff4b9c24 ole32!ThreadInvoke+0×30d
00000000`035ef360 000007fe`ff4b9d86 RPCRT4!DispatchToStubInCNoAvrf+0×14
00000000`035ef390 000007fe`ff4bc44b RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×146
00000000`035ef4b0 000007fe`ff4bc38b RPCRT4!RPC_INTERFACE::DispatchToStub+0×9b
00000000`035ef4f0 000007fe`ff4bc322
RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0×5b
00000000`035ef570 000007fe`ff4ba11d RPCRT4!LRPC_SCALL::DispatchRequest+0×422
00000000`035ef650 000007fe`ff4c7ddf RPCRT4!LRPC_SCALL::HandleRequest+0×20d
00000000`035ef780 000007fe`ff4c7995 RPCRT4!LRPC_ADDRESS::ProcessIO+0×3bf
00000000`035ef8c0 00000000`77c8b43b RPCRT4!LrpcIoComplete+0xa5
00000000`035ef950 00000000`77c8923f ntdll!TppAlpcpExecuteCallback+0×26b
00000000`035ef9e0 00000000`77a6f56d ntdll!TppWorkerThread+0×3f8
00000000`035efce0 00000000`77ca3281 kernel32!BaseThreadInitThunk+0xd
00000000`035efd10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We inspect the raw stack starting from the first top Child-SP value for RPCRT4 subtrace and find NNN:NNN data there resembling a PID:TID pair:

1: kd> dpp 00000000`035ef360 l100
[...]
00000000`035ef698 00000000`00000000
00000000`035ef6a0 00000000`00000001
00000000`035ef6a8 00000000`00000000
00000000`035ef6b0 00000000`00000000
00000000`035ef6b8 00000000`00000118
00000000`035ef6c0 00000000`0000048c
00000000`035ef6c8 00000000`00495e50 000007fe`ff57d920 RPCRT4!LRPC_ADDRESS::`vftable’
00000000`035ef6d0 00000000`00000000
[...]

We find such CID in the stack trace collection (Volume 1, page 409) and see a wait for an ALPC message reply:

THREAD fffffa8003d49b60 Cid 0118.048c Teb: 000007fffffaa000 Win32Thread:
fffff900c01e4c30 WAIT: (WrLpcReply) UserMode Non-Alertable
fffffa8003d49f20 Semaphore Limit 0×1
Waiting for reply to ALPC Message fffff8a000bdb6c0 : queued at port fffffa80042f8090
: owned by process fffffa8004803b30
Not impersonating
DeviceMap        fffff8a000008600
Owning Process   fffffa8003cf15d0 Image: ServiceB.exe
Attached Process N/A                      Image: N/A
Wait Start TickCount 1441554              Ticks: 106626 (0:00:27:43.376)
Context Switch Count 23180 LargeStack
UserTime                  00:00:00.468
KernelTime                00:00:03.057
Win32 Start Address ntdll!TppWorkerThread (0×0000000077c88f00)
Stack Init fffff88004ffcdb0 Current fffff88004ffc620
Base fffff88004ffd000 Limit fffff88004ff7000 Call 0
Priority 6 BasePriority 6 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP RetAddr Call Site
fffff880`04ffc660 fffff800`01678992 nt!KiSwapContext+0×7a
fffff880`04ffc7a0 fffff800`0167acff nt!KiCommitThreadWait+0×1d2
fffff880`04ffc830 fffff800`0168fd1f nt!KeWaitForSingleObject+0×19f
fffff880`04ffc8d0 fffff800`01977ac6 nt!AlpcpSignalAndWait+0×8f
fffff880`04ffc980 fffff800`01975a50 nt!AlpcpReceiveSynchronousReply+0×46
fffff880`04ffc9e0 fffff800`01972fcb nt!AlpcpProcessSynchronousRequest+0×33d
fffff880`04ffcb00 fffff800`01670993 nt!NtAlpcSendWaitReceivePort+0×1ab
fffff880`04ffcbb0 00000000`77cc070a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @
fffff880`04ffcc20)
00000000`018ce308 000007fe`ff4caa76 ntdll!ZwAlpcSendWaitReceivePort+0xa
00000000`018ce310 000007fe`ff4bf802 RPCRT4!LRPC_CCALL::SendReceive+0×156
00000000`018ce3d0 000007fe`ffee0900 RPCRT4!I_RpcSendReceive+0×42
00000000`018ce400 000007fe`ffee05ef ole32!ThreadSendReceive+0×40
00000000`018ce450 000007fe`ffee041b
ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xa3
00000000`018ce4f0 000007fe`ffd819c6 ole32!CRpcChannelBuffer::SendReceive2+0×11b
00000000`018ce6b0 000007fe`ffd81928 ole32!CAptRpcChnl::SendReceive+0×52
00000000`018ce780 000007fe`ffedfcf5 ole32!CCtxComChnl::SendReceive+0×68
00000000`018ce830 000007fe`ff56ba3b ole32!NdrExtpProxySendReceive+0×45
00000000`018ce860 000007fe`ffee02d0 RPCRT4!NdrpClientCall3+0×2e2
00000000`018ceb20 000007fe`ffd818a2 ole32!ObjectStublessClient+0×11d
00000000`018ceeb0 00000000`ff5afe64 ole32!ObjectStubless+0×42
[...]
00000000`018cf7a0 00000000`77c8f8eb ServiceB!Worker+0×366
00000000`018cf800 00000000`77c89d9f ntdll!RtlpTpWorkCallback+0×16b
00000000`018cf8e0 00000000`77a6f56d ntdll!TppWorkerThread+0×5ff
00000000`018cfbe0 00000000`77ca3281 kernel32!BaseThreadInitThunk+0xd
00000000`018cfc10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

Inspection of that message shows that it was directed to our server thread that triggered the breakpoint:

1: kd> !alpc /m fffff8a000bdb6c0

Message @ fffff8a000bdb6c0
MessageID      : 0x0600 (1536)
CallbackID     : 0x2D910D (2986253)
SequenceNumber : 0x0002CB50 (183120)
Type           : LPC_REQUEST
DataLength     : 0x0068 (104)
TotalLength    : 0x0090 (144)
Canceled       : No
Release        : No
ReplyWaitReply : No
Continuation   : Yes
OwnerPort      : fffffa8004823a80 [ALPC_CLIENT_COMMUNICATION_PORT]
WaitingThread  : fffffa8003d49b60
QueueType      : ALPC_MSGQUEUE_PENDING
QueuePort      : fffffa80042f8090 [ALPC_CONNECTION_PORT]
QueuePortOwnerProcess : fffffa8004803b30 (ServiceA.exe)
ServerThread    : fffffa8005e6b060
QuotaCharged    : No
CancelQueuePort  :  0000000000000000
CancelSequencePort   : 0000000000000000
CancelSequenceNumber   : 0x00000000 (0)
ClientContext  : 000000000020f0c0
ServerContext  : 0000000000000000
PortContext    : 0000000000416990
CancelPortContext : 0000000000000000
SecurityData   : 0000000000000000
View           : 0000000000000000

Instrumentation Side Effect

Sometimes added instrumentation via gflags, application and driver verifier options affect system, service or application performance and resources. For example, after enabling full page heap, one process on an x64 machine was growing up to 24GB and its user memory dump shows that every heap allocation was recorded in a stack trace database:

0:055> !gflag
Current NtGlobalFlag contents: 0x02000000
hpa - Place heap allocations at ends of pages

0:055> ~*kc

[...]

48 Id: 117fc.c164 Suspend: 1 Teb: 000007ff`fff52000 Unfrozen
Call Site
ntdll!ZwWaitForSingleObject
ntdll!RtlpWaitOnCriticalSection
ntdll!RtlEnterCriticalSection
verifier!AVrfpDphEnterCriticalSection
verifier!AVrfpDphPreProcessing
verifier!AVrfDebugPageHeapAllocate
ntdll!RtlDebugAllocateHeap
ntdll! ?? ::FNODOBFM::‘string’
ntdll!RtlAllocateHeap
msvcrt!malloc
ModuleA!foo1
[...]

49 Id: 117fc.de80 Suspend: 1 Teb: 000007ff`fff54000 Unfrozen
Call Site
ntdll!RtlCompareMemory
ntdll!RtlpLogCapturedStackTrace
ntdll!RtlLogStackTrace
verifier!AVrfpDphPlaceOnFreeList
verifier!AVrfDebugPageHeapFree
ntdll!RtlDebugFreeHeap
ntdll! ?? ::FNODOBFM::‘string’
ntdll!RtlFreeHeap
kernel32!HeapFree
msvcrt!free
ModuleB!foo2
[...]
50 Id: 117fc.3700 Suspend: 1 Teb: 000007ff`fff4e000 Unfrozen
Call Site
ntdll!ZwWaitForSingleObject
ntdll!RtlpWaitOnCriticalSection
ntdll!RtlEnterCriticalSection
verifier!AVrfpDphEnterCriticalSection
verifier!AVrfpDphPreProcessing
verifier!AVrfDebugPageHeapFree
ntdll!RtlDebugFreeHeap
ntdll! ?? ::FNODOBFM::‘string’
ntdll!RtlFreeHeap
kernel32!HeapFree
msvcrt!free
ModuleC!foo3
[...]

0:055> !runaway
User Mode Time
Thread Time
38:d090 0 days 0:02:28.793
44:ca48 0 days 0:01:04.459
48:c164 0 days 0:00:56.909
43:4458 0 days 0:00:54.475
50:3700 0 days 0:00:43.992
45:6f98 0 days 0:00:38.953
49:de80 0 days 0:00:24.211
1:391c 0 days 0:00:00.639
0:7e90 0 days 0:00:00.109
55:a300 0 days 0:00:00.046
34:10c9c 0 days 0:00:00.015
21:d054 0 days 0:00:00.015
56:b0a0 0 days 0:00:00.000
54:8b78 0 days 0:00:00.000
53:155b8 0 days 0:00:00.000
52:b444 0 days 0:00:00.000

Top modules ModuleA(B, C) from the spiking (Volume 1, page 305) and heap intensive threads are from the same vendor (Volume 5, page 128).

We were able to get a 200×27349 slice from that dump using ImageMagick⁹ and it shows almost all virtual memory space filled with traces of this pictorial form (magnified by x8):

Directing Module

In certain software behavior scenarios such as a memory leak (Volume 1, page 356) when we see top modules (page 62) calling OS API functions we might suspect them having defects. However, this might not be the case when these modules were used from a directing module keeping references or handles preventing top modules from freeing memory or releasing resources.

For example, a memory dump from a process had 2 growing heap segments and one of them had this recurrent stack trace saved in a user mode stack trace database:

38D2CE78: 02ba8 . 02ba8 [07] - busy (2b90), tail fill
Stack trace (38101) at 83e390:
7d6568be: ntdll!RtlAllocateHeapSlowly+0×00000041
7d62b846: ntdll!RtlAllocateHeap+0×00000E9F
337d0572: ModuleA!XHeapAlloc+0×00000115
[...]
338809e2: ModuleA!Execute+0×000002CD
488b3fc1: ModuleB!Execute+0×000000D3
679b8c64: ModuleC!ExecuteByHandle+0×00000074
[...]
67d241cb: ModuleD!Query+0×0000016B
67ba2ed4: ModuleE!Browse+0×000000E4
[...]
667122c6: ModuleF!Check+0×00000126
65e73826: ModuleG!Enum+0×00000406
[...]

Initially we suspected ModuleA but found a different recurrent stack trace corresponding to another growing segment:

40C81688: 000c8 . 00058 [07] - busy (40), tail fill
Stack trace (38136) at 83f6a4:
7d6568be: ntdll!RtlAllocateHeapSlowly+0×00000041
7d62b846: ntdll!RtlAllocateHeap+0×00000E9F
7c3416b3: msvcr71!_heap_alloc+0×000000E0
7c3416db: msvcr71!_nh_malloc+0×00000010
67745875: ModuleX!BufAllocate+0×00000015
6775085e: ModuleY!QueryAttribute+0×0000008E
[...]
677502b5: ModuleY!Query+0×00000015
67ba2f19: ModuleE!Browser+0×00000129
[...]
667122c6: ModuleF!Check+0×00000126
65e73826: ModuleG!Enum+0×00000406
[...]

From the common stack trace fragment (highlighted in bold italics) we transferred our investigation to ModuleE, and indeed, the similar software incident (as per the latter stack trace) was found in a troubleshooting database.

Stack Overflow (Software Implementation)

Stack Overflow pattern variants in user (Volume 2, page 279) and kernel mode (Volume 1, page 314) are ISA (Instruction Set Architecture) and processor architecture oriented. Another pattern variant is a software stack implementation where push and pop operations check a stack ADT precondition and throw a software exception (overflow or underflow) or call an assertion mechanism to display an error message. For the latter example, we look at a bugcheck for the specific stack implementation on Windows: IRP stack locations array. For a graphical reminder on how driver-to-driver communication is implemented by an IRP stack corresponding to a driver stack please refer to a UML diagram in Volume 1, page 701. The following WinDbg command output is from a kernel memory dump:

0: kd> !analyze -v
[...]
NO_MORE_IRP_STACK_LOCATIONS (35)
A higher level driver has attempted to call a lower level driver through
the IoCallDriver() interface, but there are no more stack locations in the
packet, hence, the lower level driver would not be able to access its
parameters, as there are no parameters for it. This is a disasterous
situation, since the higher level driver “thinks” it has filled in the
parameters for the lower level driver (something it MUST do before it
calls it), but since there is no stack location for the latter driver, the
former has written off of the end of the packet. This means that some
other memory has probably been trashed at this point.
Arguments:
Arg1: fffffa800500c9e0, Address of the IRP
Arg2: 0000000000000000
Arg3: 0000000000000000
Arg4: 0000000000000000
[...]

0: kd> kL 100
Child-SP RetAddr Call Site
fffff880`01fe2338 fffff800`016d7732 nt!KeBugCheckEx
fffff880`01fe2340 fffff800`01754f27 nt!KiBugCheck3+0x12
fffff880`01fe2370 fffff880`0177e271 nt! ?? ::FNODOBFM::‘string’+0×3f31b
fffff880`01fe23a0 fffff880`0177c138 DriverA!CallProvider+0×161
[...]
fffff880`01fe2cb0 fffff800`0197a7c6 nt!ExpWorkerThread+0×111
fffff880`01fe2d40 fffff800`016b5c26 nt!PspSystemThreadStartup+0×5a
fffff880`01fe2d80 00000000`00000000 nt!KxStartSystemThread+0×16
0: kd> !irp fffffa800500c9e0
Irp is active with 1 stacks 0 is current (= 0xfffffa8006c2e960)
No Mdl: No System Buffer: Thread 00000000: Irp stack trace.
cmd   flg cl   Device           File             Completion-Context
[ 4, 0] 0 e0   fffffa8004045c50 fffffa8006c2e960 fffff88005a04460-
fffffa8005b9c370 Success Error Cancel
DriverA DriverB!CompleteRoutine
Args: 00000008 00000000 00000000 00000000

0: kd> ub fffff880`0177e271
DriverA!CallProvider+0×13e:
fffff880`0177e24e mov qword ptr [r11-10h],rax
fffff880`0177e252 mov qword ptr [r11-8],r12
fffff880`0177e256 mov byte ptr [r11-45h],0E0h
fffff880`0177e25b mov rcx,qword ptr [rdi+40h]
fffff880`0177e25f call qword ptr [DriverA!_imp_IoGetAttachedDevice
(fffff880`017790b0)]
fffff880`0177e265 mov rdx,rbp
fffff880`0177e268 mov rcx,rax
fffff880`0177e26b call qword ptr [DriverA!_imp_IofCallDriver
(fffff880`01779068)]

Data Correlation

This is a general pattern where values found in different parts of a memory dump correlate between each other according to some rules, for example, in some proportion. Here we show a variant for function parameters.

A process user memory dump had a C++ exception (Volume 3, page 84) inside:

0:000> kL
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
0012e950 78158e89 kernel32!RaiseException+0x53
0012e988 7830770c msvcr80!_CxxThrowException+0×46
0012e99c 783095bc mfc80u!AfxThrowMemoryException+0×19
0012e9b4 02afa8ca mfc80u!operator new+0×27
0012e9c8 02b0992f ModuleA!std::_Allocate<...>+0×1a
0012e9e0 02b09e7c ModuleA!std::vector<double,std::allocator<double>
>::vector<double,std::allocator<double> >+0×3f
[...]

We suspected an out-of-memory condition and looked for function parameters:

0:000> kv 5
ChildEBP RetAddr Args to Child
0012e950 78158e89 e06d7363 00000001 00000003 kernel32!RaiseException+0x53
0012e988 7830770c 0012e998 783b0110 783c8d68 msvcr80!_CxxThrowException+0x46
0012e99c 783095bc 0000a7c0 0012ea40 000014f8 mfc80u!AfxThrowMemoryException+0x19
0012e9b4 02afa8ca 0000a7c0 089321b0 089321f0 mfc80u!operator new+0×27 (FPO: [Uses
EBP] [1,0,0])
0012e9c8 02b0992f 000014f8 00000000 00000008 ModuleA!std::_Allocate<...>+0×1a (FPO:
[2,3,0])

Because of FPO optimization (Volume 2, page 169) we originally thought that stack arguments would be invalid. However, knowing the function prototype and semantics of operator new¹⁰ and std::vector double element type we immediately see the correlation between 0xa7c0 and 0×14f8 which are proportional to sizeof(double) == 8:

0:000> ? 0000a7c0/000014f8
Evaluate expression: 8 = 00000000`00000008

We therefore conclude without looking at disassembly that memory allocation size was 42944 bytes:

0:000> .formats 0000a7c0
Evaluate expression:
Hex: 00000000`0000a7c0
Decimal: 42944
Octal: 0000000000000000123700
Binary: 00000000 00000000 00000000 00000000 00000000 00000000 10100111 11000000
Chars: ........
Time: Thu Jan 01 11:55:44 1970
Float: low 6.01774e-041 high 0
Double: 2.12172e-319

Truncated Stack Trace

Sometimes we see this pattern with missing stack frames. For example, in one incident, after enabling user mode stack trace database for a memory leaking (Volume 1, page 356) application we got these entries from the growing heap segment (other segments had non-truncated saved stack traces):

0bdc1350: 40010 . 40010 [101] - busy (3fff8) Internal

7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
7702661f: ntdll!RtlAllocateHeap+0x0000017c

0be01360: 40010 . 40010 [101] - busy (3fff8) Internal

7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
7702661f: ntdll!RtlAllocateHeap+0x0000017c

0be41370: 40010 . 40010 [101] - busy (3fff8) Internal

7702fbd2: ntdll!RtlAllocateHeap+0x0000021d
77005eef: ntdll!RtlpAllocateUserBlock+0x000000a2
77026a65: ntdll!RtlpLowFragHeapAllocFromContext+0x00000785
7702661f: ntdll!RtlAllocateHeap+0x0000017c

Truncated traces are different from incorrect stack traces (Volume 1, page 288) because their surviving part is correct. How can we find the rest of such stack traces? Here we can suggest looking at other heap segments and see allocations of the same size. If a truncated trace comes from a stack trace collection (Volume 1, page 409) we can compare it with a non-truncated thread stack from another process instance having the same thread position.

Least Common Frame

Sometimes simple comparison of crash signatures (page 37) is not enough to find similar support incidents. We then traverse stack trace frames to find a least common frame matching similar stack traces in a database. For example, consider this signature:

0:026> r
eax=011349ec ebx=01136738 ecx=79f943e1 edx=00000000 esi=011349ec edi=0888f3b8
eip=00dfbef8 esp=0888f348 ebp=0888f3c8 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
00dfbef8 3902 cmp dword ptr [edx],eax ds:0023:00000000=????????

0:026> k 100
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0888f3c8 792a842f 0xdfbef8
0888f3e4 792a839b mscorlib_ni+0x1e842f
0888f3fc 79e71b4c mscorlib_ni+0x1e839b
0888f40c 79e821b9 mscorwks!CallDescrWorker+0×33
0888f48c 79e8281f mscorwks!CallDescrWorkerWithHandler+0xa3
0888f4ac 79e82860 mscorwks!DispatchCallBody+0×1e
0888f510 79e828d1 mscorwks!DispatchCallDebuggerWrapper+0×3d
0888f544 79ec50f5 mscorwks!DispatchCallNoEH+0×51
0888f5a0 79e9848f mscorwks!AddTimerCallback_Worker+0×66
0888f5b4 79e9842b mscorwks!Thread::DoADCallBack+0×32a
0888f648 79e98351 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0888f684 79e984dd mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
0888f6ac 79ec4a84 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
0888f6c4 79ec5075 mscorwks!ManagedThreadBase::ThreadPool+0×13
0888f70c 79ec50a4 mscorwks!AddTimerCallbackEx+0×83
0888f720 79ec514a mscorwks!AddTimerCallback+0×10
0888f75c 79ec4e0c mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0×64
0888f7a8 79ec471e mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0×9a
0888f7bc 79ec4892 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
0888f814 79f75715 mscorwks!ThreadpoolMgr::WorkerThreadStart+0×20b
0888ffb4 7c80b729 mscorwks!Thread::intermediateThreadProc+0×49
0888ffec 00000000 kernel32!BaseThreadStart+0×37

Most likely we won't find any similar stack trace when searching for 0xdfbef8. The search for mscorlib_ni+0×1e842f brings several results but they are not crashes but hangs with the frame in the middle of a call stack. The same is for mscorlib_ni+0×1e839b. So we finally try searching a problem database for CallDescrWorker+0×33 but limit results to stack traces having the same application module name. And indeed we find the similar software incident with the same stack trace after our least common frame:

0:004> r
eax=00000024 ebx=03e6f738 ecx=738129d8 edx=00495ef0 esi=01a87c4c edi=019c5f1c
eip=00a92037 esp=03e6f6cc ebp=03e6f6e8 iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
00a92037 ?? ???
0:004> k 100
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
03e6f6c8 737d6bb5 0xa92037
03e6f6e8 737a509f mscorlib_ni+0x216bb5
03e6f6f8 737a834c mscorlib_ni+0x1e509f
03e6f70c 74171b6c mscorlib_ni+0x1e834c
03e6f71c 74182209 mscorwks!CallDescrWorker+0×33
03e6f79c 7418286f mscorwks!CallDescrWorkerWithHandler+0xa3
03e6f7bc 741828b0 mscorwks!DispatchCallBody+0×1e
03e6f820 74182921 mscorwks!DispatchCallDebuggerWrapper+0×3d
03e6f854 742ced79 mscorwks!DispatchCallNoEH+0×51
03e6f8b0 7419846f mscorwks!AddTimerCallback_Worker+0×66
03e6f8c4 7419840b mscorwks!Thread::DoADCallBack+0×32a
03e6f958 74198331 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
03e6f994 741984bd mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
03e6f9bc 742ce708 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
03e6f9d4 742cecf9 mscorwks!ManagedThreadBase::ThreadPool+0×13
03e6fa1c 742ced28 mscorwks!AddTimerCallbackEx+0×83
03e6fa30 742cedce mscorwks!AddTimerCallback+0×10
03e6fa6c 742cea90 mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0×64
03e6fab8 742ce3a2 mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0×9a
03e6facc 742ce516 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
03e6fb64 74441ec9 mscorwks!ThreadpoolMgr::WorkerThreadStart+0×20b
03e6fc84 76813677 mscorwks!Thread::intermediateThreadProc+0×49
03e6fc90 77219d72 kernel32!BaseThreadInitThunk+0xe
03e6fcd0 77219d45 ntdll!__RtlUserThreadStart+0×70
03e6fce8 00000000 ntdll!_RtlUserThreadStart+0×1b

Self-Diagnosis (Kernel Mode)

This pattern is a kernel mode counterpart to Self-Diagnosis in user mode (Volume 2, page 318). It is just a collection of bugcheck codes where a problem is usually detected before corruption causes a fault, exception or trap. Typical example would be a detection of a failed assertion or corrupt structures such as:

BAD_POOL_HEADER (19)
The pool is already corrupt at the time of the current request.
This may or may not be due to the caller.
The internal pool links must be walked to figure out a possible cause of
the problem, and then special pool applied to the suspect tags or the
driver verifier to a suspect driver.
Arguments:
Arg1: 00000020, a pool block header size is corrupt.
Arg2: 8b79d078, The pool entry we were looking for within the page.
Arg3: 8b79d158, The next pool entry.
Arg4: 8a1c0004, (reserved)

Technology-Specific Subtrace (Dynamic Memory)

Here we continue with Technology-Specific Subtrace pattern series started earlier with COM interface invocation example (page 67). We consider dynamic memory allocation example in kernel space (kernel pool). Usually pool corruption is detected during pool memory allocation or release with a special bugcheck code, for example:

BAD_POOL_HEADER (19)
The pool is already corrupt at the time of the current request.
This may or may not be due to the caller.
The internal pool links must be walked to figure out a possible cause of
the problem, and then special pool applied to the suspect tags or the
driver verifier to a suspect driver.
Arguments:
Arg1: 00000020, a pool block header size is corrupt.
Arg2: 8b79d078, The pool entry we were looking for within the page.
Arg3: 8b79d158, The next pool entry.
Arg4: 8a1c0004, (reserved)

However, pool corruption might be deeper enough to trigger an access violation even before self-diagnosis. In such cases stack subtraces with functions like ExFreePoolWithTag might point to troubleshooting and debugging directions:

ATTEMPTED_WRITE_TO_READONLY_MEMORY (be)
An attempt was made to write to readonly memory. The guilty driver is on
the stack trace (and is typically the current instruction pointer).
When possible, the guilty driver's name (Unicode string) is printed on the
bugcheck screen and saved in KiBugCheckDriver.
Arguments:
Arg1: 00470044, Virtual address for the attempted write.
Arg2: 06d39025, PTE contents.
Arg3: aec0fb30, (reserved)
Arg4: 0000000a, (reserved)

TRAP_FRAME: aec0fb30 -- (.trap 0xffffffffaec0fb30)
ErrCode = 00000003
eax=8ac12d38 ebx=8b700040 ecx=000001ff edx=00470040 esi=8ac12db8
edi=808b0b40
eip=808949e7 esp=aec0fba4 ebp=aec0fbf0 iopl=0 nv up ei pl nz na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010202
nt!ExFreePoolWithTag+0x6a3:
808949e7 895a04 mov dword ptr [edx+4],ebx ds:0023:00470044=????????
STACK_TEXT:
aec0faa0 80860121 000000be 00470044 06d39025 nt!KeBugCheckEx+0x1b
aec0fb18 8088e490 00000001 00470044 00000000 nt!MmAccessFault+0xb25
aec0fb18 808949e7 00000001 00470044 00000000 nt!KiTrap0E+0xdc
aec0fbf0 808d93b5 8ac12dc0 00000000 00000000 nt!ExFreePoolWithTag+0×6a3
aec0fc08 808cd304 e5ae5770 8ac12dc0 8aa77db0 nt!CmpFreePostBlock+0×4d
aec0fc3c 8082ea53 8ac12dc0 aec0fc88 aec0fc7c nt!CmpPostApc+0xde
aec0fc8c 80833eec 00000000 00000000 00000000 nt!KiDeliverApc+0xf9
aec0fcc4 808290bd aec0fd64 8099781c 0160fd44 nt!KiSwapThread+0×300
aec0fd0c 809978a0 00000001 00000000 f77275e0
nt!KeDelayExecutionThread+0×2ab
aec0fd54 8088b45c 00000000 0160fd74 0160fd9c nt!NtDelayExecution+0×84
aec0fd54 7c82847c 00000000 0160fd74 0160fd9c nt!KiFastCallEntry+0xfc
WARNING: Frame IP not in any known module. Following frames may be wrong.
0160fd9c 00000000 00000000 00000000 00000000 0×7c82847c

1: kd> !pool 8ac12dc0
Pool page 8ac12dc0 region is Nonpaged pool
8ac12000 size: 858 previous size: 0 (Allocated) TWPG
8ac12858 size: 8 previous size: 858 (Free) ....
8ac12860 size: 20 previous size: 8 (Allocated) VadS
8ac12880 size: 8 previous size: 20 (Free) NtFs
8ac12888 size: 20 previous size: 8 (Allocated) VadS
8ac128a8 size: 28 previous size: 20 (Allocated) Ntfn
8ac128d0 size: 30 previous size: 28 (Allocated) Vad
8ac12900 size: 40 previous size: 30 (Allocated) Muta (Protected)
8ac12940 size: 38 previous size: 40 (Allocated) Sema (Protected)
8ac12978 size: 40 previous size: 38 (Allocated) Muta (Protected)
8ac129b8 size: 270 previous size: 40 (Allocated) Thre (Protected)
8ac12c28 size: 40 previous size: 270 (Allocated) Ntfr
8ac12c68 size: d0 previous size: 40 (Allocated) DRIV
8ac12d38 is not a valid large pool allocation, checking large session
pool...
8ac12d38 is freed (or corrupt) pool
Bad previous allocation size @8ac12d38, last size was 1a

***
*** An error (or corruption) in the pool was detected;
*** Attempting to diagnose the problem.
***
*** Use !poolval 8ac12000 for more details.
***

Pool page [ 8ac12000 ] is __inVALID.

Analyzing linked list...
[ 8ac12c68 --> 8ac12db8 (size = 0x150 bytes)]: Corrupt region
Scanning for single bit errors...

None found

Module Hint

This pattern is frequently observed in dynamic memory corruption¹¹ incidents. It is similar to Execution Residue (Volume 2, page 239) or String Parameter (page 49) patterns were we have ASCII or UNICODE fragments providing troubleshooting and debugging hints. Module Hint is therefore a more specialized pattern where we can link module names to raw data. For example, a kernel memory dump saved after the detected pool corruption (Volume 2, page 204) shows P12345.DLL module name in a pool entry that can provide a link to the corresponding functionally to be reconfigured or removed:

BAD_POOL_HEADER (19)
The pool is already corrupt at the time of the current request.
This may or may not be due to the caller.
The internal pool links must be walked to figure out a possible cause of
the problem, and then special pool applied to the suspect tags or the
driver verifier to a suspect driver.
Arguments:
Arg1: 00000020, a pool block header size is corrupt.
Arg2: 8b79d078, The pool entry we were looking for within the page.
Arg3: 8b79d158, The next pool entry.
Arg4: 8a1c0004, (reserved)

STACK_TEXT:
b3e0aa4c 808947bb 00000019 00000020 8b79d078 nt!KeBugCheckEx+0x1b
b3e0aab4 b368c00f 8b79d080 00000000 00000000 nt!ExFreePoolWithTag+0×477
b3e0aac4 b366c68e 8b79d080 00000000 00000000 DriverA!MemFree+0xf
[...]
b3e0ac44 8081e0c3 808f77c9 b3e0ac64 808f77c9 nt!IovCallDriver+0×112
b3e0ac50 808f77c9 8a8eef60 8b6862a8 8a8eeef0 nt!IofCallDriver+0×13
b3e0ac64 808f856b 8ce456b0 8a8eeef0 8b6862a8
nt!IopSynchronousServiceTail+0×10b
b3e0ad00 808f109a 000009dc 00000000 00000000 nt!IopXxxControlFile+0×5e5
b3e0ad34 8088b45c 000009dc 00000000 00000000 nt!NtDeviceIoControlFile+0×2a
b3e0ad34 7c82847c 000009dc 00000000 00000000 nt!KiFastCallEntry+0xfc
WARNING: Frame IP not in any known module. Following frames may be wrong.
00f5fb18 00000000 00000000 00000000 00000000 0×7c82847c
2: kd> !pool 8b79d080
Pool page 8b79d080 region is Unknown
8b79d000 size: 30 previous size: 0 (Allocated) FSfm
8b79d030 size: 28 previous size: 30 (Allocated) VadS
8b79d058 size: 20 previous size: 28 (Allocated) ReEv
*8b79d078 size: e0 previous size: 20 (Allocated) *DRIV
Owning component : Unknown (update pooltag.txt)
8b79d158 is not a valid large pool allocation, checking large session
pool...
8b79d158 is freed (or corrupt) pool
Bad previous allocation size @8b79d158, last size was 1c

***
*** An error (or corruption) in the pool was detected;
*** Pool Region unknown (0xFFFFFFFF8B79D158)
***
*** Use !poolval 8b79d000 for more details.
***

2: kd> dc 8b79d078
8b79d078 [...] ..DRIV ......AP
8b79d088 [...] P12345.DLL......
8b79d098 [...] .....<%n........
8b79d0a8 [...] ....$...:.F...X.
[...]

Custom Exception Handler (Kernel Space)

This is a kernel space counterpart to Custom Exception Handler pattern in user space (Volume 1, page 471). In the following stack trace below we see that DriverA code intercepted an access violation exception resulted from dereferencing a NULL pointer (Volume 3, page 131) and generated a custom bugcheck:

kd> !analyze -v

[...]

EXCEPTION_RECORD: fffff8801c757158 -- (.exr 0xfffff8801c757158)
ExceptionAddress: fffff88003977de1 (DriverA!foo+0x0000000000000381)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000000070
Attempt to read from address 0000000000000070

TRAP_FRAME: fffff8801c757200 -- (.trap 0xfffff8801c757200)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=fffff8a00da3f3c0
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff88003977de1 rsp=fffff8801c757390 rbp=fffffa8009a853f0
r8=0000000000000000 r9=0000000000000000 r10=006800740020006e
r11=fffff8a00da3f3c6 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
DriverA!foo+0×381:
fffff880`03977de1 0fb74070 movzx eax,word ptr [rax+70h] ds:0703:0070=????
Resetting default scope

[...]

kd> kL 100
Child-SP RetAddr Call Site
fffff880`1c7560f8 fffff880`039498f7 nt!KeBugCheckEx
fffff880`1c756100 fffff880`039352a0 DriverA!MyBugCheckEx+0×93
fffff880`1c756140 fffff800`016f1d1c DriverA!MyExceptionFilter+0×1d0
fffff880`1c756210 fffff800`016e940d nt!_C_specific_handler+0×8c
fffff880`1c756280 fffff800`016f0a90 nt!RtlpExecuteHandlerForException+0xd
fffff880`1c7562b0 fffff800`016fd9ef nt!RtlDispatchException+0×410
fffff880`1c756990 fffff800`016c2d82 nt!KiDispatchException+0×16f
fffff880`1c757020 fffff800`016c18fa nt!KiExceptionDispatch+0xc2
fffff880`1c757200 fffff880`03977de1 nt!KiPageFault+0×23a
fffff880`1c757390 fffff880`03977754 DriverA!foo+0×381
fffff880`1c757430 fffff880`0396f006 DriverA!bar+0×74
[...]
fffff880`1c7579b0 fffff800`019a6e0a DriverA!QueryInformation+0×30b
fffff880`1c757a70 fffff800`016c2993 nt!NtQueryInformationFile+0×535
fffff880`1c757bb0 00000000`76e5fe6a nt!KiSystemServiceCopyEnd+0×13
00000000`0a08dfe8 00000000`00000000 0×76e5fe6a

kd> !exchain
24 stack frames, scanning for handlers...
Frame 0×05: nt!RtlpExecuteHandlerForException+0xd (fffff800`016e940d)
ehandler nt!RtlpExceptionHandler (fffff800`016e93d0)
Frame 0×07: nt!KiDispatchException+0×16f (fffff800`016fd9ef)
ehandler nt!_GSHandlerCheck_SEH (fffff800`0169aec0)
Frame 0×0b: DriverA!bar+0×74 (fffff880`03977754)
ehandler DriverA!__GSHandlerCheck (fffff880`039a12fc)
[...]
Frame 0×14: DriverA!QueryInformation+0×30b (fffff880`039303ab)
ehandler DriverA!_C_specific_handler (fffff880`039a1864)
Frame 0×15: nt!NtQueryInformationFile+0×535 (fffff800`019a6e0a)
ehandler nt!_C_specific_handler (fffff800`016f1c90)
Frame 0×16: nt!KiSystemServiceCopyEnd+0×13 (fffff800`016c2993)
ehandler nt!KiSystemServiceHandler (fffff800`016c2580)
Frame 0×17: error getting module for 0000000076e5fe6a

No Data Types

Sometimes we don't have symbols (No Component Symbols pattern, Volume 1, page 298) or have only a restricted set. For example, in a base OS we have data types:

0:016> dt ntdll!*
ntdll!LIST_ENTRY64
ntdll!LIST_ENTRY32
ntdll!_KUSER_SHARED_DATA
ntdll!_KSYSTEM_TIME
ntdll!_KSYSTEM_TIME
ntdll!_NT_PRODUCT_TYPE
[...]

In the “private” version we don't have them although the symbol file exists:

0:015> dt ntdll!*
0:015> !lmi ntdll
Loaded Module Info:    [ntdll]
Module:                ntdll
Base Address:          0000000076de0000
Image Name:            ntdll.dll
Machine Type:          34404 (X64)
Time Stamp:            4dcd9861 Fri May 13 21:45:21 2011
Size:                  17f000
CheckSum:              188814
Characteristics:       2022 perf
Debug Data Dirs:       Type Size VA Pointer
CODEVIEW 22, f72a8, f66a8 RSDS - GUID: {05A648A7-625D-42E7-B736-
7816F0CA1E0C}
Age: 2, Pdb:          ntdll.pdb
CLSID 8, f72a0, f66a0 [Data not mapped]
Image Type: MEMORY - Image read successfully from loaded memory.
Symbol Type: PDB - Symbols loaded successfully from symbol server.
c:mss
tdll.pdb5A648A7625D42E7B7367816F0CA1E0C2
tdll.pdb
Load Report: public symbols , not source indexed
c:mss
tdll.pdb5A648A7625D42E7B7367816F0CA1E0C2
tdll.pdb

In such cases manually loading a proximate module might help (Volume 1, page 199) although we haven't yet tested it on x64 systems). We also thought of naming the pattern as Private Modification but that would not cover many other cases where types were missing from the very beginning.

Cloud Environment

This pattern is specific to cloud platform. It covers both development (emulator, if it exists) and real (staging and deployment) environments and is best diagnosed by looking at specific infrastructure modules:

0:016> lm m Wa*
start end module name
00000000`00b00000 00000000`00b0c000 WaWorkerHost
00000000`74fb0000 00000000`74fbd000 WaRuntimeProxy

0:016> lm m *Azure*
start end module name
00000000`57cd0000 00000000`57d26000 Microsoft_WindowsAzure_StorageClient
00000000`58820000 00000000`5886c000 Microsoft_WindowsAzure_Diagnostics
00000000`5c750000 00000000`5c764000 Microsoft_WindowsAzure_ServiceRuntime

Development platform can be distinguished by looking at versions of system modules such as ntdll:

0:016> lmv m ntdll
start             end               module name
00000000`76de0000 00000000`76f5f000 ntdll
Loaded symbol image file:     ntdll.dll
Image path:                   D:WindowsSystem32
tdll.dll
Image name:                   ntdll.dll
Timestamp:                    Fri May 13 21:45:21 2011 (4DCD9861)
CheckSum:                     00188814
ImageSize:                    0017F000
File version:                 6.0.6002.18446
Product version:              6.0.6002.18446
File flags:                   0 (Mask 3F)
File OS:                      40004 NT Win32
File type:                    2.0 Dll
File date:                    00000000.00000000
Translations:                 0409.04b0
CompanyName:                  Microsoft Corporation
ProductName:                  Microsoft® Windows® Operating System
InternalName:                 ntdll.dll
OriginalFilename:             ntdll.dll
ProductVersion:               6.0.6002.18446
FileVersion:                  6.0.6002.18446 (rd_os_v1.110513-1321)
FileDescription:              NT Layer DLL
LegalCopyright:               © Microsoft Corporation. All rights reserved.
0:016> lmv m ntdll
start             end               module name
00000000`775a0000 00000000`7774b000 ntdll
Loaded symbol image file:        ntdll.dll
Image path:                      C:WindowsSystem32
tdll.dll
Image name:                      ntdll.dll
Timestamp:                       Tue Jul 14 02:32:27 2009 (4A5BE02B)
CheckSum:                        001B1CB5
ImageSize:                       001AB000
File version:                    6.1.7600.16385
Product version:                 6.1.7600.16385
File flags:                      0 (Mask 3F)
File OS:                         40004 NT Win32
File type:                       2.0 Dll
File date:                       00000000.00000000
Translations:                    0409.04b0
CompanyName:                     Microsoft Corporation
ProductName:                     Microsoft® Windows® Operating System
InternalName:                    ntdll.dll
OriginalFilename:                ntdll.dll
ProductVersion:                  6.1.7600.16385
FileVersion:                     6.1.7600.16385 (win7_rtm.090713-1255)
FileDescription:                 NT Layer DLL
LegalCopyright:                  © Microsoft Corporation. All rights reserved.

Version-Specific Extension

This is a pattern similar to Platform-Specific Debugger pattern (Volume 4, page 156) by suggesting the course of the further debugging actions. Similar instructions are given when a debugger depends on specialized modules differing from platform (or application) version. We consider here a .NET example where opening a dump shows only that it was perhaps saved manually (Volume 1, page 487) with possible hidden exceptions (Volume 1, page 271) that need to be dug out:

0:000> !analyze -v

FAULTING_IP:
+0
00000000`00000000 ?? ???

EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0

We notice a failed attempt for .NET analysis and the following instructions on how to correct it:

MANAGED_STACK: !dumpstack -EE
Failed to load data access DLL, 0×80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file. For
example, an IA64 dump file must be debugged on an IA64 machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload. If
that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your
executable path is pointing to mscorwks.dll as well.

Because we know that we have .NET framework installed on a postmortem debugging machine we check the target module version:

0:000> lmv m mscorwks
start end module name
000007fe`ee380000 000007fe`eed1d000 mscorwks (pdb symbols)
Loaded symbol image file: mscorwks.dll
Image path: C:WindowsMicrosoft.NETFramework64v2.0.50727mscorwks.dll
Image name: mscorwks.dll
Timestamp: Sun Feb 06 20:53:54 2011 (4D4F0A62)
CheckSum: 00990593
ImageSize: 0099D000
File version: 2.0.50727.5444
Product version: 2.0.50727.5444
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: mscorwks.dll
OriginalFilename: mscorwks.dll
ProductVersion: 2.0.50727.5444
FileVersion: 2.0.50727.5444 (Win7SP1GDR.050727-5400)
FileDescription: Microsoft .NET Runtime Common Language Runtime -
WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail

It is slightly newer (.5444) than we have installed (.3619). The customer also sent their framework version together with the memory dump file. So we unload the current SOS extension (for details please see Managed Code Exception pattern, Volume 1, page 331):

0:000> .chain
Extension DLL chain:
C:WindowsMicrosoft.NETFramework64v2.0.50727sos: image 2.0.50727.3619,
API 1.0.0, built Mon Oct 25 06:52:09 2010
[path: C:WindowsMicrosoft.NETFramework64v2.0.50727sos.dll]
dbghelp: image 6.11.0001.404, API 6.1.6, built Thu Feb 26 02:10:27 2009
[path: C:Program FilesDebugging Tools for Windows (x64)dbghelp.dll]
ext: image 6.11.0001.404, API 1.0.0, built Thu Feb 26 02:10:26 2009
[path: C:Program FilesDebugging Tools for Windows (x64)winextext.dll]
exts: image 6.11.0001.404, API 1.0.0, built Thu Feb 26 02:10:17 2009
[path: C:Program FilesDebugging Tools for Windows (x64)WINXPexts.dll]
uext: image 6.11.0001.404, API 1.0.0, built Thu Feb 26 02:10:20 2009
[path: C:Program FilesDebugging Tools for Windows (x64)winextuext.dll]
ntsdexts: image 6.1.7015.0, API 1.0.0, built Thu Feb 26 02:09:22 2009
[path: C:Program FilesDebugging Tools for Windows
(x64)WINXP
tsdexts.dll]

0:000> .unload C:WindowsMicrosoft.NETFramework64v2.0.50727sos
Unloading C:WindowsMicrosoft.NETFramework64v2.0.50727sos extension
DLL

and load the customer version:

0:000> .load MyDatasos.dll

0:000> .chain
Extension DLL chain:
MyDatasos.dll: image 2.0.50727.5444, API 1.0.0, built Sun Feb 06 21:14:12
2011
[path: MyDatasos.dll]
dbghelp: image 6.11.0001.404, API 6.1.6, built Thu Feb 26 02:10:27 2009
[path: C:Program FilesDebugging Tools for Windows (x64)dbghelp.dll]
ext: image 6.11.0001.404, API 1.0.0, built Thu Feb 26 02:10:26 2009
[path: C:Program FilesDebugging Tools for Windows (x64)winextext.dll]
exts: image 6.11.0001.404, API 1.0.0, built Thu Feb 26 02:10:17 2009
[path: C:Program FilesDebugging Tools for Windows (x64)WINXPexts.dll]
uext: image 6.11.0001.404, API 1.0.0, built Thu Feb 26 02:10:20 2009
[path: C:Program FilesDebugging Tools for Windows (x64)winextuext.dll]
ntsdexts: image 6.1.7015.0, API 1.0.0, built Thu Feb 26 02:09:22 2009
[path: C:Program FilesDebugging Tools for Windows
(x64)WINXP
tsdexts.dll]

0:000> .cordll -ve -u -l
CLR DLL status: No load attempts

Then we do a load attempt:

0:000> !CLRStack
CLRDLL:
C:WindowsMicrosoft.NETFramework64v2.0.50727mscordacwks.dll:2.0.50727.
3619 f:0
doesn't match desired version 2.0.50727.5444 f:0
CLRDLL: Unable to find mscordacwks_AMD64_AMD64_2.0.50727.5444.dll by
mscorwks search
CLRDLL: Unable to find ‘mscordacwks_AMD64_AMD64_2.0.50727.5444.dll’ on the
path
CLRDLL: Unable to get version info for
‘c: mssmscorwks.dll4D4F0A6299d000mscordacwks_AMD64_AMD64_2.0.50727.5444
.dll’, Win32 error 0n87
CLRDLL: ERROR: Unable to load DLL
mscordacwks_AMD64_AMD64_2.0.50727.5444.dll, Win32 error 0n87
Failed to load data access DLL, 0×80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file. For
example, an IA64 dump file must be debugged on an IA64 machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload. If
that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your
executable path is pointing to mscorwks.dll as well.

Following instructions we rename mscordacwks.dll to
mscordacwks_AMD64_AMD64_2.0.50727.5444.dll and retry:

0:000> .cordll -ve -u -l
CLR DLL status: No load attempts

0:000> !CLRStack
CLRDLL:
C:WindowsMicrosoft.NETFramework64v2.0.50727mscordacwks.dll:2.0.50727.3619 f:0
doesn't match desired version 2.0.50727.5444 f:0
CLRDLL: Loaded DLL MyDatamscordacwks_AMD64_AMD64_2.0.50727.5444.dll
OS Thread Id: 0×16e8 (0)
Child-SP RetAddr Call Site
00000000002fe570 000007feeaf8e378
System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.
UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
00000000002fe7c0 000007feeaf8dde5
System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32,
System.Windows.Forms.ApplicationContext)
00000000002fe910 000007ff002364b6
System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32,
System.Windows.Forms.ApplicationContext)
00000000002fe970 000007feee6414c2 MyApplication.Main(System.String[])

0:000> !pe
Exception object: 00000000034a13f8
Exception type: System.IO.FileNotFoundException
Message: Could not load file or assembly ‘System.Windows.Forms.XmlSerializers,
Version=2.0.0.0, Culture=neutral, PublicKeyToken= ...’ or one of its dependencies.
The system cannot find the file specified.
InnerException: System.IO.FileNotFoundException, use !PrintException 00000000034a1b28
to see more
StackTrace (generated):
SP IP Function
00000000002FD0A0 0000000000000001
mscorlib_ni!System.Reflection.Assembly._nLoad(System.Reflection.AssemblyName,
System.String, System.Security.Policy.Evidence, System.Reflection.Assembly,
System.Threading.StackCrawlMark ByRef, Boolean, Boolean)+0x2
00000000002FD0A0 000007FEED7ABF61
mscorlib_ni!System.Reflection.Assembly.InternalLoad(System.Reflection.AssemblyName,
System.Security.Policy.Evidence, System.Threading.StackCrawlMark ByRef,
Boolean)+0x1a1
00000000002FD130 000007FEED7E4804
mscorlib_ni!System.Reflection.Assembly.Load(System.Reflection.AssemblyName)+0x24
00000000002FD170 000007FEE7855C0A
System_Xml_ni!System.Xml.Serialization.TempAssembly.LoadGeneratedAssembly(System.Type
, System.String, System.Xml.Serialization.XmlSerializerImplementation ByRef)+0x11a

StackTraceString: <none>
HResult: 80070002
0:000> !PrintException 00000000034a1b28
Exception object: 00000000034a1b28
Exception type: System.IO.FileNotFoundException
Message: Could not load file or assembly
‘System.Windows.Forms.XmlSerializers, Version=2.0.0.0, Culture=neutral,
PublicKeyToken=...’ or one of its dependencies. The system cannot find the
file specified.
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80070002

Multiple Exceptions (Managed Space)

In addition to multiple exceptions is user mode / space (Volume 1, page 255) and kernel mode / space (Volume 3, page 78) patterns we can have multiple exceptions in “managed space”. After SOS extension is loaded we can use the following commands to list such exceptions (some output was skipped for formatting clarity):

0:000> !Threads
[...]
   ID OSID Exception
0  1  12c  System.IO.FileNotFoundException (0000000003bd6230)
8  2  e24  (Finalizer)
10 3  c1c  System.Reflection.TargetInvocationException (000000000492a388)
11 4  cb0  (Threadpool Completion Port)
12 5  c10
13 6  1e8  (Threadpool Completion Port)
15 7  c14  (Threadpool Worker)
16 8  edc  (Threadpool Worker)
[...]
23 e 1084  System.NullReferenceException (000000000492a300)

0:000> ~*e !pe
Exception object: 0000000003bd6230
Exception type: System.IO.FileNotFoundException
Message: Could not load file or assembly [...]
InnerException: System.IO.FileNotFoundException, use !PrintException
0000000003bd6938 to see more
StackTrace (generated):
SP IP Function
[...]
Exception object: 000000000492a388
Exception type: System.Reflection.TargetInvocationException
Message: Exception has been thrown by the target of an invocation.
InnerException: System.NullReferenceException, use !PrintException
000000000492a300 to see more
StackTrace (generated):
SP IP Function
[...]

Blocking File

This pattern often happens (but not limited to) in roaming profile scenarios In addition to Blocked Thread (Volume 1, page 184), endpoint threads of Wait Chain patterns (Volume 1, page 482), and Blocking Module (page 54). For example, an application was reported hanging and in a complete memory dump we could see a thread in a stack trace collection (Volume 1, page 409):

THREAD fffffa8005eca060 Cid 14b0.1fec Teb: 000000007ef84000 Win32Thread:
fffff900c26c2c30 WAIT: (Executive) KernelMode Non-Alertable
fffffa80048e6758 NotificationEvent
IRP List:
fffffa8005a6c160: (0006,03e8) Flags: 00060000 Mdl: 00000000
Not impersonating
DeviceMap fffff8a0055b6620
Owning Process fffffa80063dd970 Image: Application.exe
Attached Process N/A Image: N/A
Wait Start TickCount 171988390 Ticks: 26963639 (4:21:01:46.859)
Context Switch Count 226 LargeStack
UserTime 00:00:00.015
KernelTime 00:00:00.015
Win32 Start Address 0×000000006d851f62
Stack Init fffff880075a9db0 Current fffff880075a9770
Base fffff880075aa000 Limit fffff880075a4000 Call 0
Priority 10 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2
PagePriority 5
Child-SP RetAddr Call Site
fffff880`075a97b0 fffff800`0167f752 nt!KiSwapContext+0×7a
fffff880`075a98f0 fffff800`016818af nt!KiCommitThreadWait+0×1d2
fffff880`075a9980 fffff800`019b612a nt!KeWaitForSingleObject+0×19f
fffff880`075a9a20 fffff800`0198feaa nt! ?? ::NNGAKEGL::‘string’+0×1d61a
fffff880`075a9a60 fffff800`018ed0e3 nt!IopSynchronousServiceTail+0×35a
fffff880`075a9ad0 fffff800`01677853 nt!NtLockFile+0×514
fffff880`075a9bb0 00000000`77840cea nt!KiSystemServiceCopyEnd+0×13
(TrapFrame @ fffff880`075a9c20)
00000000`0798e488 00000000`7543293b ntdll!ZwLockFile+0xa
00000000`0798e490 00000000`7541cf87 wow64!whNtLockFile+0×7f
00000000`0798e510 00000000`7536276d wow64!Wow64SystemServiceEx+0xd7
00000000`0798edd0 00000000`7541d07e
wow64cpu!TurboDispatchJumpAddressEnd+0×24
00000000`0798ee90 00000000`7541c549 wow64!RunCpuSimulation+0xa
00000000`0798eee0 00000000`7786d177 wow64!Wow64LdrpInitialize+0×429
00000000`0798f430 00000000`7782308e ntdll! ?? ::FNODOBFM::‘string’+0×2bfe4
00000000`0798f4a0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

We immediately spot the anomaly of a lock file attempt and look at its IRP:

0: kd> !irp fffffa8005a6c160
Irp is active with 7 stacks 7 is current (= 0xfffffa8005a6c3e0)
No Mdl: No System Buffer: Thread fffffa8005eca060: Irp stack trace.
cmd flg cl Device File Completion-Context
[ 0, 0] 0 2 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 ffffffffc000020c
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 11, 0] 0 2 fffffa8004da0620 00000000 fffff8800177d9cc-fffffa800710e580
FileSystemmrxsmb mup!MupiUncProviderCompletion
Args: 00000000 00000000 00000000 00000000
>[ 11, 1] 0 0 fffffa8004066400 fffffa80048e66c0 00000000-00000000
FileSystemMup
Args: fffffa8004a98120 00000001 00000000 00000000

From that IRP we see a file name:

0: kd> !fileobj fffffa80048e66c0

[...]AppDataRoamingVendorProductRecentindex.dat

LockOperation Set Device Object: 0xfffffa8004066400 FileSystemMup
Vpb is NULL
Access: Read SharedRead SharedWrite SharedDelete

Flags: 0x40002
Synchronous IO
Handle Created

File Object is currently busy and has 0 waiters.

FsContext: 0xfffff8a00e8d9010 FsContext2: 0xfffff8a012e4d688
CurrentByteOffset: 0
Cache Data:
Section Object Pointers: fffffa8006086928
Shared Cache Map: 00000000
File object extension is at fffffa8005c8cbe0:

Alternatively we get a 32-bit stack trace from the virtualized process (Volume 1, page 400):

0: kd> .process /r /p fffffa80063dd970
Implicit process is now fffffa80`063dd970
Loading User Symbols

0: kd> .thread /w fffffa8005eca060
Implicit thread is now fffffa80`05eca060
The context is partially valid. Only x86 user-mode context is available.
x86 context set

0: kd:x86> .reload
Loading Kernel Symbols
Loading User Symbols
Loading unloaded module list
Loading Wow64 Symbols

0: kd:x86> kv
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr Args to Child
07ac8510 774f033f 00000390 00000000 00000000 ntdll_779d0000!ZwLockFile+0×12
07ac8590 774f00d3 061b2b68 ada9964d c0000016 kernel32!BaseDllOpenIniFileOnDisk+0×246
07ac85d0 774efae9 061b2b68 00001000 6d352f20
kernel32!BaseDllReadWriteIniFileOnDisk+0×2d
07ac85e8 775001bf 00000001 00000000 061b2b68 kernel32!BaseDllReadWriteIniFile+0xed
07ac861c 6d928401 07aca71c 00000000 00001000 kernel32!GetPrivateProfileStringW+0×35
WARNING: Stack unwind information not available. Following frames may be wrong.
07ac8640 6d9282f5 07aca71c 00000000 00000000 DLL+0×618401
[...]
07acfb14 774e3677 06757d20 07acfb60 77a09d72 DLL+0×541f6d
07acfb20 77a09d72 06757d20 eca51e43 00000000 kernel32!BaseThreadInitThunk+0xe
07acfb60 77a09d45 6d851f62 06757d20 ffffffff ntdll_779d0000!__RtlUserThreadStart+0×70
07acfb78 00000000 6d851f62 06757d20 00000000 ntdll_779d0000!_RtlUserThreadStart+0×1b

We get the same file name from a file handle:

0: kd> !handle 00000390
processor number 0, process fffffa80063dd970
PROCESS fffffa80063dd970
SessionId: 5 Cid: 14b0 Peb: 7efdf000 ParentCid: 1fac
DirBase: 48293000 ObjectTable: fffff8a010515f90 HandleCount: 342.
Image: Application.exe

Handle table at fffff8a0083e9000 with 444 Entries in use
0390: Object: fffffa80048e66c0 GrantedAccess: 00120089 Entry:
fffff8a00866fe40
Object: fffffa80048e66c0 Type: (fffffa8003cf0b40) File
ObjectHeader: fffffa80048e6690 (new version)
HandleCount: 1 PointerCount: 3
Directory Object: 00000000 Name:
[...] AppDataRoamingVendorProductRecentindex.dat {Mup}

We also c0000016 error code on raw stack and examine it too:

0: kd> !error c0000016
Error code: (NTSTATUS) 0xc0000016 (3221225494) - {Still Busy} The
specified I/O request packet (IRP) cannot be disposed of because the I/O
operation is not complete.

Quiet Dump

How would we call a memory dump where we don't see anything abnormal or even suspicious? For example, in such a dump its stack trace collection (Volume 1, page 409) would not deviate from reference stack traces and we would not see any spiking thread (Volume 1, page 305).

Pleiades

This is a cluster of modules in lm WinDbg command output that serve similar function, like print drivers in print spooler or Citrix printing services. Usually we know that anyone of them could be at fault. Another example is a group of process modules in a complete memory dump serving the same function in separate terminal services sessions.

Thread Age

Sometimes, it is useful to know how much time ago a thread was created in order to understand when other behavioral patterns possibly started to appear. For example, in user process memory dumps with saved thread time information we can see the latter using !runaway WinDbg extension command or using .ttime command. Looking at a stack trace collection (Volume 1, page 409) we notice a thread blocked in LPC call (Volume 3, page 97):

0:000> ~40 kc
Call Site
ntdll!NtAlpcSendWaitReceivePort
rpcrt4!LRPC_CCALL::SendReceive
rpcrt4!NdrpClientCall3
rpcrt4!NdrClientCall3
[...]
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

We are interested when all this started because we want to compare with other coupled process (Volume 1, page 419) memory dumps saved at different times:

0:000> !runaway f
User Mode Time
[...]
Kernel Mode Time
Thread Time
[...]
Elapsed Time
Thread Time
0:8ac 4 days 11:42:14.484
1:8b4 4 days 11:42:14.296
[...]
35:868 4 days 10:18:48.255
36:73ec 0 days 15:55:31.938
37:c0bc 0 days 10:36:53.447
38:782c 0 days 0:02:01.683
39:5864 0 days 0:00:52.236
40:5ffc 0 days 0:00:02.565

0:000> ~40s
ntdll!NtAlpcSendWaitReceivePort+0xa:
00000000`76d3ff0a c3 ret

0:040> .ttime
Created: Tue Jun 14 15:15:28.444 2011
Kernel: 0 days 0:00:00.000
User: 0 days 0:00:00.000
0:040> .time
Debug session time: Tue Jun 14 15:15:31.000 2011
System Uptime: 4 days 11:43:21.389
Process Uptime: 4 days 11:42:15.000
Kernel time: 0 days 0:00:10.000
User time: 0 days 0:04:22.000

We see that our blocked thread had only recently started compared to other threads and actually started after other dumps were saved when we look at their debug session time.

Unsynchronized Dumps

For analysis of memory dumps from coupled processes (Volume 1, page 419) or, in general, memory fibers from fiber bundle memory spaces (Volume 4, page 357) we need to know their creation times (called debug session time). In some cases we need to know their time sequence: which process memory dump was saved first and how much time had passed before the second process memory dump was saved. Beside an initial output when we open a dump .time and version WinDbg commands can be used to check this information at any time during analysis.

In one example involving printing we see a blocking thread trying to contact a print spooler service using LPC. Its thread age (page 111) is no more than 3 seconds. We also have the print spooler service process memory dump supposedly taken at the same time. However, when we check it we see it was saved 2 minutes before. Moreover, PrintIsolationHost.exe process memory dump was saved even earlier. So the whole sequence was reversed because the printing application calls the spooler and it calls the appropriate driver, not the way around.

Coupled Modules

Often we identify a pattern that points to a particular module such as a driver or DLL other modules could use functional services from and, therefore, the latter modules can be implicated in abnormal software behavior. For example, detected insufficient kernel paged pool memory (Volume 1, page 441) pointed to a driver that owns a pool tag DRV:

1: kd> !poolused 4
Sorting by Paged Pool Consumed

Tag  Allocs  Frees   Diff  Used
DRV  1466496 1422361 44135 188917256 UNKNOWN pooltag ‘DRV ‘, please update
pooltag.txt
File 6334830 6284036 50794 6735720 File objects
Thre 53721 45152 8569 4346432 Thread objects , Binary: nt!ps
[...]

This module is known to be a directing module (page 80) to other drivers (from stack trace perspective) but we also know that other (directed) modules use its services that require memory allocation.

Managed Stack Trace

Typical examples of this pattern are stack traces from !CLRStack and !pe extension commands or subtraces from !DumpStack and !EEStack extension commands:

0:000> !pe
Exception object: 0000000005a976b8
Exception type: System.FormatException
Message: Index (zero based) must be greater than or equal to zero and less
than the size of the argument list.
InnerException: <none>
StackTrace (generated):
SP IP Function
0000000000D0BE40 000007FEEC2153B0
mscorlib_ni!System.Text.StringBuilder.AppendFormat(System.IFormatProvider,
System.String, System.Object[])+0×999280
0000000000D0BEE0 000007FEEB87C0FA
mscorlib_ni!System.String.Format(System.IFormatProvider, System.String,
System.Object[])+0×5a
0000000000D0BF30 000007FF00AB336B ModuleA!ClassB.get()+0xeb

0:010> !DumpStack
OS Thread Id: 0x8dc (15)
Child-SP RetAddr Call Site
000000001f69e808 00000000774b4bc4 user32!ZwUserWaitMessage+0xa
000000001f69e810 00000000774b4edd user32!DialogBox2+0x274
000000001f69e8a0 0000000077502920 user32!InternalDialogBox+0x135
000000001f69e900 0000000077501c15 user32!SoftModalMessageBox+0x9b4
000000001f69ea30 000000007750146b user32!MessageBoxWorker+0x31d
000000001f69ebf0 0000000077501362 user32!MessageBoxTimeoutW+0xb3
000000001f69ecc0 000007fef1590ce7 user32!MessageBoxW+0x4e
000000001f69ed00 000007feeb0f5c59
mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
[...]
000000001f69e030 000007ff00a9ba1c ModuleA!ClassA.foo()+0×47
[...]
000000001f69fe30 000000007781c521 kernel32!BaseThreadInitThunk+0xd
000000001f69fe60 0000000000000000 ntdll!RtlUserThreadStart+0×1d

Problem Vocabulary

One way to quickly check for something suspicious in a memory dump is to convert it to a debugger log (Volume 5, page 41, for example, stack trace collection, Volume 1, page 409) and search for textual strings such AS “Waiting for”, “Terminate”, “Stop”, “Mutant”, “Exception”, “Crit”, “MessageBox”, “SuspendCount”, etc. The vocabulary, of course, is OS dependent, can have false positives, and can change over time. This pattern is similar to Vocabulary Index (Volume 4, page 349) software trace analysis pattern.

For example, recently in a complete memory dump involving lots of ALPC wait chains with potential inter-process deadlock we found the following thread having long waiting time (Volume 1, page 343, exceeding ALPC threads waiting times) pointing to a process object to examine further:

THREAD fffffa801338b950 Cid 02a0.7498 Teb: 000007fffffd8000 Win32Thread:
0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
fffffa8012a39b30 ProcessObject
Not impersonating
DeviceMap fffff8a000008a70
Owning Process fffffa800a31d040 Image: smss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 9752080 Ticks: 5334204 (0:23:09:06.937)
Context Switch Count 38
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address ntdll!TppWorkerThread (0×000000007722fbc0)
Stack Init fffff88020259db0 Current fffff88020259900
Base fffff8802025a000 Limit fffff88020254000 Call 0
Priority 11 BasePriority 11 UnusualBoost 0 ForegroundBoost 0 IoPriority 2
PagePriority 5
Kernel stack not resident.
Child-SP RetAddr Call Site
fffff880`20259940 fffff800`01693f92 nt!KiSwapContext+0×7a
fffff880`20259a80 fffff800`016967af nt!KiCommitThreadWait+0×1d2
fffff880`20259b10 fffff800`01984b2e nt!KeWaitForSingleObject+0×19f
fffff880`20259bb0 fffff800`0168df93 nt!NtWaitForSingleObject+0xde
fffff880`20259c20 00000000`7726135a nt!KiSystemServiceCopyEnd+0×13
(TrapFrame @ fffff880`20259c20)
00000000`0048f648 00000000`48026517 ntdll!NtWaitForSingleObject+0xa
00000000`0048f650 00000000`480269c4 smss!SmpTerminateCSR+0xa3
00000000`0048f6a0 00000000`48023670 smss!SmpStopCsr+0×44
00000000`0048f6d0 00000000`77288137 smss!SmpApiCallback+0×338
00000000`0048f900 00000000`7722feff ntdll! ?? ::FNODOBFM::‘string’+0×1f718
00000000`0048f990 00000000`77274a00 ntdll!TppWorkerThread+0×3f8
00000000`0048fc90 00000000`00000000 ntdll!RtlUserThreadStart+0×25

In that process we could see a blocking module (page 54) and recommended to contact its vendor.

Activation Context

This is a new pattern about activation contexts¹². Here we have software exceptions STATUS_SXS_*, for example:

STATUS_SXS_EARLY_DEACTIVATION 0xC015000F
STATUS_SXS_INVALID_DEACTIVATION 0xC0150010

0:000> !analyze -v

[...]

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 77a54441 (ntdll!RtlDeactivateActivationContext+0x00000154)
ExceptionCode: c015000f
ExceptionFlags: 00000000
NumberParameters: 3
Parameter[0]: 00000000
Parameter[1]: 0056cbe8
Parameter[2]: 0056cc18


EXCEPTION_CODE: (NTSTATUS) 0xc015000f - The activation context being deactivated is
not the most recently activated one.

CONTEXT: 003df6c8 -- (.cxr 0x3df6c8)
eax=003df9bc ebx=13050002 ecx=00000000 edx=00000000 esi=0056cbe8 edi=0056cc18
eip=77a54441 esp=003df9b0 ebp=003dfa0c iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
ntdll!RtlDeactivateActivationContext+0x154:
77a54441 8b36 mov esi,dword ptr [esi] ds:002b:0056cbe8=0056cbb8
Resetting default scope

STACK_TEXT:
003dfa0c 755aa138 005507d0 13050002 003dfa7c
ntdll!RtlDeactivateActivationContext+0×154
003dfa1c 002b1235 00000000 13050002 3a92c68c kernel32!DeactivateActCtx+0×31
003dfa7c 002b13b5 00000001 01f01e98 01f01ec8 TestActCtx!wmain+0×225
003dfac4 75593677 7efde000 003dfb10 77a09f02 TestActCtx!__tmainCRTStartup+0xfa
003dfad0 77a09f02 7efde000 7e35c89d 00000000 kernel32!BaseThreadInitThunk+0xe
003dfb10 77a09ed5 002b140c 7efde000 ffffffff ntdll!__RtlUserThreadStart+0×70
003dfb28 00000000 002b140c 7efde000 00000000 ntdll!_RtlUserThreadStart+0×1b

The ReactOS code for RtlDeactivateActivationContext¹³ function suggests the following line of inquiry:

0:000> dt _TEB
TestActCtx!_TEB
+0x000 NtTib : _NT_TIB
+0x01c EnvironmentPointer : Ptr32 Void
+0x020 ClientId : _CLIENT_ID
+0x028 ActiveRpcHandle : Ptr32 Void
+0x02c ThreadLocalStoragePointer : Ptr32 Void
+0x030 ProcessEnvironmentBlock : Ptr32 _PEB
+0x034 LastErrorValue : Uint4B
+0x038 CountOfOwnedCriticalSections : Uint4B
+0x03c CsrClientThread : Ptr32 Void
+0x040 Win32ThreadInfo : Ptr32 Void
+0x044 User32Reserved : [26] Uint4B
+0x0ac UserReserved : [5] Uint4B
+0x0c0 WOW32Reserved : Ptr32 Void
+0x0c4 CurrentLocale : Uint4B
+0x0c8 FpSoftwareStatusRegister : Uint4B
+0x0cc SystemReserved1 : [54] Ptr32 Void
+0x1a4 ExceptionCode : Int4B
+0×1a8 ActivationContextStack : _ACTIVATION_CONTEXT_STACK
+0×1bc SpareBytes1 : [24] UChar
+0×1d4 GdiTebBatch : _GDI_TEB_BATCH
+0×6b4 RealClientId : _CLIENT_ID
+0×6bc GdiCachedProcessHandle : Ptr32 Void
+0×6c0 GdiClientPID : Uint4B
+0×6c4 GdiClientTID : Uint4B
+0×6c8 GdiThreadLocalInfo : Ptr32 Void
+0×6cc Win32ClientInfo : [62] Uint4B
+0×7c4 glDispatchTable : [233] Ptr32 Void
+0xb68 glReserved1 : [29] Uint4B
+0xbdc glReserved2 : Ptr32 Void
+0xbe0 glSectionInfo : Ptr32 Void
+0xbe4 glSection : Ptr32 Void
+0xbe8 glTable : Ptr32 Void
+0xbec glCurrentRC : Ptr32 Void
+0xbf0 glContext : Ptr32 Void
+0xbf4 LastStatusValue : Uint4B
+0xbf8 StaticUnicodeString : _UNICODE_STRING
+0xc00 StaticUnicodeBuffer : [261] Wchar
+0xe0c DeallocationStack : Ptr32 Void
+0xe10 TlsSlots : [64] Ptr32 Void
+0xf10 TlsLinks : _LIST_ENTRY
+0xf18 Vdm : Ptr32 Void
+0xf1c ReservedForNtRpc : Ptr32 Void
+0xf20 DbgSsReserved : [2] Ptr32 Void
+0xf28 HardErrorMode : Uint4B
+0xf2c Instrumentation : [16] Ptr32 Void
+0xf6c WinSockData : Ptr32 Void
+0xf70 GdiBatchCount : Uint4B
+0xf74 InDbgPrint : UChar
+0xf75 FreeStackOnTermination : UChar
+0xf76 HasFiberData : UChar
+0xf77 IdealProcessor : UChar
+0xf78 Spare3 : Uint4B
+0xf7c ReservedForPerf : Ptr32 Void
+0xf80 ReservedForOle : Ptr32 Void
+0xf84 WaitingOnLoaderLock : Uint4B
+0xf88 Wx86Thread : _Wx86ThreadState
+0xf94 TlsExpansionSlots : Ptr32 Ptr32 Void
+0xf98 ImpersonationLocale : Uint4B
+0xf9c IsImpersonating : Uint4B
+0xfa0 NlsCache : Ptr32 Void
+0xfa4 pShimData : Ptr32 Void
+0xfa8 HeapVirtualAffinity : Uint4B
+0xfac CurrentTransactionHandle : Ptr32 Void
+0xfb0 ActiveFrame : Ptr32 _TEB_ACTIVE_FRAME
+0xfb4 FlsData : Ptr32 Void

0:000> dt _ACTIVATION_CONTEXT_STACK
TestActCtx!_ACTIVATION_CONTEXT_STACK
+0x000 Flags : Uint4B
+0x004 NextCookieSequenceNumber : Uint4B
+0x008 ActiveFrame : Ptr32 _RTL_ACTIVATION_CONTEXT_STACK_FRAME
+0x00c FrameListCache : _LIST_ENTRY

0:000> dt _RTL_ACTIVATION_CONTEXT_STACK_FRAME
ntdll!_RTL_ACTIVATION_CONTEXT_STACK_FRAME
+0x000 Previous : Ptr32 _RTL_ACTIVATION_CONTEXT_STACK_FRAME
+0x004 ActivationContext : Ptr32 _ACTIVATION_CONTEXT
+0x008 Flags : Uint4B

0:000> dd 0056cc18 l4
0056cc18 0056cbe8 0056ca6c 00000028 13050003

0:000> dd 0056cbe8
0056cbe8 0056cbb8 0056c934 00000028 13050002
0056cbf8 00000000 00000000 00000000 00000000
0056cc08 00000000 00000000 00000000 00000000
0056cc18 0056cbe8 0056ca6c 00000028 13050003
0056cc28 00000000 00000000 00000000 00000000
0056cc38 00000000 00000000 00000000 00000000
0056cc48 00000000 00000000 0000000c 00000000
0056cc58 00000000 00000000 00000000 00000000

0:000> dd 0056cbb8
0056cbb8 00000000 0056c7fc 00000028 13050001
0056cbc8 00000000 00000000 00000000 00000000
0056cbd8 00000000 00000000 00000000 00000000
0056cbe8 0056cbb8 0056c934 00000028 13050002
0056cbf8 00000000 00000000 00000000 00000000
0056cc08 00000000 00000000 00000000 00000000
0056cc18 0056cbe8 0056ca6c 00000028 13050003
0056cc28 00000000 00000000 00000000 00000000

We see that a different cookie was found on top of the thread activation stack and the code raised the runtime exception.

Stack Trace Set

Whereas Stack Trace Collection (Volume 1, page 409) pattern covers all thread stack traces from a memory dump this pattern covers only unique non-duplicated thread stack traces differing for example, in stack frame modules and function names. In user process memory dumps it is !uniqstack WinDbg command (don't forget that command has optional parameters, for example,-v to simulate verbose ~*kv output):

0:000> ~
. 0 Id: f00.f04 Suspend: 0 Teb: 7efdd000 Unfrozen
1 Id: f00.f18 Suspend: 1 Teb: 7efda000 Unfrozen
2 Id: f00.f1c Suspend: 1 Teb: 7efd7000 Unfrozen

0:000> ~*kc

. 0 Id: f00.f04 Suspend: 0 Teb: 7efdd000 Unfrozen

ntdll!NtWaitForMultipleObjects
KERNELBASE!WaitForMultipleObjectsEx
kernel32!WaitForMultipleObjectsExImplementation
kernel32!WaitForMultipleObjects
kernel32!WerpReportFaultInternal
kernel32!WerpReportFault
kernel32!BasepReportFault
kernel32!UnhandledExceptionFilter
ntdll!__RtlUserThreadStart
ntdll!_EH4_CallFilterFunc
ntdll!_except_handler4
ntdll!ExecuteHandler2
ntdll!ExecuteHandler
ntdll!KiUserExceptionDispatcher
KERNELBASE!DebugBreak
ApplicationK!main
ApplicationK!__tmainCRTStartup
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

1 Id: f00.f18 Suspend: 1 Teb: 7efda000 Unfrozen

ntdll!NtDelayExecution
KERNELBASE!SleepEx
KERNELBASE!Sleep
kernel32!WerpReportFault
kernel32!BasepReportFault
kernel32!UnhandledExceptionFilter
ntdll!__RtlUserThreadStart
ntdll!_EH4_CallFilterFunc
ntdll!_except_handler4
ntdll!ExecuteHandler2
ntdll!ExecuteHandler
ntdll!KiUserExceptionDispatcher
ApplicationK!thread_two
ApplicationK!_callthreadstart
ApplicationK!_threadstart
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

2 Id: f00.f1c Suspend: 1 Teb: 7efd7000 Unfrozen

ntdll!NtDelayExecution
KERNELBASE!SleepEx
KERNELBASE!Sleep
kernel32!WerpReportFault
kernel32!BasepReportFault
kernel32!UnhandledExceptionFilter
ntdll!__RtlUserThreadStart
ntdll!_EH4_CallFilterFunc
ntdll!_except_handler4
ntdll!ExecuteHandler2
ntdll!ExecuteHandler
ntdll!KiUserExceptionDispatcher
ApplicationK!thread_two
ApplicationK!_callthreadstart
ApplicationK!_threadstart
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

0:000> !uniqstack
Processing 3 threads, please wait

. 0 Id: f00.f04 Suspend: 0 Teb: 7efdd000 Unfrozen
Start: ApplicationK!mainCRTStartup (013a137c)
Priority: 0 Priority class: 32 Affinity: 3
ChildEBP RetAddr
0037f1a4 770d0bdd ntdll!NtWaitForMultipleObjects+0x15
0037f240 7529162d KERNELBASE!WaitForMultipleObjectsEx+0x100
0037f288 75291921 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0037f2a4 752b9b2d kernel32!WaitForMultipleObjects+0x18
0037f310 752b9bca kernel32!WerpReportFaultInternal+0x186
0037f324 752b98f8 kernel32!WerpReportFault+0x70
0037f334 752b9875 kernel32!BasepReportFault+0x20
0037f3c0 77b10df7 kernel32!UnhandledExceptionFilter+0x1af
0037f3c8 77b10cd4 ntdll!__RtlUserThreadStart+0x62
0037f3dc 77b10b71 ntdll!_EH4_CallFilterFunc+0x12
0037f404 77ae6ac9 ntdll!_except_handler4+0x8e
0037f428 77ae6a9b ntdll!ExecuteHandler2+0x26
0037f4d8 77ab010f ntdll!ExecuteHandler+0x24
0037f4d8 770d280c ntdll!KiUserExceptionDispatcher+0xf
0037f824 013a1035 KERNELBASE!DebugBreak+0x2
0037f828 013a1325 ApplicationK!main+0x25
0037f870 75293677 ApplicationK!__tmainCRTStartup+0xfb
0037f87c 77ad9f02 kernel32!BaseThreadInitThunk+0xe
0037f8bc 77ad9ed5 ntdll!__RtlUserThreadStart+0x70
0037f8d4 00000000 ntdll!_RtlUserThreadStart+0x1b

. 1 Id: f00.f18 Suspend: 1 Teb: 7efda000 Unfrozen
Start: ApplicationK!_threadstart (013a10d1)
Priority: 0 Priority class: 32 Affinity: 3
ChildEBP RetAddr
0080f9ac 770d31bb ntdll!NtDelayExecution+0x15
0080fa14 770d3a8b KERNELBASE!SleepEx+0x65
0080fa24 752d28dd KERNELBASE!Sleep+0xf
0080fa38 752b98f8 kernel32!WerpReportFault+0x3f
0080fa48 752b9875 kernel32!BasepReportFault+0x20
0080fad4 77b10df7 kernel32!UnhandledExceptionFilter+0x1af
0080fadc 77b10cd4 ntdll!__RtlUserThreadStart+0x62
0080faf0 77b10b71 ntdll!_EH4_CallFilterFunc+0x12
0080fb18 77ae6ac9 ntdll!_except_handler4+0x8e
0080fb3c 77ae6a9b ntdll!ExecuteHandler2+0x26
0080fbec 77ab010f ntdll!ExecuteHandler+0x24
0080fbec 013a1000 ntdll!KiUserExceptionDispatcher+0xf
0080ff38 013a10ab ApplicationK!thread_two
0080ff70 013a1147 ApplicationK!_callthreadstart+0x1b
0080ff78 75293677 ApplicationK!_threadstart+0x76
0080ff84 77ad9f02 kernel32!BaseThreadInitThunk+0xe
0080ffc4 77ad9ed5 ntdll!__RtlUserThreadStart+0x70
0080ffdc 00000000 ntdll!_RtlUserThreadStart+0x1b

Total threads: 3
Duplicate callstacks: 1(windbg thread #s follow):
2

Generally, any property can be chosen to form such a set from a collection of stack traces.

Special Thread (.NET CLR)

When analyzing memory dumps from specific application platforms we see threads having definite purpose as a part of that specific platform architecture, design and implementation. For example, in applications and services involving .NET CLR we see the following threads:

0:000> !Threads -special
ThreadCount:           9
UnstartedThread:       0
BackgroundThread:      7
PendingThread:         0
DeadThread:            1
Hosted Runtime:        no
PreEmptive GC Alloc Lock
  ID OSID ThreadOBJ State   GC       Context           Domain    Count APT
Exception
0 1  b10  002fbe88  6020    Enabled  0acbdebc:0acbf5a4 002f17d0 0      STA
2 2  bf0  00306b18  b220    Enabled  00000000:00000000 002f17d0 0      MTA
(Finalizer)
3 3  b34  0034c188  b220    Enabled  00000000:00000000 002f17d0 0      MTA
XXXX 5    0037e3e0  19820   Enabled  00000000:00000000 002f17d0 0      Ukn
5 7  700  04b606c8  200b220 Enabled  00000000:00000000 002f17d0 0      MTA
6 4  ec4  04baffa0  200b220 Enabled  00000000:00000000 002f17d0 0      MTA
8 8  10c  04bf19b8  8009220 Enabled  00000000:00000000 002f17d0 0      MTA
(Threadpool Completion Port)
9 11 464  0be106d8  1220    Enabled  00000000:00000000 002f17d0 0      Ukn
10 10 da0 003c7958  7220    Disabled 00000000:00000000 0be1dd00 0      STA

  OSID Special thread type
1 c08  DbgHelper
2 bf0  Finalizer
7 f54  Gate
8 10c  IOCompletion
9 464  ADUnloadHelper

Dynamic Memory Corruption (Managed Heap)

Here's a managed heap counterpart to process heap (Volume 1, page 257) and kernel pool (Volume 2, page 204) corruption patterns. It is usually detected by CLR during GC phases. Here is a typical stack from CLR 2 (CLR 4 is similar):

0:000> kL
ChildEBP RetAddr
002baae0 779b06a0 ntdll!KiFastSystemCallRet
002baae4 772c77d4 ntdll!NtWaitForSingleObject+0xc
002bab54 772c7742 kernel32!WaitForSingleObjectEx+0xbe
002bab68 7a0c0a43 kernel32!WaitForSingleObject+0x12
002bab98 7a0c0e89 mscorwks!ClrWaitForSingleObject+0x24
002bb054 7a0c2bfd mscorwks!RunWatson+0x1df
002bb798 7a0c3171 mscorwks!DoFaultReportWorker+0xb62
002bb7d4 7a106b2d mscorwks!DoFaultReport+0xc3
002bb7fc 7a1061ac mscorwks!WatsonLastChance+0x43
002bbcb8 7a10624f mscorwks!EEPolicy::LogFatalError+0x3ae
002bbcd0 79ffee2f mscorwks!EEPolicy::HandleFatalError+0x36
002bbcf4 79f04f1f mscorwks!CLRVectoredExceptionHandlerPhase3+0xc1
002bbd28 79f04e98 mscorwks!CLRVectoredExceptionHandlerPhase2+0x20
002bbd5c 79f9149e mscorwks!CLRVectoredExceptionHandler+0x10a
002bbd70 779b1039 mscorwks!CLRVectoredExceptionHandlerShimX86+0x27
002bbd94 779b100b ntdll!ExecuteHandler2+0x26
002bbe3c 779b0e97 ntdll!ExecuteHandler+0x24
002bbe3c 79f69360 ntdll!KiUserExceptionDispatcher+0xf
002bc13c 79f663f1 mscorwks!SVR::heap_segment_next_rw+0xf
002bc228 79f65d63 mscorwks!WKS::gc_heap::plan_phase+0×37c
002bc248 79f6614c mscorwks!WKS::gc_heap::gc1+0×6e
002bc25c 79f65f5d mscorwks!WKS::gc_heap::garbage_collect+0×261
002bc288 79f663c2 mscorwks!WKS::GCHeap::GarbageCollectGeneration+0×1a9
002bc314 79ef1566 mscorwks!WKS::gc_heap::try_allocate_more_space+0×12e
002bc328 79ef1801 mscorwks!WKS::gc_heap::allocate_more_space+0×11
002bc348 79e7510e mscorwks!WKS::GCHeap::Alloc+0×3b
002bc364 79e86713 mscorwks!Alloc+0×60
002bc3a0 79e86753 mscorwks!SlowAllocateString+0×29
002bc3ac 79eb4efb mscorwks!UnframedAllocateString+0xc
002bc3e0 79e91f58 mscorwks!AllocateStringObject+0×2e
002bc424 79e82892 mscorwks!GlobalStringLiteralMap::AddStringLiteral+0×3f
002bc438 79e82810 mscorwks!GlobalStringLiteralMap::GetStringLiteral+0×43
002bc47c 79e82956 mscorwks!AppDomainStringLiteralMap::GetStringLiteral+0×72
002bc494 79e81b6f mscorwks!BaseDomain::GetStringObjRefPtrFromUnicodeString+0×31
002bc4cc 79ef4704 mscorwks!Module::ResolveStringRef+0×88
002bc4e4 79f23132 mscorwks!ConstructStringLiteral+0×39
002bc558 7908c351 mscorwks!CEEInfo::constructStringLiteral+0×108
002bc57c 7906276d mscorjit!Compiler::fgMorphConst+0xa3
002bc598 79065ea0 mscorjit!Compiler::fgMorphTree+0×63
002bc610 79062bb5 mscorjit!Compiler::fgMorphArgs+0×86
002bc63c 7906311f mscorjit!Compiler::fgMorphCall+0×2c1
002bc658 79065ea0 mscorjit!Compiler::fgMorphTree+0xa3
002bc6d0 79062bb5 mscorjit!Compiler::fgMorphArgs+0×86
002bc6fc 7906311f mscorjit!Compiler::fgMorphCall+0×2c1
002bc718 790650fa mscorjit!Compiler::fgMorphTree+0xa3
002bc738 79065026 mscorjit!Compiler::fgMorphStmts+0×63
002bc774 79064f9f mscorjit!Compiler::fgMorphBlocks+0×79
002bc788 79064e63 mscorjit!Compiler::fgMorph+0×60
002bc798 790614e6 mscorjit!Compiler::compCompile+0×5f
002bc7f0 79061236 mscorjit!Compiler::compCompile+0×2df
002bc884 7906118c mscorjit!jitNativeCode+0xb8
002bc8bc 79f0f9cf mscorjit!CILJit::compileMethod+0×3d
002bc928 79f0f945 mscorwks!invokeCompileMethodHelper+0×72
002bc96c 79f0f8da mscorwks!invokeCompileMethod+0×31
002bc9c4 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0×84
002bcd7c 79f0e795 mscorwks!UnsafeJitFunction+0×230
002bce20 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0×1c1
002bce78 79e8809e mscorwks!MethodDesc::DoPrestub+0×486
002bcec8 00330836 mscorwks!PreStubWorker+0xeb
WARNING: Frame IP not in any known module. Following frames may be wrong.
002bcee0 79e7c74b 0×330836
002bcf10 79e7c6cc mscorwks!CallDescrWorker+0×33
002bcf90 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
002bd0d0 79e7c783 mscorwks!MethodDesc::CallDescr+0×19c
002bd0ec 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0×1f
002bd100 79e8b983 mscorwks!MethodDescCallSite::Call_RetArgSlot+0×18
002bd1d8 79e8b8e6 mscorwks!MethodTable::RunClassInitWorker+0×8b
002bd260 79e8b7fa mscorwks!MethodTable::RunClassInitEx+0×11e
002bd724 79ebcee6 mscorwks!MethodTable::DoRunClassInitThrowing+0×2f0
002bd79c 79fc49db mscorwks!MethodTable::CheckRunClassInitNT+0×8c
002bd82c 790a2801 mscorwks!CEEInfo::initClass+0×19b
002bddcc 79062cdc mscorjit!Compiler::impExpandInline+0×2aaa
002bde24 79062b7c mscorjit!Compiler::fgMorphCallInline+0xf8
002bde50 7906311f mscorjit!Compiler::fgMorphCall+0×27b
002bde6c 790650fa mscorjit!Compiler::fgMorphTree+0xa3
002bde8c 79065026 mscorjit!Compiler::fgMorphStmts+0×63
002bdec8 79064f9f mscorjit!Compiler::fgMorphBlocks+0×79
002bdedc 79064e63 mscorjit!Compiler::fgMorph+0×60
002bdeec 790614e6 mscorjit!Compiler::compCompile+0×5f
002bdf44 79061236 mscorjit!Compiler::compCompile+0×2df
002bdfd8 7906118c mscorjit!jitNativeCode+0xb8
002be010 79f0f9cf mscorjit!CILJit::compileMethod+0×3d
002be07c 79f0f945 mscorwks!invokeCompileMethodHelper+0×72
002be0c0 79f0f8da mscorwks!invokeCompileMethod+0×31
002be118 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0×84
002be4d0 79f0e795 mscorwks!UnsafeJitFunction+0×230
002be574 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0×1c1
002be5cc 79e8809e mscorwks!MethodDesc::DoPrestub+0×486
002be61c 00330836 mscorwks!PreStubWorker+0xeb
002be634 0570c859 0×330836
002be69c 0595bcc1 0×570c859
002be700 0595b954 0×595bcc1
002be704 099b66e0 0×595b954
002be708 002be728 0×99b66e0
002be70c 09589c90 0×2be728
002be728 099b67b8 0×9589c90
002be72c 00000000 0×99b67b8

Usually !VerifyHeap SOS command helps to find the first invalid object on managed heap and shows the last valid one. Sometimes the corruption can deeply affect heap or when a crash happens during traversal GC state might not be valid for analysis:

0:000> !VerifyHeap
-verify will only produce output if there are errors in the heap
The garbage collector data structures are not in a valid state for traversal.
It is either in the “plan phase,” where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
Error requesting heap segment 80018001
Failed to retrieve segments for gc heap
Unable to build snapshot of the garbage collector state

0:000> !DumpHeap
The garbage collector data structures are not in a valid state for traversal.
It is either in the “plan phase,” where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
Error requesting heap segment 80018001
Failed to retrieve segments for gc heap
Unable to build snapshot of the garbage collector state

In such cases it is recommended to collect several dumps to catch more consistent heap state:

0:000> !VerifyHeap
-verify will only produce output if there are errors in the heap
object 0981f024: does not have valid MT
curr_object: 0981f024
Last good object: 0981f010

Then we can use !DumpObj (!do) command to check objects and d* WinDbg command variations to inspect raw memory.

Stack Trace Collection (Managed Space)

Here we have a managed space counterpart to unmanaged Stack Trace Collection (Volume 1, page 409) pattern. When looking at crash dumps from a different than a postmortem analysis machine we might need the appropriate version-specific SOS extension (page 99). To list all managed stack traces we use this combined command:

0:000> ~*e !CLRStack
OS Thread Id: 0×36f0 (0)
Child SP IP         Call Site
0031cf84 779b0f34 [HelperMethodFrame: 0031cf84]
0031cfd8 6b65665e System.Collections.ArrayList.GetEnumerator()
0031cfe4 059f4c92
ActiproSoftware.SyntaxEditor.EditorView.a(System.Windows.Forms.PaintEventA
rgs, System.Drawing.Rectangle, ActiproSoftware.SyntaxEditor.DocumentLine,
ActiproSoftware.SyntaxEditor.DisplayLine,
ActiproSoftware.SyntaxEditor.EditPositionRange, Int32 ByRef)
0031e158 05a01798
ActiproSoftware.SyntaxEditor.EditorView.OnRender(System.Windows.Forms.Pain
tEventArgs)
0031e748 04c5f888
ActiproSoftware.WinUICore.UIElement.Render(System.Windows.Forms.PaintEvent
Args)
0031e758 04c5f602
ActiproSoftware.WinUICore.UIControl.OnRenderChildElements(System.Windows.F
orms.PaintEventArgs)
0031e80c 04c5f1ac
ActiproSoftware.WinUICore.UIControl.Render(System.Windows.Forms.PaintEvent
Args)
0031e81c 04c5e6fe
ActiproSoftware.WinUICore.UIControl.a(System.Windows.Forms.PaintEventArgs)
0031e9a4 04c5e415
ActiproSoftware.WinUICore.UIControl.OnPaint(System.Windows.Forms.PaintEven
tArgs)
0031e9b4 69f156f5
System.Windows.Forms.Control.PaintWithErrorHandling(System.Windows.Forms.P
aintEventArgs, Int16)
0031e9e8 69f1809e
System.Windows.Forms.Control.WmPaint(System.Windows.Forms.Message ByRef)
0031ead4 69f073b1
System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
0031ead8 69f102aa [InlinedCallFrame: 0031ead8]
0031eb2c 69f102aa
System.Windows.Forms.ScrollableControl.WndProc(System.Windows.Forms.Messag
e ByRef)
0031eb38 048269c3
ActiproSoftware.SyntaxEditor.SyntaxEditor.WndProc(System.Windows.Forms.Mes
sage ByRef)
0031ebe8 69f070f3
System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.
Forms.Message ByRef)
0031ebf0 69f07071
System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Fo
rms.Message ByRef)
0031ec04 69f06fb6 System.Windows.Forms.NativeWindow.Callback(IntPtr,
Int32, IntPtr, IntPtr)
0031ee44 00e71365 [InlinedCallFrame: 0031ee44]
0031ee40 69f22eec DomainNeutralILStubClass.IL_STUB_PInvoke(MSG ByRef)
0031ee44 69f171ff [InlinedCallFrame: 0031ee44]
System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
0031ee88 69f171ff
System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.
UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr, Int32,
Int32)
0031ee8c 69f16e2c [InlinedCallFrame: 0031ee8c]
0031ef24 69f16e2c
System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32,
System.Windows.Forms.ApplicationContext)
0031ef7c 69f16c81
System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32,
System.Windows.Forms.ApplicationContext)
0031efac 69ea366d
System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
0031efc0 003c3f0d LINQPad.Program.Run(System.String, Boolean,
System.String, Boolean, Boolean, System.String)
0031f0b4 003c1515 LINQPad.Program.Go(System.String[])
0031f2e4 003c0584 LINQPad.Program.Start(System.String[])
0031f324 003c034f LINQPad.ProgramStarter.Run(System.String[])
0031f330 003c00f5 LINQPad.Loader.Main(System.String[])
0031f570 6c1721db [GCFrame: 0031f570]
OS Thread Id: 0×36f8 (1)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×36fc (2)
Child SP IP          Call Site
03c2fb78 779b0f34 [DebuggerU2MCatchHandlerFrame: 03c2fb78]
OS Thread Id: 0×3700 (3)
Child SP IP       Call Site
0470f0cc 779b0f34 [InlinedCallFrame: 0470f0cc]
0470f0c8 699380b7
DomainNeutralILStubClass.IL_STUB_PInvoke(Microsoft.Win32.SafeHandles.SafeP
ipeHandle, IntPtr)
0470f0cc 699cc07c [InlinedCallFrame: 0470f0cc]
Microsoft.Win32.UnsafeNativeMethods.ConnectNamedPipe(Microsoft.Win32.SafeH
andles.SafePipeHandle, IntPtr)
0470f130 699cc07c
System.IO.Pipes.NamedPipeServerStream.WaitForConnection()
0470f140 003c31ed LINQPad.Program.Listen()
0470f1d8 6b64ae5b
System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0470f1e8 6b5d7ff4
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object, Boolean)
0470f20c 6b5d7f34
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object)
0470f228 6b64ade8 System.Threading.ThreadHelper.ThreadStart()
0470f44c 6c1721db [GCFrame: 0470f44c]
0470f710 6c1721db [DebuggerU2MCatchHandlerFrame: 0470f710]
OS Thread Id: 0×3704 (4)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×3710 (5)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×3714 (6)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×3718 (7)
Child SP IP          Call Site
0596ee40 779b0f34 [GCFrame: 0596ee40]
0596ef34 779b0f34 [HelperMethodFrame_1OBJ: 0596ef34]
System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
0596ef90 6b5d9140 System.Threading.Monitor.Wait(System.Object, Int32,
Boolean)
0596efa0 6bb9428a System.Threading.Monitor.Wait(System.Object)
0596efa4 003cb55a ActiproSoftware.SyntaxEditor.SemanticParserService.c()
0596f068 6b64ae5b
System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0596f078 6b5d7ff4
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object, Boolean)
0596f09c 6b5d7f34
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object)
0596f0b8 6b64ade8 System.Threading.ThreadHelper.ThreadStart()
0596f2dc 6c1721db [GCFrame: 0596f2dc]
0596f5a0 6c1721db [DebuggerU2MCatchHandlerFrame: 0596f5a0]
OS Thread Id: 0×3764 (8)
Child SP IP          Call Site
0564f148 779b0f34 [HelperMethodFrame_1OBJ: 0564f148]
System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.S
afeHandle, UInt32, Boolean, Boolean)
0564f1f0 6b64b5ef
System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices
.SafeHandle, Int64, Boolean, Boolean)
0564f20c 6b64b5ad System.Threading.WaitHandle.WaitOne(Int32, Boolean)
0564f224 6b64b570 System.Threading.WaitHandle.WaitOne()
0564f22c 04b607d4
ActiproBridge.ReferenceManager+?15?.<StartAdvanceFeeder>b__4()
0564f268 6b64ae5b
System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0564f278 6b5d7ff4
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object, Boolean)
0564f29c 6b5d7f34
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object)
0564f2b8 6b64ade8 System.Threading.ThreadHelper.ThreadStart()
0564f4dc 6c1721db [GCFrame: 0564f4dc]
0564f7a0 6c1721db [DebuggerU2MCatchHandlerFrame: 0564f7a0]
OS Thread Id: 0×3768 (9)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×376c (10)
Child SP IP          Call Site
GetFrameContext failed: 1
OS Thread Id: 0×3798 (11)
Child SP IP          Call Site
GetFrameContext failed: 1
OS Thread Id: 0×37e0 (12)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
OS Thread Id: 0×37f8 (13)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process

Duplicate Extension

This pattern is a Duplicate Module (Volume 2, page 294) equivalent for a debugger that uses loaded modules to extend its functionality. For example, in the case of WinDbg there is a possibility that two different Version-Specific Extensions (page 99) are loaded wreaking havoc on debugging process (Debugger DLL Hell). For example, we loaded a specific version of SOS extension and successfully got a stack trace:

0:000> lmv m mscorwks
start end module name
79e70000 7a3ff000 mscorwks (deferred)
Image path:          C:WindowsMicrosoft.NETFrameworkv2.0.50727mscorwks.dll
Image name:          mscorwks.dll
Timestamp:       Wed Oct 24 08:41:29 2007 (471EF729)
CheckSum:        00597AA8
ImageSize:       0058F000
File version:    2.0.50727.1433
Product version: 2.0.50727.1433
File flags:      0 (Mask 3F)
File OS:         4 Unknown Win32
File type:       2.0 Dll
File date:       00000000.00000000
Translations:    0409.04b0
CompanyName:     Microsoft Corporation
ProductName:     Microsoft® .NET Framework
InternalName:    mscorwks.dll
OriginalFilename: mscorwks.dll
ProductVersion: 2.0.50727.1433
FileVersion:     2.0.50727.1433 (REDBITS.050727-1400)
FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail

0:000> .chain
Extension DLL search Path:
[...]
Extension DLL chain:
dbghelp: image 6.12.0002.633, API 6.1.6, built Mon Feb 01 20:08:26 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)dbghelp.dll]
ext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:31 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)winextext.dll]
exts: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:24 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)WINXPexts.dll]
uext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:23 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)winextuext.dll]
ntsdexts: image 6.1.7650.0, API 1.0.0, built Mon Feb 01 20:08:08 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)WINXP
tsdexts.dll]

0:000> .load .load C:Frameworks32-
bitFramework.UpdatesMicrosoft.NETFrameworkv2.0.50727sos
0:000> .chain
Extension DLL search Path:
[...]
Extension DLL chain:
C:Frameworks32-bitFramework.UpdatesMicrosoft.NETFrameworkv2.0.50727sos: image
2.0.50727.1433, API 1.0.0, built Wed Oct 24 04:41:30 2007
[path: C:Frameworks32-
bitFramework.UpdatesMicrosoft.NETFrameworkv2.0.50727sos.dll]
dbghelp: image 6.12.0002.633, API 6.1.6, built Mon Feb 01 20:08:26 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)dbghelp.dll]
ext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:31 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)winextext.dll]
exts: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:24 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)WINXPexts.dll]
uext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:23 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)winextuext.dll]
ntsdexts: image 6.1.7650.0, API 1.0.0, built Mon Feb 01 20:08:08 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)WINXP
tsdexts.dll]

0:000> !CLRStack
OS Thread Id: 0xdd0 (0)
ESP      EIP
002eeaa8 77c40f34 [InlinedCallFrame: 002eeaa8]
System.Windows.Forms.UnsafeNativeMethods.WaitMessage()
002eeaa4 7b08374f
System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.
UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
002eeb44 7b0831a5
System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32,
System.Windows.Forms.ApplicationContext)
002eebbc 7b082fe3
System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32,
System.Windows.Forms.ApplicationContext)
002eebec 7b0692c2 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
002eebfc 00833264 LINQPad.Program.Run(System.String, Boolean, System.String, Boolean,
Boolean, System.String)
002eec50 008311dc LINQPad.Program.Go(System.String[])
002eedac 00830545 LINQPad.Program.Start(System.String[])
002eede0 00830362 LINQPad.ProgramStarter.Run(System.String[])
002eede8 008300e3 LINQPad.Loader.Main(System.String[])
002ef00c 79e7c74b [GCFrame: 002ef00c]

Then we tried the default analysis command !analyze -v -hang and continued using SOS commands. Unfortunately they no longer worked correctly:

0:000> !CLRStack
OS Thread Id: 0xdd0 (0)
ESP EIP
002eeaa8 77c40f34 [InlinedCallFrame: 002eeaa8]
002eeaa4 7b08374f
002eeb44 7b0831a5
002eebbc 7082fe3
002eebec 7b0692c2
002eebfc 00833264
002eec50 008311dc
002eedac 00830545
002eede0 00830362
002eede8 008300e3
002ef00c 79e7c74b [GCFrame: 002ef00c]

Looking at loaded extensions list we see that an additional wrong version of SOS.DLL was loaded and that one gets all SOS commands:

0:000> .chain
Extension DLL search Path:
[...]
Extension DLL chain:
C:WindowsMicrosoft.NETFrameworkv2.0.50727sos: image 2.0.50727.4963, API 1.0.0,
built Thu Jul 07 03:08:08 2011
[path: C:WindowsMicrosoft.NETFrameworkv2.0.50727sos.dll]
C:Frameworks32-bitFramework.UpdatesMicrosoft.NETFrameworkv2.0.50727sos: image
2.0.50727.1433, API 1.0.0, built Wed Oct 24 04:41:30 2007
[path: C:Frameworks32-
bitFramework.UpdatesMicrosoft.NETFrameworkv2.0.50727sos.dll]
dbghelp: image 6.12.0002.633, API 6.1.6, built Mon Feb 01 20:08:26 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)dbghelp.dll]
ext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:31 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)winextext.dll]
exts: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:24 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)WINXPexts.dll]
uext: image 6.12.0002.633, API 1.0.0, built Mon Feb 01 20:08:23 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)winextuext.dll]
ntsdexts: image 6.1.7650.0, API 1.0.0, built Mon Feb 01 20:08:08 2010
[path: C:Program Files (x86)Debugging Tools for Windows (x86)WINXP
tsdexts.dll]

If we specify the full path to the correct extension we get the right stack trace:

0:000> !C:Frameworks32-
bitFramework.UpdatesMicrosoft.NETFrameworkv2.0.50727sos.CLRStack
OS Thread Id: 0xdd0 (0)
ESP     EIP
002eeaa8 77c40f34 [InlinedCallFrame: 002eeaa8]
System.Windows.Forms.UnsafeNativeMethods.WaitMessage()
002eeaa4 7b08374f
System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.
UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
002eeb44 7b0831a5
System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32,
System.Windows.Forms.ApplicationContext)
002eebbc 7b082fe3
System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32,
System.Windows.Forms.ApplicationContext)
002eebec 7b0692c2 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
002eebfc 00833264 LINQPad.Program.Run(System.String, Boolean, System.String, Boolean,
Boolean, System.String)
002eec50 008311dc LINQPad.Program.Go(System.String[])
002eedac 00830545 LINQPad.Program.Start(System.String[])
002eede0 00830362 LINQPad.ProgramStarter.Run(System.String[])
002eede8 008300e3 LINQPad.Loader.Main(System.String[])
002ef00c 79e7c74b [GCFrame: 002ef00c]

To avoid confusion we unload the last loaded extension:

0:000> .unload C:WindowsMicrosoft.NETFrameworkv2.0.50727sos
Unloading C:WindowsMicrosoft.NETFrameworkv2.0.50727sos extension DLL

0:000> !CLRStack
OS Thread Id: 0xdd0 (0)
ESP      EIP
002eeaa8 77c40f34 [InlinedCallFrame: 002eeaa8]
System.Windows.Forms.UnsafeNativeMethods.WaitMessage()
002eeaa4 7b08374f
System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.
UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
002eeb44 7b0831a5
System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32,
System.Windows.Forms.ApplicationContext)
002eebbc 7b082fe3
System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32,
System.Windows.Forms.ApplicationContext)
002eebec 7b0692c2 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
002eebfc 00833264 LINQPad.Program.Run(System.String, Boolean, System.String, Boolean,
Boolean, System.String)
002eec50 008311dc LINQPad.Program.Go(System.String[])
002eedac 00830545 LINQPad.Program.Start(System.String[])
002eede0 00830362 LINQPad.ProgramStarter.Run(System.String[])
002eede8 008300e3 LINQPad.Loader.Main(System.String[])
002ef00c 79e7c74b [GCFrame: 002ef00c]

Deadlock (Managed Space)

Now we illustrate a synchronization block deadlock pattern in managed code. Here we can use either a manual !syncblk WinDbg command coupled with a stack trace and disassembly analysis or a SOSEX¹⁴ extension !dlk command (which automates the whole detection process).

0:011> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
373 052cbf1c 3 1 08f69280 bc0 14 0a1ffd84 System.String
375 052cbd3c 3 1 08f68728 b6c 12 0a1ffd4c System.String

0:011> ~12s
[...]

0:012> k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be
wrong.
09c8ebd0 79ed98fd ntdll!KiFastSystemCallRet
09c8ec38 79ed9889 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
09c8ec58 79ed9808 mscorwks!Thread::DoAppropriateAptStateWait+0x3c
09c8ecdc 79ed96c4 mscorwks!Thread::DoAppropriateWaitWorker+0x13c
09c8ed2c 79ed9a62 mscorwks!Thread::DoAppropriateWait+0x40
09c8ed88 79e78944 mscorwks!CLREvent::WaitEx+0xf7
09c8ed9c 79ed7b37 mscorwks!CLREvent::Wait+0x17
09c8ee28 79ed7a9e mscorwks!AwareLock::EnterEpilog+0x8c
09c8ee44 79ebd7e4 mscorwks!AwareLock::Enter+0x61
09c8eee4 074c1f38 mscorwks!JIT_MonEnterWorker_Portable+0xb3
09c8ef0c 793b0d1f 0×74c1f38
09c8ef14 79373ecd mscorlib_ni+0×2f0d1f
09c8ef28 793b0c68 mscorlib_ni+0×2b3ecd
09c8ef40 79e7c74b mscorlib_ni+0×2f0c68
09c8ef50 79e7c6cc mscorwks!CallDescrWorker+0×33
09c8efd0 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
09c8f110 79e7c783 mscorwks!MethodDesc::CallDescr+0×19c
09c8f12c 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0×1f
09c8f140 79fc58cd mscorwks!MethodDescCallSite::Call_RetArgSlot+0×18
09c8f328 79ef3207 mscorwks!ThreadNative::KickOffThread_Worker+0×190
09c8f33c 79ef31a3 mscorwks!Thread::DoADCallBack+0×32a
09c8f3d0 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
09c8f40c 79f01723 mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
09c8f41c 79f02a5d mscorwks!Thread::RaiseCrossContextException+0×434
09c8f4cc 79f02ab7 mscorwks!Thread::DoADCallBack+0xda
09c8f4e8 79ef31a3 mscorwks!Thread::DoADCallBack+0×310
09c8f57c 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
09c8f5b8 79ef4826 mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
09c8f5e0 79fc57b1 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
09c8f5f8 79fc56ac mscorwks!ManagedThreadBase::KickOff+0×13
09c8f694 79f95a2e mscorwks!ThreadNative::KickOffThread+0×269
09c8fd34 76573833 mscorwks!Thread::intermediateThreadProc+0×49
09c8fd40 77c1a9bd kernel32!BaseThreadInitThunk+0xe
09c8fd80 00000000 ntdll!LdrInitializeThunk+0×4d

0:012> ub 074c1f38
074c1f11 eb10 jmp 074c1f23
074c1f13 8b0df8927b02 mov ecx,dword ptr ds:[27B92F8h]
074c1f19 e8367ef271 call mscorlib_ni+0×329d54 (793e9d54)
074c1f1e e89272a472 call mscorwks!JIT_EndCatch (79f091b5)
074c1f23 b9d0070000 mov ecx,7D0h
074c1f28 e8c432b072 call mscorwks!ThreadNative::Sleep (79fc51f1)
074c1f2d 8b0d88dc7b02 mov ecx,dword ptr ds:[27BDC88h]
074c1f33 e811389b72 call mscorwks!JIT_MonEnterWorker (79e75749)

0:012> dp 27BDC88h l1
027bdc88 0a1ffd84

0:012> ~14s

0:014> k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be
wrong.
0b83ed04 79ed98fd ntdll!KiFastSystemCallRet
0b83ed6c 79ed9889 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
0b83ed8c 79ed9808 mscorwks!Thread::DoAppropriateAptStateWait+0x3c
0b83ee10 79ed96c4 mscorwks!Thread::DoAppropriateWaitWorker+0x13c
0b83ee60 79ed9a62 mscorwks!Thread::DoAppropriateWait+0x40
0b83eebc 79e78944 mscorwks!CLREvent::WaitEx+0xf7
0b83eed0 79ed7b37 mscorwks!CLREvent::Wait+0x17
0b83ef5c 79ed7a9e mscorwks!AwareLock::EnterEpilog+0x8c
0b83ef78 79ebd7e4 mscorwks!AwareLock::Enter+0x61
0b83f018 074c5681 mscorwks!JIT_MonEnterWorker_Portable+0xb3
0b83f01c 793b0d1f 0×74c5681
0b83f024 79373ecd mscorlib_ni+0×2f0d1f
0b83f038 793b0c68 mscorlib_ni+0×2b3ecd
0b83f050 79e7c74b mscorlib_ni+0×2f0c68
0b83f060 79e7c6cc mscorwks!CallDescrWorker+0×33
0b83f0e0 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
0b83f220 79e7c783 mscorwks!MethodDesc::CallDescr+0×19c
0b83f23c 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0×1f
0b83f250 79fc58cd mscorwks!MethodDescCallSite::Call_RetArgSlot+0×18
0b83f438 79ef3207 mscorwks!ThreadNative::KickOffThread_Worker+0×190
0b83f44c 79ef31a3 mscorwks!Thread::DoADCallBack+0×32a
0b83f4e0 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0b83f51c 79f01723 mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
0b83f52c 79f02a5d mscorwks!Thread::RaiseCrossContextException+0×434
0b83f5dc 79f02ab7 mscorwks!Thread::DoADCallBack+0xda
0b83f5f8 79ef31a3 mscorwks!Thread::DoADCallBack+0×310
0b83f68c 79ef30c3 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0b83f6c8 79ef4826 mscorwks!Thread::ShouldChangeAbortToUnload+0×30a
0b83f6f0 79fc57b1 mscorwks!Thread::ShouldChangeAbortToUnload+0×33e
0b83f708 79fc56ac mscorwks!ManagedThreadBase::KickOff+0×13
0b83f7a4 79f95a2e mscorwks!ThreadNative::KickOffThread+0×269
0b83ff3c 76573833 mscorwks!Thread::intermediateThreadProc+0×49
0b83ff48 77c1a9bd kernel32!BaseThreadInitThunk+0xe
0b83ff88 00000000 ntdll!LdrInitializeThunk+0×4d

0:014> ub 074c5681
074c565c 080c54 or byte ptr [esp+edx*2],cl
074c565f 07 pop es
074c5660 8b0d88dc7b02 mov ecx,dword ptr ds:[27BDC88h]
074c5666 e8de009b72 call mscorwks!JIT_MonEnterWorker (79e75749)
074c566b a1240a5407 mov eax,dword ptr ds:[07540A24h]
074c5670 3105280a5407 xor dword ptr ds:[7540A28h],eax
074c5676 8b0d84dc7b02 mov ecx,dword ptr ds:[27BDC84h]
074c567c e8c8009b72 call mscorwks!JIT_MonEnterWorker (79e75749)

0:014> dp 27BDC84h l1
027bdc84 0a1ffd4c

0:014> !dlk
Examining SyncBlocks...
Scanning for ReaderWriterLock instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for ReaderWriterLockSlim instances...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
Could not find symbol ntdll!RtlCriticalSectionList.
Scanning for threads waiting on SyncBlocks...
Scanning for threads waiting on ReaderWriterLock locks...
Scanning for threads waiting on ReaderWriterLocksSlim locks...
Scanning for threads waiting on CriticalSections...
*DEADLOCK DETECTED*
CLR thread 0xd holds the lock on SyncBlock 052cbd3c
OBJ:0a1ffd4c[System.String] STRVAL=critical section 1
...and is waiting for the lock on SyncBlock 052cbf1c
OBJ:0a1ffd84[System.String] STRVAL=critical section 2
CLR thread 0xb holds the lock on SyncBlock 052cbf1c
OBJ:0a1ffd84[System.String] STRVAL=critical section 2
...and is waiting for the lock on SyncBlock 052cbd3c
OBJ:0a1ffd4c[System.String] STRVAL=critical section 1
CLR Thread 0xd is waiting at UserQuery+ClassMain.thread_proc_1()(+0×42
IL)(+0×60 Native)
CLR Thread 0xb is waiting at UserQuery+ClassMain.thread_proc_2()(+0×19
IL)(+0×21 Native)

1 deadlock detected.

Caller-n-Callee

We noticed this pattern when analyzing the output of !DumpStack WinDbg SOS extension command:

0:011> !DumpStack
OS Thread Id: 0xac (11)
[...]
ChildEBP RetAddr  Caller, Callee
[...]
0b73f65c 77c416dc ntdll!RtlAllocateHeap+0×17c, calling
ntdll!RtlpLowFragHeapAllocFromContext
0b73f688 77c486cd ntdll!RtlAllocateHeap+0×193, calling ntdll!memset
0b73f6b0 7653a467 kernel32!TlsSetValue+0×4c, calling ntdll!RtlAllocateHeap
0b73f6cc 77a01c48 urlmon!CUrlMkTls::TLSAllocData+0×3f, calling kernel32!TlsSetValue
0b73f6dc 77a0198d urlmon!CUrlMkTls::CUrlMkTls+0×29, calling
urlmon!CUrlMkTls::TLSAllocData
0b73f6e8 77a01be5 urlmon!TlsDllMain+0×100, calling urlmon!EnsureFeatureCache
0b73f6f4 6d016a21 mshtml!DllMain+0×10, calling kernel32!GetCurrentThreadId
0b73f704 6d016b6c mshtml!_CRT_INIT+0×281, calling mshtml!DllMain
0b73f71c 7239133e msimtf!_CRT_INIT+0×281, calling msimtf!DllMain
0b73f728 72391375 msimtf!_CRT_INIT+0×3e7, calling msimtf!_SEH_epilog4
0b73f764 6d016ad0 mshtml!_DllMainStartup+0×56, calling mshtml!_DllMainCRTStartup
0b73f778 72391375 msimtf!_CRT_INIT+0×3e7, calling msimtf!_SEH_epilog4
0b73f77c 77c4a604 ntdll!LdrpCallInitRoutine+0×14
0b73f7a4 77c1ab6c ntdll!LdrpInitializeThread+0×1e9, calling
ntdll!RtlLeaveCriticalSection
0b73f7ac 77c1a9ea ntdll!LdrpInitializeThread+0×1cd, calling ntdll!_SEH_epilog4
0b73f800 77c1ab15 ntdll!LdrpInitializeThread+0×11f, calling
ntdll!RtlActivateActivationContextUnsafeFast
0b73f804 77c1ab53 ntdll!LdrpInitializeThread+0×167, calling
ntdll!RtlDeactivateActivationContextUnsafeFast
0b73f838 77c1a9ea ntdll!LdrpInitializeThread+0×1cd, calling ntdll!_SEH_epilog4
0b73f83c 77c405a0 ntdll!NtTestAlert+0xc
0b73f840 77c1a968 ntdll!_LdrpInitialize+0×29c, calling ntdll!_SEH_epilog4
0b73f8a0 77c3f3d0 ntdll!NtContinue+0xc
0b73f8a4 77c1a98a ntdll!LdrInitializeThunk+0×1a, calling ntdll!NtContinue
0b73fb30 6afd59f6 clr!Thread::intermediateThreadProc+0×39, calling
clr!_alloca_probe_16
0b73fb44 76573833 kernel32!BaseThreadInitThunk+0xe
0b73fb50 77c1a9bd ntdll!_RtlUserThreadStart+0×23

Obviously the command collected “call-type” execution residue (Volume 2, page 239) from the raw stack. The “calling” part wasn't found in the nearby region:

0:011> dps 0b73f7a4-20 0b73f7a4+20
0b73f784 72390000 msimtf!_imp__RegOpenKeyW <PERF> (msimtf+0×0)
0b73f788 00000002
0b73f78c 00000000
0b73f790 00000001
0b73f794 0b73f80c
0b73f798 0b73f80c
0b73f79c 00000001
0b73f7a0 05636578
0b73f7a4 0b73f83c
0b73f7a8 77c1ab6c ntdll!LdrpInitializeThread+0×1e9
0b73f7ac 77ca5340 ntdll!LdrpLoaderLock
0b73f7b0 77c1a9ea ntdll!LdrpInitializeThread+0×1cd
0b73f7b4 0b7321f2
0b73f7b8 7ff4e000
0b73f7bc 7ffdf000
0b73f7c0 77ca51f4 ntdll!LdrpProcessInitialized
0b73f7c4 00000000

We tried to disassemble backwards the addresses and found the callees:

0:011> ub 77c1ab6c
ntdll!LdrpInitializeThread+0×16b:
77c1ab57 90 nop
77c1ab58 90 nop
77c1ab59 90 nop
77c1ab5a 90 nop
77c1ab5b 90 nop
77c1ab5c ff054452ca77 inc dword ptr [ntdll!LdrpActiveThreadCount
(77ca5244)]
77c1ab62 684053ca77 push offset ntdll!LdrpLoaderLock (77ca5340)
77c1ab67 e8bd820000 call ntdll!RtlLeaveCriticalSection (77c22e29)

0:011> ub 77a01be5
urlmon!TlsDllMain+0×2f:
77a01bce 8d4510 lea eax,[ebp+10h]
77a01bd1 50 push eax
77a01bd2 8d4d0c lea ecx,[ebp+0Ch]
77a01bd5 e88efdffff call urlmon!CUrlMkTls::CUrlMkTls (77a01968)
77a01bda 397d10 cmp dword ptr [ebp+10h],edi
77a01bdd 7c09 jl urlmon!TlsDllMain+0×103 (77a01be8)
77a01bdf 56 push esi
77a01be0 e887fcffff call urlmon!EnsureFeatureCache (77a0186c)

In the past we were frequently referencing this pattern especially when discussing coincidental symbolic information (Volume 1, page 390) but didn't name it.

We can also run !DumpStack command against every thread (including non-managed) to get the summary of the call-type execution residue:

0:011> ~4s
eax=76573821 ebx=00000002 ecx=00000000 edx=74d01909 esi=00000000
edi=00000000
eip=77c40f34 esp=0478f8a0 ebp=0478f93c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
77c40f34 c3 ret

0:004> k
ChildEBP RetAddr
0478f89c 77c40690 ntdll!KiFastSystemCallRet
0478f8a0 76577e09 ntdll!ZwWaitForMultipleObjects+0xc
0478f93c 7674c4af kernel32!WaitForMultipleObjectsEx+0x11d
0478f990 76748b7b user32!RealMsgWaitForMultipleObjectsEx+0x13c
0478f9ac 74d01965 user32!MsgWaitForMultipleObjects+0x1f
0478f9f8 76573833 GdiPlus!BackgroundThreadProc+0x59
0478fa04 77c1a9bd kernel32!BaseThreadInitThunk+0xe
0478fa44 00000000 ntdll!_RtlUserThreadStart+0x23

0:004> !DumpStack
OS Thread Id: 0x950 (4)
Current frame: ntdll!KiFastSystemCallRet
ChildEBP RetAddr Caller, Callee
0478f89c 77c40690 ntdll!ZwWaitForMultipleObjects+0xc
0478f8a0 76577e09 kernel32!WaitForMultipleObjectsEx+0x11d, calling
ntdll!NtWaitForMultipleObjects
0478f914 76751a91 user32!UserCallWinProcCheckWow+0x5c, calling
ntdll!RtlActivateActivationContextUnsafeFast
0478f918 76751b41 user32!UserCallWinProcCheckWow+0x16a, calling
ntdll!RtlDeactivateActivationContextUnsafeFast
0478f93c 7674c4af user32!RealMsgWaitForMultipleObjectsEx+0x13c, calling
kernel32!WaitForMultipleObjectsEx
0478f968 76752a65 user32!DispatchMessageWorker+0x396, calling user32!_SEH_epilog4
0478f980 76743c64 user32!PeekMessageA+0x129, calling user32!_PeekMessage
0478f990 76748b7b user32!MsgWaitForMultipleObjects+0x1f, calling
user32!MsgWaitForMultipleObjectsEx
0478f9ac 74d01965 GdiPlus!BackgroundThreadProc+0x59, calling
user32!MsgWaitForMultipleObjects
0478f9f8 76573833 kernel32!BaseThreadInitThunk+0xe
0478fa04 77c1a9bd ntdll!_RtlUserThreadStart+0x23

Handled Exception (User Space)

In some previous articles we stated that we don't see traces of such exceptions on user space stacks. This is not strictly true. Although, we won't see exception codes when we inspect raw stack data we, nevertheless, in some cases might see execution residue left after calling exception handlers. For example, we can see that when we launch TestWER (page 280) tool and select ‘Handled Exception’ checkbox:

If we then click on a button and then save a process memory dump using Task Manager we find the following traces on a raw stack:

0:000> !teb
TEB at 7efdd000
ExceptionList:        0018fe20
StackBase:            00190000
StackLimit:           0018d000
SubSystemTib:         00000000
FiberData:            00001e00
ArbitraryUserPointer: 00000000
Self:                 7efdd000
EnvironmentPointer:   00000000
ClientId:             00000b38 . 00000f98
RpcHandle:            00000000
Tls Storage:          7efdd02c
PEB Address:          7efde000
LastErrorValue:       0
LastStatusValue:      c0000034
Count Owned Locks:    0
HardErrorMode:        0
0:000> dps 0018d000 00190000
[...]
0018f414 00000000
0018f418 0018f840
0018f41c 0018f4cc
0018f420 77726a9b ntdll!ExecuteHandler+0×24
0018f424 0018f4e4
0018f428 0018f840
0018f42c 0018f534
0018f430 0018f4b8
0018f434 00412600 TestWER!_except_handler4
0018f438 00000001
0018f43c 00000000
[...]

0:000> ub 77726a9b
ntdll!ExecuteHandler+0x7:
77726a7e 33f6 xor esi,esi
77726a80 33ff xor edi,edi
77726a82 ff742420 push dword ptr [esp+20h]
77726a86 ff742420 push dword ptr [esp+20h]
77726a8a ff742420 push dword ptr [esp+20h]
77726a8e ff742420 push dword ptr [esp+20h]
77726a92 ff742420 push dword ptr [esp+20h]
77726a96 e808000000 call ntdll!ExecuteHandler2 (77726aa3)

If we compare the output above with the raw stack fragment from the second chance exception memory dump (after we relaunch TestWER, don't select ‘Handled Exception’ checkbox and click on the big lightning button) we would see the similar call fragment:

[...]
0018f3f4 00dd0aa7
0018f3f8 0018f41c
0018f3fc 77726ac9 ntdll!ExecuteHandler2+0x26
0018f400 fffffffe
0018f404 0018ffc4
0018f408 0018f534
0018f40c 0018f4b8
0018f410 0018f840
0018f414 77726add ntdll!ExecuteHandler2+0x3a
0018f418 0018ffc4
0018f41c 0018f4cc
0018f420 77726a9b ntdll!ExecuteHandler+0×24
0018f424 0018f4e4
0018f428 0018ffc4
0018f42c 0018f534
0018f430 0018f4b8
0018f434 77750ae5 ntdll!_except_handler4
0018f438 00000000
0018f43c 0018f4e4
0018f440 0018ffc4
0018f444 77726a3d ntdll!RtlDispatchException+0×127
0018f448 0018f4e4
0018f44c 0018ffc4
0018f450 0018f534
0018f454 0018f4b8
0018f458 77750ae5 ntdll!_except_handler4
0018f45c 00000111
0018f460 0018f4e4
[...]

Sometimes, we can also see “Unwind”, “StackWalk”, “WalkFrames”, “EH”, “Catch” functions too. Sometimes we don't see anything because such residue was overwritten by subsequent function calls after handled exceptions happened sometime in the past.

Handled Exception (.NET CLR)

Similar to unmanaged user space handled exceptions (page 141) residue we can see similar one on raw stacks of .NET CLR threads. Here are some typical fragments (x86, CLR 4 has similar residue):

[...]
09c8e1e0 79ef2dee mscorwks!ExInfo::Init+0x41
09c8e1e4 00004000
09c8e1e8 79f088cc mscorwks!‘string’
09c8e1ec 79f088c2 mscorwks!ExInfo::UnwindExInfo+0x14d
09c8e1f0 08f68728
09c8e1f4 95f5b898
09c8e1f8 09c8e1a4
09c8e1fc 09c8e92c
09c8e200 7a34d0d8 mscorwks!GetManagedNameForTypeInfo+0x22b02
09c8e204 79f091ee mscorwks!COMPlusCheckForAbort+0x15
09c8e208 00000000
09c8e20c 0aada664
09c8e210 0aaabff4
09c8e214 00000000
09c8e218 09c8eeec
09c8e21c 074c1f23
09c8e220 09c8ef0c
09c8e224 79f091cb mscorwks!JIT_EndCatch+0x16
09c8e228 09c8ef0c
09c8e22c 09c8eeec
09c8e230 074c1f23
09c8e234 09c8e25c
09c8e238 0009c108
09c8e23c 09c8e460
09c8e240 09c8e5c4
09c8e244 00071d88
09c8e248 08f68728
09c8e24c 79e734c4 mscorwks!ClrFlsSetValue+0x57
09c8e250 95f5b8e4
09c8e254 0aada634
09c8e258 08f68728
09c8e25c 0aada90c
09c8e260 0aaabff4
09c8e264 00000002
09c8e268 09c8e304
09c8e26c 0aada664
09c8e270 00000000
09c8e274 09c8ef0c
09c8e278 09c8e234
09c8e27c 074c1f13
09c8e280 00000000
09c8e284 08f688a0
09c8e288 09c8e234
09c8e28c 79f00c0b mscorwks!Thread::ReturnToContext+0x4e2
09c8e290 0aada90c
09c8e294 09c8eef4
09c8e298 09c8e2bc
09c8e29c 79f08eb8 mscorwks!EEJitManager::ResumeAtJitEH+0x28
09c8e2a0 09c8e460
09c8e2a4 074c1ed8
09c8e2a8 074b41a8
09c8e2ac 00000000
09c8e2b0 08f68728
09c8e2b4 00000000
09c8e2b8 09c8e410
09c8e2bc 09c8e3c8
09c8e2c0 79f08df5 mscorwks!COMPlusUnwindCallback+0x7c3
09c8e2c4 09c8e460
09c8e2c8 074b41a8
09c8e2cc 00000000
09c8e2d0 08f68728
09c8e2d4 00000000
09c8e2d8 0009c108
09c8e2dc 09c8e410
09c8e2e0 09c8e5c4
09c8e2e4 074b41a8
09c8e2e8 09c8e3a4
09c8e2ec 79e734c4 mscorwks!ClrFlsSetValue+0x57
09c8e2f0 95f5b984
09c8e2f4 0009c128
09c8e2f8 09c8e3a4
09c8e2fc 00000000
09c8e300 00000000
09c8e304 00000002
[...]
09c8e4e4 00000000
09c8e4e8 79f09160 mscorwks!_CT??_R0H+0x34b4
09c8e4ec ffffffff
09c8e4f0 73792e2f msvcr80!_getptd+0x6
09c8e4f4 ffffffff
09c8e4f8 737b7a78 msvcr80!__FrameUnwindToState+0xd9
09c8e4fc 737b7a5e msvcr80!__FrameUnwindToState+0xbf
09c8e500 95f5bc05
09c8e504 e06d7363
09c8e508 1fffffff
09c8e50c 19930522
09c8e510 ffffffff
09c8e514 ffffffff
09c8e518 09c8e500
09c8e51c 09c8e554
09c8e520 09c8e5a8
09c8e524 73798cd9 msvcr80!_except_handler4
09c8e528 efbc0d3d
09c8e52c fffffffe
09c8e530 737b7a5e msvcr80!__FrameUnwindToState+0xbf
09c8e534 737b89cb msvcr80!__InternalCxxFrameHandler+0x6d
09c8e538 09c8eab0
09c8e53c 09c8e6a0
09c8e540 79f09160 mscorwks!_CT??_R0H+0x34b4
09c8e544 ffffffff
09c8e548 00000000
09c8e54c 00000000
09c8e550 00000000
09c8e554 09c8e590
09c8e558 737b8af1 msvcr80!__CxxFrameHandler3+0x26
09c8e55c 09c8e600
09c8e560 09c8eab0
09c8e564 01010101
09c8e568 09000000
09c8e56c 09c8f160
09c8e570 07540c00
09c8e574 00071d88
09c8e578 08e65d48
09c8e57c 09c8e5ec
09c8e580 074c1ec8
09c8e584 00000024
09c8e588 00000001
09c8e58c 0009c108
09c8e590 08f68728
09c8e594 00000000
09c8e598 00000000
09c8e59c 09c8eb38
09c8e5a0 00000000
09c8e5a4 09c8e6a0
09c8e5a8 09c8f15c
09c8e5ac 09c8f15c
09c8e5b0 09c8eb38
09c8e5b4 95f5bf28
09c8e5b8 09c8e8f4
09c8e5bc 79e84bf2 mscorwks!Thread::StackWalkFrames+0xb8
09c8e5c0 08f68728
09c8e5c4 09c8ea40
09c8e5c8 79e84bf2 mscorwks!Thread::StackWalkFrames+0xb8
09c8e5cc 09c8e5ec
09c8e5d0 79f07d64 mscorwks!COMPlusUnwindCallback
09c8e5d4 09c8ea40
09c8e5d8 00000005
09c8e5dc 00000000
09c8e5e0 08f68728
09c8e5e4 08f688a0
09c8e5e8 08f68728
09c8e5ec 09c8ec20
09c8e5f0 00000000
09c8e5f4 09c8ecbc
09c8e5f8 09c8ecc0
09c8e5fc 09c8ecc4
09c8e600 09c8ecc8
09c8e604 09c8eccc
09c8e608 09c8ecd0
09c8e60c 09c8ecd4
09c8e610 09c8eeec
09c8e614 09c8ecd8
09c8e618 09c8ecd8
09c8e61c 00000024
09c8e620 00000000
09c8e624 0009c108
09c8e628 08f68728
09c8e62c 00000000
09c8e630 00000000
09c8e634 79e71ba4 mscorwks!Thread::CatchAtSafePoint
09c8e638 00000000
09c8e63c 79e71ba4 mscorwks!Thread::CatchAtSafePoint
09c8e640 09c8f15c
09c8e644 09c8f15c
09c8e648 00000000
09c8e64c 95f5bcc0
09c8e650 09c8e988
09c8e654 79e84bf2 mscorwks!Thread::StackWalkFrames+0xb8
09c8e658 09c8ea40
09c8e65c 79e84bf2 mscorwks!Thread::StackWalkFrames+0xb8
09c8e660 09c8e680
09c8e664 79f07957 mscorwks!COMPlusThrowCallback
09c8e668 09c8ea40
09c8e66c 00000000
09c8e670 00000000
09c8e674 0aada90c
09c8e678 09c8ea40
09c8e67c 79e84bff mscorwks!Thread::StackWalkFrames+0xc5
09c8e680 09c8ec20
09c8e684 00000000
09c8e688 09c8ecbc
09c8e68c 09c8ecc0
09c8e690 09c8ecc4
09c8e694 09c8ecc8
[...]
09c8e8f0 95f5b264
09c8e8f4 09c8e914
09c8e8f8 79f07d5e mscorwks!UnwindFrames+0x62
09c8e8fc 79f07d64 mscorwks!COMPlusUnwindCallback
09c8e900 09c8ea40
09c8e904 00000005
09c8e908 00000000
09c8e90c 09c8ef6c
09c8e910 08f68728
09c8e914 09c8e9a4
09c8e918 79f089cc mscorwks!COMPlusAfterUnwind+0x97
09c8e91c 08f68728
09c8e920 09c8ea40
09c8e924 00000001
09c8e928 00000000
09c8e92c 09c8ef6c
09c8e930 79f0a3d9 mscorwks!COMPlusNestedExceptionHandler
09c8e934 09c8f160
09c8e938 00000000
09c8e93c 00000000
09c8e940 cccccccc
[...]

Sometimes we can see ‘ExecuteHandler’ calls if they were not overwritten:

[...]
09d2e6e0 00000000
09d2e6e4 00000720
09d2e6e8 77c41039 ntdll!ExecuteHandler2+0x26
09d2e6ec 09d2e7c8
09d2e6f0 09d2eb60
09d2e6f4 09d2e7e4
09d2e6f8 09d2e7a4
09d2e6fc 09d2eb60
09d2e700 77c4104d ntdll!ExecuteHandler2+0x3a
09d2e704 09d2eb60
09d2e708 09d2e7b0
09d2e70c 77c4100b ntdll!ExecuteHandler+0x24
09d2e710 09d2e7c8
09d2e714 00000001
09d2e718 09d2e6b0
09d2e71c 09d2e7a4
09d2e720 09d2e784
09d2e724 76545ac9 kernel32!_except_handler4
[...]

If there are such traces they can be visible as Caller-n-Callee (page 138) pattern:

0:011> !DumpStack
OS Thread Id: 0x3cc (11)
Current frame: ntdll!KiFastSystemCallRet
ChildEBP RetAddr Caller, Callee
09d2e690 77c40690 ntdll!ZwWaitForMultipleObjects+0xc
09d2e694 76577e09 kernel32!WaitForMultipleObjectsEx+0x11d, calling
ntdll!NtWaitForMultipleObjects
09d2e6d8 76578101 kernel32!WaitForMultipleObjectsEx+0x33, calling ntdll!
09d2e6e4 77c41039 ntdll!ExecuteHandler2+0×26
09d2e708 77c4100b ntdll!ExecuteHandler+0×24, calling ntdll!ExecuteHandler2
09d2e730 6baa516a clr!WaitForMultipleObjectsEx_SO_TOLERANT+0×56, calling
RtlActivateActivationContextUnsafeFast
kernel32!WaitForMultipleObjectsEx
09d2e794 6baa4f98 clr!Thread::DoAppropriateAptStateWait+0×4d, calling
clr!WaitForMultipleObjectsEx_SO_TOLERANT
09d2e7b4 6baa4dd8 clr!Thread::DoAppropriateWaitWorker+0×17d, calling
clr!Thread::DoAppropriateAptStateWait
09d2e848 6baa4e99 clr!Thread::DoAppropriateWait+0×60, calling
clr!Thread::DoAppropriateWaitWorker
09d2e8b4 6baa4f17 clr!CLREvent::WaitEx+0×106, calling
clr!Thread::DoAppropriateWait
09d2e8e0 6baa484b clr!CLRGetTickCount64+0×6b, calling clr!_allmul
09d2e908 6ba4d409 clr!CLREvent::Wait+0×19, calling clr!CLREvent::WaitEx
[...]

Execution Residue (Managed Space)

This is a .NET counterpart to unmanaged and native code execution residue (Volume 2, page 239) pattern. Here we can use SOS extension !DumpStack command for call level execution residue (see Caller-n-Callee pattern example, page 138) and !DumpStackObjects (!dso) for managed object references found on a raw stack:

0:011> !DumpStackObjects
OS Thread Id: 0x8e0 (11)
ESP/REG Object Name
09efe4b8 0a2571bc System.Threading.Thread
09efe538 0a1ffddc System.Threading.Thread
09efe844 0a1ffba8 UserQuery
09efe974 0a1ffce0 System.Signature
09efea20 0a1ffd10 System.RuntimeTypeHandle[]
09efeae8 08985e14 System.Object[] (System.Reflection.AssemblyName[])
09efeaec 0a1ffa78 System.Diagnostics.Stopwatch
09efeaf0 0a1ffa6c LINQPad.Extensibility.DataContext.QueryExecutionManager
09efeafc 0a1ffba8 UserQuery
09efeb00 0a1ffa58 System.RuntimeType
09efeb04 08995474 LINQPad.ObjectGraph.Formatters.XhtmlWriter
09efeb08 08985dfc System.Reflection.Assembly
09efeb0c 08985dc8 LINQPad.ExecutionModel.ResultData
09efeb10 08984548 LINQPad.ExecutionModel.Server
09efebdc 0a1ffbe8 System.Reflection.RuntimeMethodInfo
09efebe0 0a1fcfc4 LINQPad.ExecutionModel.ConsoleTextReader
09efebe4 0a1fcddc System.IO.StreamReader+NullStreamReader
09efebe8 0899544c System.IO.TextWriter+SyncTextWriter
09efebec 08985efc System.Reflection.AssemblyName
09efebf0 08985d4c System.String
C:UsersTrainingAppDataLocalTempLINQPadfcamvgpa
09efec30 08984548 LINQPad.ExecutionModel.Server
09efeedc 08985910 System.Threading.ThreadStart
0:011> !DumpObj 0a2571bc
Name: System.Threading.Thread
MethodTable: 790fe704
EEClass: 790fe694
Size: 56(0×38) bytes
(C:WindowsassemblyGAC_32mscorlib2.0.0.0__b77a5c561934e089mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
7910a5c4 4000634 4 ....Contexts.Context 0 instance 08980ee4 m_Context
79104de8 4000635 8 ....ExecutionContext 0 instance 00000000 m_ExecutionContext
790fd8c4 4000636 c System.String 0 instance 00000000 m_Name
790fe3b0 4000637 10 System.Delegate 0 instance 00000000 m_Delegate
79130084 4000638 14 System.Object[][] 0 instance 00000000 m_ThreadStaticsBuckets7912d7c0 4000639 18 System.Int32[] 0 instance 00000000 m_ThreadStaticsBits
791028f4 400063a 1c ...ation.CultureInfo 0 instance 00000000 m_CurrentCulture
791028f4 400063b 20 ...ation.CultureInfo 0 instance 00000000 m_CurrentUICulture
790fd0f0 400063c 24 System.Object 0 instance 00000000 m_ThreadStartArg
791016bc 400063d 28 System.IntPtr 1 instance 8f69280 DONT_USE_InternalThread
79102290 400063e 2c System.Int32 1 instance 2 m_Priority
79102290 400063f 30 System.Int32 1 instance 11 m_ManagedThreadId
7910a7a8 4000640 168 ...LocalDataStoreMgr 0 shared static s_LocalDataStoreMgr
≫ Domain:Value 000710a8:06c42ef4 08e65d48:00000000 ≪
790fd0f0 4000641 16c System.Object 0 shared static s_SyncObject
≫ Domain:Value 000710a8:017b25d8 08e65d48:0898381c ≪

Although unmanaged, CLR and JIT-code residue is useful for analysis, for example, as shown in Handled Exception (page 144) pattern examples.

Annotated Disassembly (JIT .NET code)

When disassembling JIT code it is good to see annotated function calls with full type and token information:

0:000> !CLRStack
OS Thread Id: 0xbf8 (0)
ESP      EIP
001fef90 003200a4 ClassMain.DoWork()
001fef94 00320082 ClassMain.Main(System.String[])
001ff1b0 79e7c74b [GCFrame: 001ff1b0]

0:000> !U 00320082
Normal JIT generated code
ClassMain.Main(System.String[])
Begin 00320070, size 13
00320070 b960300d00 mov ecx,0D3060h (MT: ClassMain)
00320075 e8a21fdaff call 000c201c (JitHelp: CORINFO_HELP_NEWSFAST)
0032007a 8bc8 mov ecx,eax
0032007c ff159c300d00 call dword ptr ds:[0D309Ch] (ClassMain.DoWork(),
mdToken: 06000002)
≫> 00320082 c3 ret

However, this doesn't work when we disable the output of raw bytes:

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> !U 00320082
Normal JIT generated code
ClassMain.Main(System.String[])
Begin 00320070, size 13
00320070 mov ecx,0D3060h
00320075 call 000c201c
0032007a mov ecx,eax
0032007c call dword ptr ds:[0D309Ch]
≫> 00320082 ret

Here we can still double check JIT-ed function calls manually:

0:000> dd 0D309Ch l1
000d309c 00320098
0:000> !IP2MD 00320098
MethodDesc: 000d3048
Method Name: ClassMain.DoWork()
Class: 000d1180
MethodTable: 000d3060
mdToken: 06000002
Module: 000d2c3c
IsJitted: yes
m_CodeOrIL: 00320098

Wait Chain (Mutex Objects)

This is an additional variation of the general Wait Chain (Volume 1, page 482) pattern where mutexes (mutants) are involved in thread wait chains, for example:

THREAD fffffa8019388b60 Cid 02e8.cfd0 Teb: 000007fffffa2000 Win32Thread:
0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
fffffa800d75daf0 Mutant - owning thread fffffa800ea2ab60
[...]

THREAD fffffa8016abab60 Cid 02e8.ec34 Teb: 000007fffffae000 Win32Thread:
0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
fffffa800d75daf0 Mutant - owning thread fffffa800ea2ab60
[...]

We have seen such dependencies in various previous pattern interaction case studies such as:

• Inconsistent Dump, Stack Trace Collection, LPC, Thread, Process, Executive Resource Wait Chains, Missing Threads and Waiting Thread Time (Volume 5, page 133)

• Inconsistent Dump, Blocked Threads, Wait Chains, Incorrect Stack Trace and Process Factory (Volume 3, page 279)

• Semantic Split pattern example (Volume 3, page 120)

• Insufficient Memory, Handle Leak, Wait Chain, Deadlock, Inconsistent Dump and Overaged System (Volume 3, page 175)

• Blocked GUI Thread, Wait Chain and Virtualized Process (Volume 3, page 170)

• LPC/ALPC Wait Chain pattern example (Volume 3, page 97)

• Mixed object Deadlock pattern example (Volume 3, page 85)

• LPC Deadlock pattern example (Volume 1, page 474)

Another example we show here is an unusual number of mutant dependencies in one complete memory dump from a hang system:

AppA(KTHREAD-1) -> AppB(KTHREAD-2)
AppB(KTHREAD-3) -> ServiceA(KTHREAD-4)
AppA(KTHREAD-5) -> ServiceA(KTHREAD-4)
AppB(KTHREAD-6) -> AppA(KTHREAD-7)
AppB(KTHREAD-6) -> AppC(KTHREAD-8)
AppC(KTHREAD-9) -> ServiceA(KTHREAD-4)
AppC(KTHREAD-10) -> AppB(KTHREAD-11)

Here the notation AppX(N)->AppY(M) means that a thread N from AppX process is waiting for a mutant that is owned by a thread M from AppY process. Because AppB, AppC and ServiceA belonged to the Same Vendor (Volume 5, page 128) it was advised to check with that ISV.

Inline Function Optimization (Managed Code)

In addition to inline function optimization (Volume 2, page 322) of unmanaged and native code we can see similar approach to JIT-compiled code:

public class ClassMain
{
   public bool time2stop = false;

   public static void Main(string[] args)
   {
      new ClassMain().Main();
   }

   public void Main()
   {
      while (!time2stop)
      {
         DoWork();
      }
   }

   volatile int inSensor, outSensor;

   void DoWork()
   {
      outSensor ^= inSensor;
   }
}

0:000> kL
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
001fefa0 79e7c6cc 0×3200a4
001ff020 79e7c8e1 mscorwks!CallDescrWorkerWithHandler+0xa3
001ff160 79e7c783 mscorwks!MethodDesc::CallDescr+0×19c
001ff17c 79e7c90d mscorwks!MethodDesc::CallTargetWorker+0×1f
001ff190 79eefb9e mscorwks!MethodDescCallSite::Call_RetArgSlot+0×18
001ff2f4 79eef830 mscorwks!ClassLoader::RunMain+0×263
001ff55c 79ef01da mscorwks!Assembly::ExecuteMainMethod+0xa6
001ffa2c 79fb9793 mscorwks!SystemDomain::ExecuteMainMethod+0×43f
001ffa7c 79fb96df mscorwks!ExecuteEXE+0×59
001ffac4 736455ab mscorwks!_CorExeMain+0×15c
001ffad0 73747f16 mscoreei!_CorExeMain+0×38
001ffae0 73744de3 mscoree!ShellShim__CorExeMain+0×99
001ffae8 76573833 mscoree!_CorExeMain_Exported+0×8
001ffaf4 77c1a9bd kernel32!BaseThreadInitThunk+0xe
001ffb34 00000000 ntdll!_RtlUserThreadStart+0×23
0:000> r
eax=00000000 ebx=001fefbc ecx=015316e0 edx=0037a238 esi=0037a238
edi=00000000
eip=003200a4 esp=001fef90 ebp=001fefa0 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
003200a4 80790c00 cmp byte ptr [ecx+0Ch],0 ds:0023:015316ec=00

0:000> !IP2MD 003200a4
MethodDesc: 000d3048
Method Name: ClassMain.Main()
Class: 000d1180
MethodTable: 000d3060
mdToken: 06000002
Module: 000d2c3c
IsJitted: yes
m_CodeOrIL: 00320098

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> !U 003200a4
Normal JIT generated code
ClassMain.Main()
Begin 00320098, size 13
00320098 cmp byte ptr [ecx+0Ch],0
0032009c jne 003200aa
0032009e mov eax,dword ptr [ecx+4]
003200a1 xor dword ptr [ecx+8],eax
≫> 003200a4 cmp byte ptr [ecx+0Ch],0
003200a8 je 0032009e
003200aa ret

We see that DoWork code was inlined into Main function code.

Technology-Specific Subtrace (JIT .NET Code)

When looking at process memory dumps and seeing CLR threads (Volume 4, page 163) we can find fragments of JIT-ed code return addresses on the unmanaged stack trace:

0:011> kL
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0b73e120 057223e2 0×572240f
0b73e134 6af44a2a 0×57223e2
0b73e1b0 6af44bcc clr!CallDescrWorkerWithHandler+0×8e
0b73e2f0 6af44c01 clr!MethodDesc::CallDescr+0×194
0b73e30c 6af44c21 clr!MethodDesc::CallTargetWorker+0×21
0b73e324 6afb7856 clr!MethodDescCallSite::Call+0×1c
0b73e4e8 6afb7ba3 clr!CallWithValueTypes_RetArgSlotWrapper+0×5c
0b73e7b4 6afb7d65 clr!InvokeImpl+0×621
0b73e880 6963d689 clr!RuntimeMethodHandle::InvokeMethodFast+0×180
0b73e8d4 6963d3d0 mscorlib_ni+0×2bd689
0b73e90c 6963bfed mscorlib_ni+0×2bd3d0
0b73e934 69643284 mscorlib_ni+0×2bbfed
0b73e958 6af3de7e mscorlib_ni+0×2c3284
0b73eb64 05720988 clr!ListLockEntry::Release+0×68
0b73ebc0 6962ae5b 0×5720988
0b73ebd0 695b7ff4 mscorlib_ni+0×2aae5b
0b73ebec 695b7f34 mscorlib_ni+0×237ff4
0b73ec0c 6962ade8 mscorlib_ni+0×237f34
0b73ec24 6af221db mscorlib_ni+0×2aade8
0b73ec34 6af44a2a clr!CallDescrWorker+0×33
0b73ecb0 6af44bcc clr!CallDescrWorkerWithHandler+0×8e
0b73ede8 6af44c01 clr!MethodDesc::CallDescr+0×194
0b73ee04 6b0bb512 clr!MethodDesc::CallTargetWorker+0×21
0b73f010 6afd5c05 clr!ThreadNative::KickOffThread_Worker+0×1e1
0b73f024 6afd5c87 clr!Thread::DoExtraWorkForFinalizer+0×114
0b73f0d4 6afd5d42 clr!Thread::ShouldChangeAbortToUnload+0×101
0b73f134 6afc37a2 clr!Thread::ShouldChangeAbortToUnload+0×399
0b73f140 6b0a6465 clr!Thread::RaiseCrossContextException+0×3f8
0b73f220 6afc37cf clr!Thread::DoADCallBack+0xf0
0b73f240 6afd5c87 clr!Thread::DoExtraWorkForFinalizer+0xfa
0b73f2f0 6afd5d42 clr!Thread::ShouldChangeAbortToUnload+0×101
0b73f350 6afd5dd9 clr!Thread::ShouldChangeAbortToUnload+0×399
0b73f374 6b0bb3e5 clr!Thread::ShouldChangeAbortToUnload+0×43a
0b73f38c 6b0bb2e0 clr!ManagedThreadBase::KickOff+0×15
0b73f424 6afd5a08 clr!ThreadNative::KickOffThread+0×23e
0b73fb44 76573833 clr!Thread::intermediateThreadProc+0×4b
0b73fb50 77c1a9bd kernel32!BaseThreadInitThunk+0xe

With the correct loaded CLR version extension (page 99) we can inspect these addresses and get their method names, module and class addresses using !IP2MD WinDbg SOS extension command:

0:011> !IP2MD 0x572240f
MethodDesc: 057420e8
Method Name: UserQuery+ClassMain.Main()
Class: 057341d8
MethodTable: 05742108
mdToken: 06000004
Module: 05741048
IsJitted: yes
CodeAddr: 05722400
Transparency: Critical

0:011> !IP2MD 0x57223e2
MethodDesc: 0574204c
Method Name: UserQuery.RunUserAuthoredQuery()
Class: 057340a4
MethodTable: 0574206c
mdToken: 06000001
Module: 05741048
IsJitted: yes
CodeAddr: 057223d0
Transparency: Critical

0:011> !IP2MD 0x5720988
MethodDesc: 056e601c
Method Name: LINQPad.ExecutionModel.Server.StartClrQuery()
Class: 0571f6e4
MethodTable: 056e60e4
mdToken: 06000c59
Module: 056e336c
IsJitted: yes
CodeAddr: 05720910
Transparency: Critical

These method calls can also be seen on a managed stack trace (page 115):

0:011> !CLRStack
OS Thread Id: 0xac (11)
Child SP IP Call  Site
0b73e120 0572240f UserQuery+ClassMain.Main()
0b73e128 057223e2 UserQuery.RunUserAuthoredQuery()
0b73e674 6af221db [DebuggerU2MCatchHandlerFrame: 0b73e674]
0b73e640 6af221db [CustomGCFrame: 0b73e640]
0b73e614 6af221db [GCFrame: 0b73e614]
0b73e5f8 6af221db [GCFrame: 0b73e5f8]
0b73e81c 6af221db [HelperMethodFrame_PROTECTOBJ: 0b73e81c]
System.RuntimeMethodHandle._InvokeMethodFast(System.IRuntimeMethodInfo,
System.Object, System.Object[], System.SignatureStruct ByRef,
System.Reflection.MethodAttributes, System.RuntimeType)
0b73e898 6963d689
System.RuntimeMethodHandle.InvokeMethodFast(System.IRuntimeMethodInfo,
System.Object, System.Object[], System.Signature,
System.Reflection.MethodAttributes, System.RuntimeType)
0b73e8ec 6963d3d0
System.Reflection.RuntimeMethodInfo.Invoke(System.Object,
System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[],
System.Globalization.CultureInfo, Boolean)
0b73e928 6963bfed
System.Reflection.RuntimeMethodInfo.Invoke(System.Object,
System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[],
System.Globalization.CultureInfo)
0b73e94c 69643284 System.Reflection.MethodBase.Invoke(System.Object,
System.Object[])
0b73e958 0572134c LINQPad.ExecutionModel.Server.RunClrQuery()
0b73eb6c 05720988 LINQPad.ExecutionModel.Server.StartClrQuery()
0b73ebc8 6962ae5b
System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0b73ebd8 695b7ff4
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object, Boolean)
0b73ebfc 695b7f34
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object)
0b73ec18 6962ade8 System.Threading.ThreadHelper.ThreadStart()
0b73ee30 6af221db [GCFrame: 0b73ee30]
0b73f0f4 6af221db [DebuggerU2MCatchHandlerFrame: 0b73f0f4]
0b73f18c 6af221db [ContextTransitionFrame: 0b73f18c]
0b73f310 6af221db [DebuggerU2MCatchHandlerFrame: 0b73f310]

Double IRP Completion

Similar to Double Free (process heap, Volume 1, page 378) and Double Free (kernel pool, Volume 1, page 387) that might be detected through instrumentation (Volume 5, page 108) such as gflags and Driver Verifier there is also an IRP double completion variant implemented through Self-Diagnosis (kernel mode, page 89). Here's a typical example:

0: kd> !analyze -v

[...]

MULTIPLE_IRP_COMPLETE_REQUESTS (44)
A driver has requested that an IRP be completed (IoCompleteRequest()), but
the packet has already been completed. This is a tough bug to find because
the easiest case, a driver actually attempted to complete its own packet
twice, is generally not what happened. Rather, two separate drivers each
believe that they own the packet, and each attempts to complete it. The
first actually works, and the second fails. Tracking down which drivers in
the system actually did this is difficult, generally because the trails of
the first driver have been covered by the second. However, the driver
stack for the current request can be found by examining the DeviceObject
fields in each of the stack locations.
Arguments:
Arg1: fffffa80104aa010, Address of the IRP
Arg2: 0000000000000eae
Arg3: 0000000000000000
Arg4: 0000000000000000

STACK_TEXT:
fffff880`0e322428 fffff800`01666224 : 00000000`00000044 fffffa80`104aa010
00000000`00000eae 00000000`00000000 : nt!KeBugCheckEx
fffff880`0e322430 fffff880`03dd121f : fffffa80`0dc12c50 fffffa80`107750c8
fffffa80`104aa010 fffff880`0e322580 : nt! ?? ::FNODOBFM::`string`+0x3eb3d
fffff880`0e322520 fffff880`03def17f : fffffa80`0dc12c50 fffffa80`104aa010
fffffa80`0cacb610 00000000`00000001 : DriverA!DriverA::Create+0x3bf
[...]
fffff880`0e322740 fffff800`01972ba4 : fffffa80`0dc129f0 00000000`00000000
fffffa80`0fe7a010 00000000`00000001 : nt!IopParseDevice+0x5a7
fffff880`0e3228d0 fffff800`01977b7d : fffffa80`0fe7a010 fffff880`0e322a30
fffffa80`00000040 fffffa80`0cae5080 : nt!ObpLookupObjectName+0x585
fffff880`0e3229d0 fffff800`0197e647 : 00000000`000007ff 00000000`00000003
fffff8a0`05716d01 00000000`00000000 : nt!ObOpenObjectByName+0x1cd
fffff880`0e322a80 fffff800`01988398 : 00000000`03f3e510 fffff8a0`c0100000
fffff8a0`0c26fe50 00000000`03f3e118 : nt!IopCreateFile+0x2b7
fffff880`0e322b20 fffff800`0167b813 : fffffa80`0e10db30 00000000`00000001
fffffa80`1002b060 fffff800`0198f294 : nt!NtCreateFile+0x78
fffff880`0e322bb0 00000000`772efc0a : 000007fe`f62c358f 00000000`03f3e1b0
00000000`7719fd72 000007fe`f62c6490 : nt!KiSystemServiceCopyEnd+0x13
00000000`03f3e068 000007fe`f62c358f : 00000000`03f3e1b0 00000000`7719fd72
000007fe`f62c6490 00000000`00000005 : ntdll!NtCreateFile+0xa

[...]

0: kd> !irp fffffa80104aa010
Irp is active with 1 stacks 3 is current (= 0xfffffa80104aa170)
No Mdl: No System Buffer: Thread fffffa801002b060: Irp is completed.
Pending has been returned
cmd flg cl Device File Completion-Context
[ 0, 0] 0 2 fffffa800dc129f0 00000000 00000000-00000000
DriverDriverA
Args: 00000000 00000000 00000000 ffffffffc00a0006

---

⁶ http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

⁷ http://www.dumpanalysis.org/blog/index.php/2010/09/21/contention-patterns/

⁸ http://www.dumpanalysis.org/blog/index.php/2009/02/17/wait-chain-patterns/

⁹ http://www.imagemagick.org/

¹⁰ http://msdn.microsoft.com/en-us/library/t48aek43(v=VS.80).aspx

¹¹ http://www.dumpanalysis.org/blog/index.php/2009/02/17/dynamic-memorycorruption-patterns/

¹² http://msdn.microsoft.com/en-us/library/aa374153(v=vs.85).aspx

¹³ http://doxygen.reactos.org/d4/dbc/lib_2rtl_2actctx_8c_a52533b501a01935d624ca160b7dd7dc7.html

¹⁴ http://www.stevestechspot.com/default.aspx