When we dump a stack trace collection (Volume 1, page 409) from a complete or kernel memory dump we see lots of synchronization and notification events, for example:
THREAD fffffa8003d33120 Cid 0734.0868 Teb: 000007fffffd4000 Win32Thread: fffff900c07182e0 WAIT: (UserRequest) UserMode Alertable fffffa8003413d20 NotificationEvent fffffa80020b5170 NotificationEvent fffffa80017f31e0 NotificationEvent fffffa80013f8cf0 NotificationEvent fffffa8002547ee0 NotificationEvent fffffa8002547e80 NotificationEvent fffffa8004186100 NotificationEvent fffffa8003dcfa80 NotificationEvent fffffa8003df6870 NotificationEvent fffffa8003bbd5e0 NotificationEvent fffffa8003b5d4e0 NotificationEvent fffffa800390c690 NotificationEvent fffffa8003dbc410 NotificationEvent fffffa8003dbc3b0 NotificationEvent fffffa8004041040 NotificationEvent fffffa8003dde8a0 NotificationEvent fffffa80038f4530 NotificationEvent fffffa800401fa50 NotificationEvent fffffa800398a550 NotificationEvent fffffa8003b587e0 NotificationEvent fffffa800398d200 SynchronizationEvent IRP List: fffffa800150b010: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa80015c1010: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa80017f53a0: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa80014ccca0: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa80011fa710: (0006,03a0) Flags: 00060000 Mdl: 00000000 fffffa80011d6070: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa80030b5450: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa8004149810: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa800419b500: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa80040c2520: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa8003b75520: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa8004082ca0: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa8004082010: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa800403aa40: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa800403a010: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa8003dbc9d0: (0006,0358) Flags: 00060000 Mdl: 00000000 fffffa8003dbc010: (0006,0358) Flags: 00060000 Mdl: 00000000 Not impersonating DeviceMap fffff88001697690 Owning Process fffffa80039bac10 Image: explorer.exe Attached Process N/A Image: N/A Wait Start TickCount 41897 Ticks: 561 (0:00:00:08.765) Context Switch Count 4744 LargeStack UserTime 00:00:00.187 KernelTime 00:00:01.218 Win32 Start Address SHLWAPI!WrapperThreadProc (0×000007fefe854f20) Stack Init fffff9800ef2fdb0 Current fffff9800ef2f260 Base fffff9800ef30000 Limit fffff9800ef27000 Call 0 Priority 12 BasePriority 8 PriorityDecrement 2 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff980`0ef2f2a0 fffff800`0185d695 nt!KiSwapContext+0×84 fffff980`0ef2f3e0 fffff800`0185ad2f nt!KiSwapThread+0×125 fffff980`0ef2f440 fffff800`01ac1813 nt!KeWaitForMultipleObjects+0×703 fffff980`0ef2f4b0 fffff800`01ac1a03 nt!ObpWaitForMultipleObjects+0×216 fffff980`0ef2f960 fffff800`0184dcf3 nt!NtWaitForMultipleObjects+0xe2 fffff980`0ef2fbb0 00000000`776f082a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff980`0ef2fc20) 00000000`03acf718 00000000`774ced73 ntdll!NtWaitForMultipleObjects+0xa 00000000`03acf720 00000000`775ee97d kernel32!WaitForMultipleObjectsEx+0×10b 00000000`03acf830 00000000`775ee86e USER32!RealMsgWaitForMultipleObjectsEx+0×129 00000000`03acf8d0 000007fe`fed47a9a USER32!MsgWaitForMultipleObjectsEx+0×46 00000000`03acf910 000007fe`fe854d48 SHELL32!CChangeNotify::ThreadProc+0xba 00000000`03acfb90 00000000`774ccdcd SHLWAPI!WrapperThreadProc+0xfc 00000000`03acfc70 00000000`776ec6e1 kernel32!BaseThreadInitThunk+0xd 00000000`03acfca0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
The problem with such synchronization objects as events is that they don't have an owner field as in other structures:
1: kd> dt -r _KEVENT ntdll!_KEVENT +0x000 Header : _DISPATCHER_HEADER +0x000 Type : UChar +0x001 Abandoned : UChar +0x001 Absolute : UChar +0x001 NpxIrql : UChar +0x001 Signalling : UChar +0x002 Size : UChar +0x002 Hand : UChar +0x003 Inserted : UChar +0x003 DebugActive : UChar +0x003 DpcActive : UChar +0x000 Lock : Int4B +0x004 SignalState : Int4B +0x008 WaitListHead : _LIST_ENTRY +0x000 Flink : Ptr64 _LIST_ENTRY +0x008 Blink : Ptr64 _LIST_ENTRY
1: kd> dt _KMUTANT nt!_KMUTANT +0x000 Header : _DISPATCHER_HEADER +0x018 MutantListEntry : _LIST_ENTRY +0×028 OwnerThread : Ptr64 _KTHREAD +0×030 Abandoned : UChar +0×031 ApcDisable : UChar
Fortunately many of such events are created to wait for asynchronous I/O and their addresses are stored in IRP structures that also have an associated thread. For example, in the thread above we find one of notification events, fffffa80020b5170, in an IRP fffffa800150b010:
1: kd> !irp fffffa800150b010 -v Irp is active with 9 stacks 9 is current (= 0xfffffa800150b320) No Mdl: No System Buffer: Thread fffffa8003d33120: Irp stack trace. Flags = 00060000 ThreadListEntry.Flink = fffffa80015c1030 ThreadListEntry.Blink = fffffa8003d334d8 IoStatus.Status = 00000000 IoStatus.Information = fffff88002ac09c0 RequestorMode = 00000001 Cancel = 00 CancelIrql = 0 ApcEnvironment = 00 UserIosb = 04127ca0 UserEvent = fffffa80020b5170 Overlay.AsynchronousParameters.UserApcRoutine = 00000000 Overlay.AsynchronousParameters.UserApcContext = 04127ca0 Overlay.AllocationSize = 00000000 - 00000000 CancelRoutine = fffff80001912fc0 nt!FsRtlCancelNotify UserBuffer = 04127498 & Tail.Overlay.DeviceQueueEntry = fffffa800150b088 Tail.Overlay.Thread = fffffa8003d33120 Tail.Overlay.AuxiliaryBuffer = 00000000 Tail.Overlay.ListEntry.Flink = fffff88002ac0a00 Tail.Overlay.ListEntry.Blink = fffff88002ac0a00 Tail.Overlay.CurrentStackLocation = fffffa800150b320 Tail.Overlay.OriginalFileObject = fffffa8002390770 Tail.Apc = 00000000 Tail.CompletionKey = 00000000 cmd flg cl Device File Completion-Context [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 > [ c, 2] 0 1 fffffa8001bcb030 fffffa8002390770 00000000- 00000000 pending FileSystemNtfs Args: 00000800 00000015 00000000 00000000
Another example is when IRP fffffa80022d4410 was created by one thread fffffa8002119700 but another thread fffffa8001fda450 is waiting for its associated event fffffa8002093190:
1: kd> !irpfind Irp [ Thread ] irpStack: (Mj,Mn) DevObj [Driver] MDL Process fffffa8002f2d980 [00000000] irpStack: ( f, 0) fffffa8002f6f050 [ Driverusbehci] fffffa800200dc40 [fffffa80030a0710] irpStack: ( d, 0) fffffa8001f192d0 [ FileSystemNpfs] fffffa800203d280 [fffffa80035b1b10] irpStack: ( e, 0) fffffa8001c09b50 [ Driver etbt] 0xfffffa8002f10970 fffffa800228e4e0 [00000000] irpStack: ( e, 0) fffffa8002be1840 [ DriverCmBatt] fffffa800229c6a0 [00000000] irpStack: ( f, 0) fffffa8002f2c050 [ Driverusbuhci] fffffa800229a6a0 [00000000] irpStack: ( f, 0) fffffa8002f2c050 [ Driverusbuhci] fffffa80022946a0 [00000000] irpStack: ( f, 0) fffffa8002f2c050 [ Driverusbuhci] fffffa80022a28a0 [00000000] irpStack: ( f, 0) fffffa80022d04c0 [ Driverusbhub] fffffa80022a6230 [00000000] irpStack: ( f, 0) fffffa8002f2c050 [ Driverusbuhci] fffffa80022d4410 [fffffa8002119700] irpStack: ( e,2d) fffffa80022ee720 [ DriverAFD] [...]
1: kd> !irp fffffa80022d4410 -v Irp is active with 4 stacks 4 is current (= 0xfffffa80022d45b8) No Mdl: System buffer=fffffa8001fda150: Thread fffffa8002119700: Irp stack trace. Flags = 00060030 ThreadListEntry.Flink = fffffa8002396c80 ThreadListEntry.Blink = fffffa8002119ab8 IoStatus.Status = 00000000 IoStatus.Information = 00000000 RequestorMode = 00000001 Cancel = 00 CancelIrql = 0 ApcEnvironment = 00 UserIosb = 7fefc914858 UserEvent = fffffa8002093190 Overlay.AsynchronousParameters.UserApcRoutine = 00000000 Overlay.AsynchronousParameters.UserApcContext = 7fefc914858 Overlay.AllocationSize = 00000000 - 00000000 CancelRoutine = fffff980044ad250 afd!AfdCancelAddressListChange UserBuffer = 00000000 & Tail.Overlay.DeviceQueueEntry = fffffa80022d4488 Tail.Overlay.Thread = fffffa8002119700 Tail.Overlay.AuxiliaryBuffer = 00000000 Tail.Overlay.ListEntry.Flink = 00000000 Tail.Overlay.ListEntry.Blink = 00000000 Tail.Overlay.CurrentStackLocation = fffffa80022d45b8 Tail.Overlay.OriginalFileObject = fffffa8002bbe050 Tail.Apc = 00000000 Tail.CompletionKey = 00000000 cmd flg cl Device File Completion-Context [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 > [ e,2d] 5 1 fffffa80022ee720 fffffa8002bbe050 00000000- 00000000 pending DriverAFD Args: fffffa8002bbef50 fffffa8002bbef50 fffff9800446cf00 fffffa800250d220
THREAD fffffa8002119700 Cid 0310.0318 Teb: 000007fffffdc000 Win32Thread: fffff900c07b6d60 WAIT: (DelayExecution) UserMode Non-Alertable fffffa80021197b8 NotificationTimer IRP List: fffffa80022d4410: (0006,03a0) Flags: 00060030 Mdl: 00000000 fffffa8002396c60: (0006,03a0) Flags: 00060030 Mdl: 00000000 Not impersonating DeviceMap fffff880017b4c70 Owning Process fffffa800209f880 Image: svchost.exe Attached Process N/A Image: N/A Wait Start TickCount 42336 Ticks: 122 (0:00:00:01.906) Context Switch Count 159 LargeStack UserTime 00:00:00.031 KernelTime 00:00:00.140 Win32 Start Address ADVAPI32!ScSvcctrlThreadW (0×000007fefe2b4bd0) Stack Init fffff9800ccabdb0 Current fffff9800ccab990 Base fffff9800ccac000 Limit fffff9800cca6000 Call 0 Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff980`0ccab9d0 fffff800`0185d695 nt!KiSwapContext+0×84 fffff980`0ccabb10 fffff800`0185bbe9 nt!KiSwapThread+0×125 fffff980`0ccabb70 fffff800`01a8b1cd nt!KeDelayExecutionThread+0×339 fffff980`0ccabbf0 fffff800`0184dcf3 nt!NtDelayExecution+0×5c fffff980`0ccabc20 00000000`776f05ba nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff980`0ccabc20) 00000000`000ff9d8 00000000`774cd908 ntdll!NtDelayExecution+0xa 00000000`000ff9e0 000007fe`fc8ba8c0 kernel32!SleepEx+0×84 00000000`000ffa60 000007fe`fc8b17bd rpcss!ObjectExporterWorkerThread+0×50b 00000000`000ffb30 000007fe`fc8b27f2 rpcss!ScmServiceMain+0xe4 00000000`000ffb60 00000000`ffaa1771 rpcss!ServiceMain+0×251 00000000`000ffe20 000007fe`fe2b4bf5 svchost!ServiceStarter+0×1ea 00000000`000ffeb0 00000000`774ccdcd ADVAPI32!ScSvcctrlThreadW+0×25 00000000`000ffee0 00000000`776ec6e1 kernel32!BaseThreadInitThunk+0xd 00000000`000fff10 00000000`00000000 ntdll!RtlUserThreadStart+0×1d
THREAD fffffa8001fda450 Cid 0310.031c Teb: 000007fffffda000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Alertable fffffa8002012ef0 SynchronizationTimer fffffa800208d7a0 SynchronizationEvent fffffa80023bb820 SynchronizationEvent fffffa80023bb740 SynchronizationEvent fffffa8001fd9730 SynchronizationEvent fffffa8002093190 SynchronizationEvent fffffa8001a0eee0 SynchronizationEvent Not impersonating DeviceMap fffff880017b4c70 Owning Process fffffa800209f880 Image: svchost.exe Attached Process N/A Image: N/A Wait Start TickCount 5767 Ticks: 36691 (0:00:09:33.296) Context Switch Count 8 UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address ntdll!TppWaiterpThread (0×00000000776c6930) Stack Init fffff9800d2c0db0 Current fffff9800d2c0260 Base fffff9800d2c1000 Limit fffff9800d2bb000 Call 0 Priority 10 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 Kernel stack not resident. Child-SP RetAddr Call Site fffff980`0d2c02a0 fffff800`0185d695 nt!KiSwapContext+0×84 fffff980`0d2c03e0 fffff800`0185ad2f nt!KiSwapThread+0×125 fffff980`0d2c0440 fffff800`01ac1813 nt!KeWaitForMultipleObjects+0×703 fffff980`0d2c04b0 fffff800`01ac1a03 nt!ObpWaitForMultipleObjects+0×216 fffff980`0d2c0960 fffff800`0184dcf3 nt!NtWaitForMultipleObjects+0xe2 fffff980`0d2c0bb0 00000000`776f082a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff980`0d2c0c20) 00000000`00d1fb08 00000000`776c6b07 ntdll!NtWaitForMultipleObjects+0xa 00000000`00d1fb10 00000000`774ccdcd ntdll!TppWaiterpThread+0×14d 00000000`00d1fdb0 00000000`776ec6e1 kernel32!BaseThreadInitThunk+0xd 00000000`00d1fde0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d