Appendix

Memory Analysis as a Service

MAaaS includes 2 complementary DA+TA services:

1. Dump Analysis as a Service (DAaaS)

2. Trace Analysis as a Service (TAaaS)

Software Diagnostics Services (former Memory Dump Analysis Services)¹²⁵ is the first organization to provide such a service at an audit and certification levels.

Stack Overflow Patterns

• Stack Overflow (kernel mode) – Volume 1, page 314

• Stack Overflow (user mode) – Volume 2, page 279

• Stack Overflow (software implementation) – page 82

.NET / CLR / Managed Space Patterns

• CLR Thread (Volume 4, page 163)

• Managed Code Exception (Volume 1, page 331)

• Nested Exceptions (managed code) (Volume 2, page 310)

• Mixed Exception (Volume 4, page 145)

• Memory Leak (.NET heap) (Volume 1, page 371)

• JIT Code (.NET) (Volume 3, page 132)

• Managed Stack Trace (page 115)

• Multiple Exceptions (managed space) (page 104)

• Version-Specific Extension (page 99)

• Caller-n-Callee (page 138)

• Technology-Specific Subtrace (JIT .NET code) (page 157)

• Inline Function Optimization (managed code) (page 155)

• Annotated Disassembly (JIT .NET code) (page 151)

• Handled Exception (.NET CLR) (page 144)

• Execution Residue (managed space) (page 149)

• Deadlock (managed space) (page 135)

• Duplicate Extension (page 131)

• Stack Trace Collection (managed space) (page 127)

• Dynamic Memory Corruption (managed heap) (page 124)

• Special Thread (.NET CLR) (page 123)

Stack Trace Patterns

• Stack Trace (Volume 1, page 395)

• Stack Trace Collection (unmanaged space, Volume 1, page 409)

• Special Stack Trace (Volume 1, page 479)

• Exception Stack Trace (Volume 4, page 337)

• Dual Stack Trace (page 52)

• Truncated Stack Trace (page 86)

• Managed Stack Trace (page 115)

• Incorrect Stack Trace (Volume 1, page 288)

• Stack Trace Set (page 120)

• Stack Trace Collection (managed space) (page 127)

Symbol Patterns

• No Component Symbols (Volume 1, page 298)

• Coincidental Symbolic Information (Volume 1, page 390)

• Incorrect Symbolic Information (Volume 5, page 71)

Analysis Compass

The Software Behavior Analysis Patterns and Checklists: Corporate Reference Poster Edition for Windows Crash and Hang Memory Dump and Software Trace Analysis (ISBN: 978-1908043030) features “Analysis Compass” based on the reversal of an extended version of pattern-to-command table published in the printed version of Debugged! MZ/PE March, 2009 issue¹²⁶. Here's a sneak preview screenshot of this feature:

The notation F“Function” means searching for frames having the given Function name in stack traces from Stack Trace Collection (Volume 1, page 409). S“String” means searching for String in the output of WinDbg commands, for example, !process 0 ff.

Software Trace Analysis Checklist

Because the number of software trace patterns¹²⁷ is growing there's another checklist in addition to memory dump analysis checklist¹²⁸. The goal is to help experienced engineers not to miss any important information. The checklist doesn't prescribe any specific steps, just lists all possible points to double check when looking at a software trace.

General:

• Check overall trace time delta (Volume 5, page 282)

• Check no trace metafile (Volume 5, page 296) message density (Volume 4, page 335)

• Check whether a trace is a multi-part or a circular (Volume 3, page 346)

• Check for basic facts (Volume 3, page 345) and the story (software narrative)

• Check for any exceptions (Volume 4, page 337), non-false positive errors (Volume 5, page 303) and periodic errors (Volume 3, page 344)

• Check for significant events (Volume 5, page 281)

• Check for discontinuities (Volume 4, page 341) in the time domain

• Check for message current (Volume 4, page 335) and acceleration (Volume 5, page 284) in the frequency domain

Crash Dump Analysis Checklist

General:

• Symbol servers (.symfix)

• Internal database(s) search

• Google or Microsoft search for suspected components as this could be a known issue. Sometimes a simple search immediately points to the fix on a vendor's site

• The tool used to save a dump (to flag false positive, incomplete or inconsistent dumps)

• OS/SP version (version)

• Language

• Debug time

• System uptime

• Computer name (dS srv!srvcomputername or !envvar COMPUTERNAME)

• List of loaded and unloaded modules (Imv or !dlls)

• Hardware configuration (!sysinfo)

• .kframes 100

Application crash or hang:

• Default analysis (!analyze-v or !analyze-v -hang for hangs)

• Critical sections (!cs -s -l –o, !locks) for both crashes and hangs

• Component timestamps, duplication and paths. DLL Hell? (lmv and !dlls)

• Do any newer components exist?

• Process threads (~^*kv or !uniqstack) for multiple exceptions and blocking functions

• Process uptime

• Your components on the full raw stack of the problem thread

• Your components on the full raw stack of the main application thread

• Process size

• Number of threads

• Gflags value (!gflag)

• Time consumed by thread (!runaway)

• Environment (!peb)

• Import table (!dh)

• Hooked functions (!chkimg)

• Exception handlers (!exchain)

• Computer name (!envvar COMPUTERNAME)

• Process heap stats and validation (!heap -s, !heap -s -v)

• CLR threads? (mscorwks or clr modules on stack traces) Yes: use .NET checklist below

• Hidden (unhandled and handled) exceptions on thread raw stacks

System hang:

• Default analysis (!analyze-v -hang)

• ERESOURCE contention (!locks)

• Processes and virtual memory including session space (!vm 4)

• Important services are present and not hanging (for example, terminal or IMA services for Citrix environments)

• Pools (!poolused)

• Waiting threads (!stacks)

• Critical system queues (!exqueue f)

• I/O (!irpfind)

• The list of all thread stack traces (!process 0ff for W2K3/XP/Vista/W2K8, ListProcessStacks script for W2K, Volume 1, page 222)

• LPC/ALPC chain for suspected threads (!lpc message or !alpc/m after search for “Waiting for reply to LPC” or “Waiting for reply to ALPC” in !process 0 ff output)

• Mutants (search for “Mutants - owning thread” in !process 0 ff output)

• Critical sections for suspected processes (!cs -l -o -s )

• Sessions, session processes (!session, !sprocess)

• Processes (size, handle table size) (!process 0 0)

• Running threads (!running)

• Ready threads (!ready)

• DPC queues (!dpcs)

• The list of APCs (!apc)

• Internal queued spinlocks (!qlocks)

• Computer name (dS srv!srvcomputername)

• File cache, VACB (!filecache)

• File objects for blocked thread IRPs (!irp -> !fileobj)

BSOD:

• Default analysis (!analyze-v)

• Pool address (!pool)

• Component timestamps.

• Processes and virtual memory (!vm 4)

• Current threads on other processors

• Raw stack

• Bugcheck description (including ln exception address for corrupt or truncated dumps)

• Bugcheck callback data (!bugdump for systems prior to Windows XP SP1)

• Bugcheck secondary callback data (.enumtag)

• Computer name (dS srv!srvcomputername)

• Hardware configuration (!sysinfo)

.NET application or service:

• CLR module and SOS extension versions (Imv and .chain)

• Managed exceptions (~*e !pe)

• Nested managed exceptions (!pe -nested)

• Managed threads (!Threads -special)

• Managed stack traces (~*e !CLRStack)

• Managed execution residue (~*e !DumpStackObjects and !DumpRuntimeTypes)

• Managed heap (!VerifyHeap, !DumpHeap -stat and !eeheap -gc)

• GC handles (!GCHandIes)

• Finalizer queue (!FinalizeQueue)

• Sync blocks (!syncblk)

---

¹²⁵ http://www.DumpAnalysis.com

¹²⁶ http://www.debuggingexperts.com/debugged-march-09

¹²⁷ http://www.dumpanalysis.org/blog/index.php/trace-analysis-patterns/

¹²⁸ http://www.dumpanalysis.org/blog/index.php/2007/06/20/crash-dump-analysis-checklist/