PART 13: Miscellaneous
GI Index of Memory Dump Analysis
GI Index (Google Insights for Search) shows that crash dump analysis and debugging slowly decline in popularity:
GI for memory dump analysis searches coincides with the birth of DumpAnalysis.org in later 2006 (the only mystery for me is mid 2005 peak):
The birth of memory dump analysis as a discipline correlates with the emergence of memory forensics in the early 2006:
We also see that malware analysis is on the rise:
Perhaps mid 2005 peak for memory dump analysis searches coincides with the interest in malware analysis.
The New School of Debugging
This is a new initiative to integrate traditional multidisciplinary debugging103 approaches and methodologies with multiplatform pattern-driven software problem solving104, unified debugging patterns105, best practices in memory dump analysis (page 19) and software tracing106, computer security, economics, and the new emerging trends (page 256).
TestWER Tool to Test Windows Error Reporting
TestDefaultDebugger107 had been updated to version 2.0 with the following new features:
• Renamed to TestWER to reflect the shift from default debuggers to WER and LocalDumps108 (W2K8, Vista and Windows 7)
• The ability to generate software exceptions (by default TestWER generates hardware NULL pointer access violation exception)
• An option to handle generated exceptions (hardware and software) to test such scenarios
• Symbol PDB files are included in the package
Here's an example stack trace when we select a software exception option and push the big button:
0:000> !analyze -v [...] EXCEPTION_RECORD: 0012f4a4 -- (.exr 0x12f4a4) ExceptionAddress: 7742fbae (kernel32!RaiseException+0x00000058) ExceptionCode: e06d7363 (C++ EH exception) ExceptionFlags: 00000001 NumberParameters: 3 Parameter[0]: 19930520 Parameter[1]: 0012f83c Parameter[2]: 00428ef8 pExceptionObject: 0012f83c _s_ThrowInfo : 00428ef8 Type : int [...] 0:000> k ChildEBP RetAddr 0012edb4 775c5610 ntdll!KiFastSystemCallRet 0012edb8 7743a5d7 ntdll!NtWaitForMultipleObjects+0xc 0012ee54 7743a6f0 kernel32!WaitForMultipleObjectsEx+0x11d 0012ee70 774ae2a5 kernel32!WaitForMultipleObjects+0x18 0012eedc 774ae4d1 kernel32!WerpReportFaultInternal+0x16d 0012eef0 7748ff4d kernel32!WerpReportFault+0x70 0012ef7c 004187ee kernel32!UnhandledExceptionFilter+0x1b5 0012f2b4 004161e8 TestWER!abort+0x10f 0012f2e4 004130cb TestWER!terminate+0x33 0012f2ec 7748febe TestWER!__CxxUnhandledExceptionFilter+0x3c 0012f374 775d7fc1 kernel32!UnhandledExceptionFilter+0x127 0012f37c 77569bdc ntdll!__RtlUserThreadStart+0x6f 0012f390 77564067 ntdll!_EH4_CallFilterFunc+0x12 0012f3b8 775c5f79 ntdll!_except_handler4+0x8e 0012f3dc 775c5f4b ntdll!ExecuteHandler2+0x26 0012f48c 775c5dd7 ntdll!ExecuteHandler+0x24 0012f48c 7742fbae ntdll!KiUserExceptionDispatcher+0xf 0012f7e0 00412ac2 kernel32!RaiseException+0×58 0012f818 0041ff1f TestWER!_CxxThrowException+0×48 0012f85c 00403620 TestWER!CTestDefaultDebuggerDlg::OnBnClickedButton1+0xaf 0012f86c 0040382f TestWER!_AfxDispatchCmdMsg+0×45 0012f89c 00402df6 TestWER!CCmdTarget::OnCmdMsg+0×11c 0012f8c0 00409ef8 TestWER!CDialog::OnCmdMsg+0×1d 0012f910 0040a792 TestWER!CWnd::OnCommand+0×92 0012f9ac 00406dea TestWER!CWnd::OnWndMsg+0×36 0012f9cc 00409713 TestWER!CWnd::WindowProc+0×24 0012fa34 004097a2 TestWER!AfxCallWndProc+0×9a 0012fa54 7716fd72 TestWER!AfxWndProc+0×36 0012fa80 7716fe4a user32!InternalCallWinProc+0×23 0012faf8 77170943 user32!UserCallWinProcCheckWow+0×14b 0012fb38 77170b36 user32!SendMessageWorker+0×4b7 0012fb58 747eb4ba user32!SendMessageW+0×7c 0012fb78 747eb51c comctl32!Button_NotifyParent+0×3d 0012fb94 747eb627 comctl32!Button_ReleaseCapture+0×112 0012fbf4 7716fd72 comctl32!Button_WndProc+0xa98 0012fc20 7716fe4a user32!InternalCallWinProc+0×23 0012fc98 7717018d user32!UserCallWinProcCheckWow+0×14b 0012fcfc 7717022b user32!DispatchMessageWorker+0×322 0012fd0c 771651f5 user32!DispatchMessageW+0xf 0012fd30 0040618c user32!IsDialogMessageW+0×586 0012fd44 004071e2 TestWER!CWnd::IsDialogMessageW+0×32 0012fd50 00402dd3 TestWER!CWnd::PreTranslateInput+0×2d 0012fd64 00408dc1 TestWER!CDialog::PreTranslateMessage+0×9a 0012fd78 00403f35 TestWER!CWnd::WalkPreTranslateTree+0×23 0012fd90 00404090 TestWER!AfxInternalPreTranslateMessage+0×41 0012fd9c 00403f80 TestWER!CWinThread::PreTranslateMessage+0xd 0012fda8 004040dd TestWER!AfxPreTranslateMessage+0×19 0012fdb8 00408598 TestWER!AfxInternalPumpMessage+0×2d 0012fde0 00403440 TestWER!CWnd::RunModalLoop+0xc5 0012fe2c 004204ee TestWER!CDialog::DoModal+0×12d 0012fee4 0041fdf5 TestWER!CTestDefaultDebuggerApp::InitInstance+0xce 0012fef8 0040fc3e TestWER!AfxWinMain+0×48 0012ff88 7743d0e9 TestWER!__tmainCRTStartup+0×112 0012ff94 775a19bb kernel32!BaseThreadInitThunk+0xe 0012ffd4 775a198e ntdll!__RtlUserThreadStart+0×23 0012ffec 00000000 ntdll!_RtlUserThreadStart+0×1b
Here's WinDbg output when we choose to handle exceptions:
0:001> g (628.a90): Access violation - code c0000005 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. TestWER64!CTestDefaultDebuggerDlg::OnBnClickedButton1+0×44: 00000001`40024774 c704250000000000000000 mov dword ptr [0],0 ds:00000000`00000000=???????? 0:000> g (628.a90): C++ EH exception - code e06d7363 (first chance)
Moving to ARM
With Windows moving to ARM platforms we recommend learning ARM assembly language and system programming. Being mostly exposed to PDP-11, VAX-11 and Intel architectures our interest in ARM systems began with porting an old project109 to Windows Mobile and writing a tool called WindowHistory Mobile110 to assist us in debugging. We remember how we were surprised when WinDbg was able to open a crash dump and show ARM code! For ARM studies we highly recommend the book “ARM System Developer's Guide: Designing and Optimizing System Software”
We also plan to create a separate version of or popular “Windows Debugging: Practical Foundations” book111 with ARM examples and an ARM section on Asmpedia112.
The New School of Debugging: What's New
The New School of Debugging places pattern-driven memory and software trace analysis as a solid foundation for any debugging methodology, processes, tools and techniques. Unified Debugging Patterns (page 172) and Generative Debugging Framework (page 173) are additional features of the new school.
A.C.P. Root Cause Analysis Methodology
It's a very simple methodology summarized in just 3 words applicable to any problem solving domain. Its activities include software troubleshooting and debugging:
Artifacts. Checklists. Patterns.
As an example of checklists and patterns please see these 3 presentations related to memory dump and software trace artifacts:
• Introduction to Pattern-Driven Software Problem Solving113
• Fundamentals of Complete Crash and Hang Memory Dump Analysis114
• Pattern-Driven Memory Dump Analysis115
TestWAER Tool to Test Windows Azure Error Reporting
To generate crash dumps for the presentation The Old New Crash: Cloud Memory Dump Analysis116 I created the tool similar to TestWER (page 280). It is a .NET worker service role for Windows Azure that does a NULL pointer assignment. Here is a source code from WorkerRole.cs (EnableCollection is enabled twice just in case):
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Threading;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure.Diagnostics.Management;
namespace TestWAERWorker
{
public class WorkerRole : RoleEntryPoint
{
public override void Run()
{
Trace.WriteLine(“TestWAERWorker entry point called”,
“Information”);
Thread.Sleep(60000);
unsafe
{
int* p = null;
*p = 1;
};
while (true)
{
Thread.Sleep(10000);
Trace.WriteLine(“Working”, “Information”);
}
}
public override bool OnStart()
{
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;
CrashDumps.EnableCollection(true);
DiagnosticMonitorConfiguration config =
DiagnosticMonitor.GetDefaultInitialConfiguration();
string conn_str =
RoleEnvironment.GetConfigurationSettingValue(“Microsoft.WindowsAzure.Plugi
ns.Diagnostics.ConnectionString”);
CloudStorageAccount account =
CloudStorageAccount.Parse(conn_str);
config.Directories.ScheduledTransferPeriod =
TimeSpan.FromMinutes(1);
DiagnosticMonitor diagnosticMonitor =
DiagnosticMonitor.Start(account, config);
CrashDumps.EnableCollection(true);
return base.OnStart();
}
}
}
The forum discussion really helped us to get things right117. So I'd like to thank Oliver Bock here.
To deploy it to Windows Azure please don't forget to add your storage account name and its key in the service configuration file. For your development environment you will need to replace the whole value with “UseDevelopmentStorage=true”.
The package can be downloaded from here118.
103 http://www.dumpanalysis.org/Review+of+Debugging+by+Thinking
104 http://www.dumpanalysis.com/files/Pattern-Driven-Software-Problem-Solving-Introduction.pdf
105 http://www.dumpanalysis.org/blog/index.php/unified-debugging-patterns/
106 http://www.dumpanalysis.org/blog/index.php/software-tracing-best-practices
107 http://support.citrix.com/article/CTX111901
108 http://support.citrix.com/article/CTX118614
109 http://www.dumpanalysis.org/blog/index.php/2006/10/30/applying-api-wrapper-pattern/
110 http://www.dumpanalysis.org/blog/index.php/2006/11/22/windowhistory-mobile-new-release/
111 http://www.dumpanalysis.org/Forthcoming+Windows+Debugging%3A+Practical+Foundations
113 http://www.dumpanalysis.com/files/Pattern-Driven-Software-Problem-Solving-Introduction.pdf
114 http://www.dumpanalysis.com/files/FundamentalsCompleteDumpAnalysis.pdf
115 http://www.dumpanalysis.org/blog/index.php/2011/03/16/virtual-presentation-pattern-driven-memory-dump-analysis/
116 http://www.dumpanalysis.com/CMDA-materials
117 http://stackoverflow.com/questions/5727692/azure-crash-dumps-not-appearing