Sometimes we have memory leaks related to the growing number of page tables. One reason for that could be the growing number of zombie processes (Volume 2, page 196) noticeable with tens of thousands of them.
1: kd> !process 0 0 [...] PROCESS fffffa80266bd6f0 SessionId: 0 Cid: 0a6c Peb: 7fffffdc000 ParentCid: 03ac DirBase: 9d35a000 ObjectTable: fffff8a00170ac80 HandleCount: 152. Image: svchost.exe [...] PROCESS fffffa8027de9b30 SessionId: 0 Cid: 21d0 Peb: 7fffffdf000 ParentCid: 02e0 DirBase: 37881000 ObjectTable: 00000000 HandleCount: 0. Image: conhost.exe [...] PROCESS fffffa8028eb0600 SessionId: 0 Cid: ab88 Peb: 7fffffdf000 ParentCid: 02e0 DirBase: 27a2f000 ObjectTable: 00000000 HandleCount: 0. Image: conhost.exe [...]
Even zombies have at least one remaining page (page directory) from the former page tables of their virtual to physical memory mapping (!dd is the same as dd WinDbg command but for physical memory):
1: kd> !dd 9d35a000
#9d35a000 9dd62867 03c00000 00000000 00000000
#9d35a010 00000000 00000000 00000000 00000000
#9d35a020 00000000 00000000 00000000 00000000
#9d35a030 00000000 00000000 00000000 00000000
#9d35a040 00000000 00000000 00000000 00000000
#9d35a050 00000000 00000000 00000000 00000000
#9d35a060 00000000 00000000 00000000 00000000
#9d35a070 00000000 00000000 9d45e867 49500000
1: kd> !dd 37881000
#37881000 00000000 00000000 00000000 00000000
#37881010 00000000 00000000 00000000 00000000
#37881020 00000000 00000000 00000000 00000000
#37881030 00000000 00000000 00000000 00000000
#37881040 00000000 00000000 00000000 00000000
#37881050 00000000 00000000 00000000 00000000
#37881060 00000000 00000000 00000000 00000000
#37881070 00000000 00000000 00000000 00000000
1: kd> !dd 27a2f000
#27a2f000 00000000 00000000 00000000 00000000
#27a2f010 00000000 00000000 00000000 00000000
#27a2f020 00000000 00000000 00000000 00000000
#27a2f030 00000000 00000000 00000000 00000000
#27a2f040 00000000 00000000 00000000 00000000
#27a2f050 00000000 00000000 00000000 00000000
#27a2f060 00000000 00000000 00000000 00000000
#27a2f070 00000000 00000000 00000000 00000000
We also see that 2 conhost.exe processes have identical physical to virtual mapping because their user space mappings are no longer valid (zeroed) and svchost.exe process has user space mapping:
1: kd> !ptov 27a2f000 Amd64PtoV: pagedir 27a2f000 27a2f000 fffff6fb`7dbed000 71530000 fffff6fb`7dbee000 19d000 fffff6fb`7dbef000 199000 fffff6fb`7dbf0000 b6a04000 fffff6fb`7dbf1000 b1f57000 fffff6fb`7dbf2000 29c4000 fffff6fb`7dbf3000 1c53000 fffff6fb`7dbf5000 [...] 2e4d8000 fffffa80`28f2d000 2c3d7000 fffffa80`28f2e000 30ed6000 fffffa80`28f2f000 2efd5000 fffffa80`28f30000 2ded4000 fffffa80`28f31000 2a5d3000 fffffa80`28f32000 bb400000 fffffa80`29600000 (large page) bb200000 fffffa80`29800000 (large page) 100000 ffffffff`ffd00000 105000 ffffffff`ffd01000 101000 ffffffff`ffd02000 102000 ffffffff`ffd03000 103000 ffffffff`ffd04000 104000 ffffffff`ffd05000 fec00000 ffffffff`ffd06000 1000 ffffffff`ffd07000 106000 ffffffff`ffd08000 123000 ffffffff`ffd09000 0 ffffffff`ffd0a000 124000 ffffffff`ffd0b000 2000 ffffffff`ffd0c000 e00c7000 ffffffff`ffd0d000 e0080000 ffffffff`ffd0e000 107000 ffffffff`ffd25000 108000 ffffffff`ffd26000 109000 ffffffff`ffd27000 10a000 ffffffff`ffd28000 10b000 ffffffff`ffd29000 10c000 ffffffff`ffd2a000 10d000 ffffffff`ffd2b000 10e000 ffffffff`ffd2c000 10f000 ffffffff`ffd2d000 110000 ffffffff`ffd2e000 111000 ffffffff`ffd2f000 112000 ffffffff`ffd30000 113000 ffffffff`ffd31000 114000 ffffffff`ffd32000 115000 ffffffff`ffd33000 116000 ffffffff`ffd34000 117000 ffffffff`ffd35000 118000 ffffffff`ffd36000 119000 ffffffff`ffd37000 11a000 ffffffff`ffd38000 11b000 ffffffff`ffd39000 11c000 ffffffff`ffd3a000 11d000 ffffffff`ffd3b000 11e000 ffffffff`ffd3c000 11f000 ffffffff`ffd3d000 120000 ffffffff`ffd3e000 121000 ffffffff`ffd3f000 122000 ffffffff`ffd40000 fee00000 ffffffff`fffe0000
1: kd> !ptov 37881000 Amd64PtoV: pagedir 37881000 37881000 fffff6fb`7dbed000 8d482000 fffff6fb`7dbee000 19d000 fffff6fb`7dbef000 199000 fffff6fb`7dbf0000 b6a04000 fffff6fb`7dbf1000 b1f57000 fffff6fb`7dbf2000 29c4000 fffff6fb`7dbf3000 1c53000 fffff6fb`7dbf5000 [...] 2e4d8000 fffffa80`28f2d000 2c3d7000 fffffa80`28f2e000 30ed6000 fffffa80`28f2f000 2efd5000 fffffa80`28f30000 2ded4000 fffffa80`28f31000 2a5d3000 fffffa80`28f32000 bb400000 fffffa80`29600000 (large page) bb200000 fffffa80`29800000 (large page) 100000 ffffffff`ffd00000 105000 ffffffff`ffd01000 101000 ffffffff`ffd02000 102000 ffffffff`ffd03000 103000 ffffffff`ffd04000 104000 ffffffff`ffd05000 fec00000 ffffffff`ffd06000 1000 ffffffff`ffd07000 106000 ffffffff`ffd08000 123000 ffffffff`ffd09000 0 ffffffff`ffd0a000 124000 ffffffff`ffd0b000 2000 ffffffff`ffd0c000 e00c7000 ffffffff`ffd0d000 e0080000 ffffffff`ffd0e000 107000 ffffffff`ffd25000 108000 ffffffff`ffd26000 109000 ffffffff`ffd27000 10a000 ffffffff`ffd28000 10b000 ffffffff`ffd29000 10c000 ffffffff`ffd2a000 10d000 ffffffff`ffd2b000 10e000 ffffffff`ffd2c000 10f000 ffffffff`ffd2d000 110000 ffffffff`ffd2e000 111000 ffffffff`ffd2f000 112000 ffffffff`ffd30000 113000 ffffffff`ffd31000 114000 ffffffff`ffd32000 115000 ffffffff`ffd33000 116000 ffffffff`ffd34000 117000 ffffffff`ffd35000 118000 ffffffff`ffd36000 119000 ffffffff`ffd37000 11a000 ffffffff`ffd38000 11b000 ffffffff`ffd39000 11c000 ffffffff`ffd3a000 11d000 ffffffff`ffd3b000 11e000 ffffffff`ffd3c000 11f000 ffffffff`ffd3d000 120000 ffffffff`ffd3e000 121000 ffffffff`ffd3f000 122000 ffffffff`ffd40000 fee00000 ffffffff`fffe0000
1: kd> !ptov 9d35a000 Amd64PtoV: pagedir 9d35a000 9E587000 10000 6871E000 20000 AF5AA000 30000 AF5AB000 31000 AFAAC000 32000 AFBAD000 33000 AF2F5000 40000 9D66B000 50000 22199000 60000 9D962000 E5000 9D261000 E6000 9DC60000 E7000 9D256000 EA000 9D84F000 EB000 9E4EC000 EC000 9E081000 ED000 9D876000 EE000 9E271000 EF000 B8BFD000 F0000 B8EFE000 F1000 B86FF000 F2000 B5302000 F3000 B5202000 F4000 B5502000 F5000 B7F03000 F6000 B8404000 F7000 B8415000 100000 B8B16000 101000 B1B17000 102000 [...] 2CD4000 77512000 5D7000 77515000 5D8000 77516000 4D9000 77517000 B358F000 77590000 AEF04000 77591000 68624000 77592000 64B26000 77593000 AF4C6000 77595000 B2042000 7EFE0000 B2143000 7EFE1000 B1A56000 7EFE2000 B1A57000 7EFE3000 B1B58000 7EFE4000 1BA000 7FFE0000 9DA69000 BFEB0000 AEEAE000 FFEA0000 AF191000 FFEA1000 9D76A000 FFEA2000 AE793000 FFEA3000 9DC8E000 FFEA5000 B7EB7000 FFEA6000 9DFFC000 FFEA7000 [...] 2e4d8000 fffffa80`28f2d000 2c3d7000 fffffa80`28f2e000 30ed6000 fffffa80`28f2f000 2efd5000 fffffa80`28f30000 2ded4000 fffffa80`28f31000 2a5d3000 fffffa80`28f32000 bb400000 fffffa80`29600000 (large page) bb200000 fffffa80`29800000 (large page) 100000 ffffffff`ffd00000 105000 ffffffff`ffd01000 101000 ffffffff`ffd02000 102000 ffffffff`ffd03000 103000 ffffffff`ffd04000 104000 ffffffff`ffd05000 fec00000 ffffffff`ffd06000 1000 ffffffff`ffd07000 106000 ffffffff`ffd08000 123000 ffffffff`ffd09000 0 ffffffff`ffd0a000 124000 ffffffff`ffd0b000 2000 ffffffff`ffd0c000 e00c7000 ffffffff`ffd0d000 e0080000 ffffffff`ffd0e000 107000 ffffffff`ffd25000 108000 ffffffff`ffd26000 109000 ffffffff`ffd27000 10a000 ffffffff`ffd28000 10b000 ffffffff`ffd29000 10c000 ffffffff`ffd2a000 10d000 ffffffff`ffd2b000 10e000 ffffffff`ffd2c000 10f000 ffffffff`ffd2d000 110000 ffffffff`ffd2e000 111000 ffffffff`ffd2f000 112000 ffffffff`ffd30000 113000 ffffffff`ffd31000 114000 ffffffff`ffd32000 115000 ffffffff`ffd33000 116000 ffffffff`ffd34000 117000 ffffffff`ffd35000 118000 ffffffff`ffd36000 119000 ffffffff`ffd37000 11a000 ffffffff`ffd38000 11b000 ffffffff`ffd39000 11c000 ffffffff`ffd3a000 11d000 ffffffff`ffd3b000 11e000 ffffffff`ffd3c000 11f000 ffffffff`ffd3d000 120000 ffffffff`ffd3e000 121000 ffffffff`ffd3f000 122000 ffffffff`ffd40000 fee00000 ffffffff`fffe0000
In order to check user space virtual addresses we have to switch to the corresponding process context:
1: kd> !pte fffffa80`28f2d000 VA fffffa8028f2d000 PXE at FFFFF6FB7DBEDFA8 PPE at FFFFF6FB7DBF5000 PDE at FFFFF6FB7EA00A38 PTE at FFFFF6FD40147968 contains 0000000001C53863 contains 0000000001C54863 contains 0000000049320863 contains 000000002E4D8963 pfn 1c53 -DA- KWEV pfn 1c54 -DA- KWEV pfn 49320 -DA- KWEV pfn 2e4d8 -G-DA- KWEV
1: kd> .process /r /p fffffa80266bd6f0 Implicit process is now fffffa80`266bd6f0 Loading User Symbols 1: kd> !pte 10000 VA 0000000000010000 PXE at FFFFF6FB7DBED000 PPE at FFFFF6FB7DA00000 PDE at FFFFF6FB40000000 PTE at FFFFF68000000080 contains 03C000009DD62867 contains 031000009D865867 contains 7C2000009DD66867 contains 9CB000009E587867 pfn 9dd62 -DA- UWEV pfn 9d865 -DA- UWEV pfn 9dd66 -DA- UWEV pfn 9e587 -DA- UW- V
This pattern came to our attention after seeing memory dumps generated after the growing number of memory allocated for page tables exceeded a gigabyte.