If you’re a kernel developer, and you’ve upgraded to Vista, then one of the changes that you may have noticed is that you can’t perform local kernel debugging anymore.
This is true even if you elevate WinDbg. If you try, you’ll get an error message stating that the debugger failed to get KD version information (error 5), which corresponds to the Win32 ERROR_ACCESS_DENIED error code.
This is due to a change from Vista RC2 to Vista RTM, where the kernel function responsible for supporting much of the local KD functionality in WinDbg (KdSystemDebugControl) was altered to require the system to be booted with /DEBUG. This is apparent if we compare RC2 to RTM.
RC2 has the following check in KdSystemDebugControl (one comparison against KdpBootedNodebug):
nt!KdSystemDebugControl: push 0F4h push 81841938 call nt!_SEH_prolog4 xor ebx,ebx mov dword ptr [ebp-28h],ebx mov dword ptr [ebp-20h],ebx mov dword ptr [ebp-24h],ebx cmp byte ptr [nt!KdpBootedNodebug],bl je nt!KdSystemDebugControl+0x2c ; Success mov eax,0C0000022h ; STATUS_ACCESS_DENIED
On Vista RTM, two additional checks were added against nt!KdPitchDebugger and nt!KdDebuggerEnabled (disregard the fact that the RTM disassembly is from the x64 version; both the x86 and x64 Vista versions have the same checks):
nt!KdSystemDebugControl: mov qword ptr [rsp+8],rbx mov qword ptr [rsp+10h],rdi push r12 sub rsp,170h mov r10,rdx and dword ptr [rsp+44h],0 and qword ptr [rsp+48h],0 and qword ptr [rsp+50h],0 cmp byte ptr [nt!KdpBootedNodebug)],0 jne nt!KdSystemDebugControl+0x8b7 ; Fail cmp byte ptr [nt!KdPitchDebugger],0 jne nt!KdSystemDebugControl+0x8b7 ; Fail cmp byte ptr [nt!KdDebuggerEnabled],0 je nt!KdSystemDebugControl+0x8b7 ; Fail
The essence of these checks is that you need to be booted with /DEBUG enabled in order for local kernel debugging to work.
There is a simple way to accomplish this, however, without the usual painful aspects of having a kernel debugger attached (e.g. breaking on user mode exceptions or breakpoints).
All you have to do is enable kernel debugging, and then disable user mode exception handling. This requires the following options to be set via BCDEdit.exe, the Vista boot configuration database manager:
- bcdedit /debug on. This enables kernel debugging for the booted OS configuration.
- bcdedit /dbgsettings <type> /start disable /noumex (where type corresponds to a usable KD type on your computer, such as 1394). This disables user mode exception handling for the kernel debugger. You should still be able to boot the system without a kernel debugger attached.
After setting these options, reboot, and you should be set. You’ll now be able to use local KD (you must still remember to elevate the debugger, though), but you won’t have user mode programs try to break into the kernel debugger when they crash without a user mode debugger attached.
Note, however, that you’ll still be able to break in to the system with a kernel debugger after boot if you choose these options (and if the box crashes in kernel mode, it’ll freeze waiting for a debugger to attach). However, at least you will not have to contend with errant user mode programs causing the system to break into the kernel debugger.