Fast kernel debugging for VMware, part 1: Overview

Note: If you are just looking for the high speed VMware kernel debugging program, then you can find it here. This post series outlines the basic design principles behind VMKD.

One of the interesting talks at Blue Hat was one about virtualization and security. There’s a lot of good stuff that was touched on (such as the fact that VMware still implements a hub, meaning that VMs on the same VMnet can still sniff eachother’s traffic).

Anyways, watching the talk got me thinking again about how much kernel debugging VMs is a slow and painful experience today, especially if you’ve used 1394 debugging frequently.

While kd-over-1394 is quite fast (as is local kd), you can’t do that in any virtualization software that I’m aware of today (none of them virtualize 1394, and furthermore, as far as I know none of them even support USB2 debugging either, even VMware Workstation 6).

This means that if you’re kernel debugging a VM in today’s world, you’re pretty much left high and dry and have to use the dreaded virtual serial port approach. Because the virtual serial port has to act like a real serial port, it’s also slow, just like a real serial port (otherwise, timings get off and programs that talk to the serial port break all over the place). This means that although you might be debugging a completely local VM, you’re still throttled to serial port speeds (115200bps). Although it should certainly technically be possible to do better in a VM, none of the virtualization vendors support the virtual hardware required for the other, faster KD transports.

However, that got me thinking a bit. Windows doesn’t really need a virtual serial port or a virtual 1394 port to serve as a kernel debugging target because of any intrinsic, special property of serial or 1394 or USB2. Those interfaces are really just mechanisms to move bits from the target computer to the debugger computer and back again, while requiring minimal interaction with the rest of the system (it is important that the kernel debugger code in the target computer be as minimalistic and self-contained as possible or many situations where you just can’t debug code because it is used by the kernel debugger itself would start cropping up – this is why there isn’t a TCP transport for kernel debugging, among other things).

Now with a VM (as opposed to a physical computer), getting bits to and from an external kernel debugger and the kernel running in the VM is really quite easy. After all, the VM monitor can just directly read and write from the VM’s physical memory, just like that, without a need for indirecting through a real (or virtual) I/O interconnect interface.

So I got to be thinking that it should theoretically be possible to write a kernel debugger transport module that instead of talking to a serial, 1394, or USB2 port, talks to the local VMM and asks it to copy memory to and from the outside world (and thus the kernel debugger). After the data is safely out of the VM, it can be transported over to the kernel debugger (and back) with the mechanism of choice.

It turns out that Windows KD support is implemented in a way that is fairly conducive to this approach. The KD protocol is divided up into essentially two different parts. There’s the high level half, which is essentially a command set that allows the kernel debugger to request that the KD stub in the kernel perform an operation (like change the active register set, set a breakpoint, write memory, or soforth). The high level portion of the KD protocol sits on top of what I call the low level or (framing) portion of the KD protocol, which is a (potentially hardware dependant) transport interface that provides for reliable delivery of high level KD requests and responses between the KD program on a remote computer and the KD stub in the kernel of the target computer.

The low level KD protocol is abstracted out via a set of kernel debugger protocol modules (which are simple kernel mode DLLs) that are used by the kernel to talk to the various pieces of hardware that are supported for kernel debugging. For example, there is a module to talk to the serial port (kdcom.dll), and a module to talk to the 1394 controller (kd1394.dll).

These modules export a uniform API that essentially allows the kernel to request reliable (“mostly reliable”) transport of a high level KD request (say a notification that an exception has occured) from the kernel to the KD program, and back again.

This interface is fortunate from the perspective of someone who might want to, say, develop a high speed kernel debugger module for a VM running under a known VM monitor. Such a KD protocol module could take advantage of the fact that it knows that it’s running under a specific VM monitor, and use the VM monitor’s built-in VM exit / VM enter capabilities to quickly tell the VM monitor to copy data into and out of the VM. (Most VMs have some sort of “backdoor” interface for optimized drivers and enhanced guest capabilities, such as a way for the guest to tell the host when its mouse pointer has left the guest’s screen. For example, in the case of VMware, there is a “VMware Tools” program that you can install which provides this capability through a special “backdoor” interface to that allows the VM to request a “VM exit” for the purposes of having the VM monitor perform a specialized task.)

Next time: Examining the KD module interface, and more.

9 Responses to “Fast kernel debugging for VMware, part 1: Overview”

  1. Steve says:

    This is an extremly cool idea!!!

    And it works great too!!! I can’t beleive that this hasn’t been implemented by vmware themselves ;)

    You should try to contact microsoft to see if you can get it work this way with their vms as well!

    I haven’t been this happy with my debugger since 1394! Finally a responsive debugger!!!

    Thanks dude!!

  2. Skywing says:

    I certainly agree that it’d be great if the virtualization vendors did this natively. With source level integration, a lot of the annoying parts of VMKD relating to things like loading the hook DLL into the user mode VM process (and having to manually find the right PID associated with the VM you want), poking around in the VM process for undocumented functions, and soforth could easily be eliminated. (If anyone from VMware / Virtual Server happens to read this and is so inclined, feel free to drop me a mail if you’re interested in doing that, btw!)

    I’m also working on trying to get the DbgEng people to open up a nice pluggable interface to allow native VM debugging transports to more easily integrate with the debugger as well (as it is, VMKD internally has to basically reimplement the entire KDCOM framing protocol, for reasons that I’ll go into in other posts). For a couple of reasons, direct integration with DbgEng would also be much better than how VMKD currently operates (among other things, the speed at which dump files are written could be vastly improved even over the performance improvements that VMKD currently brings over serial port debugging).

    No response from the debugger folks yet, though, on that front.

  3. […] Nynaeve Adventures in Windows debugging and reverse engineering. « Fast kernel debugging for VMware, part 1: Overview […]

  4. I’ve always been really annoyed by the CPU spinning that’s going on in the VM when you’re broken into the debugger. It’s particularly annoying if you’re trying to compile something at the same time…

    Can’t wait to try this out.

  5. Steve says:

    I’ve been using this for a couple of days, and it works great in most cases. I have one case where I’m getting problems:

    If I reboot my vm, I cannot re-establish connection with it afterwards. I cannot start the kdvmware driver, I get an error: Element not found.

    I suspect I have to re-inject the dll in some way on the host side, but I guess I need to remove it first, and I havent’ gotten around that.

    In other words I’ve tried re-injecting the dll after the reboot, and restarting the kdvmware service without any success. Any pointers?

  6. Skywing says:

    If you just hit “reset” in the VMware console, the vmware-vmx process should be reused and the DLL won’t need to be reinjected. (Attempting to inject the DLL multiple times into the vmware-vmx process is harmless and will have no effect.)

    vmxpatch.dll is designed to be able to handle driver reconnects across its lifetime (e.g. across a reboot); this worked in my tests.

    These are most of the error statuses that can be returned from kdvmware’s DriverEntry along with the net start error message:

    STATUS_PORT_DISCONNECTED (“The handle is invalid.”) – couldn’t connect to vmxpatch.dll
    STATUS_NO_MEMORY (“Not enough storage space is available to process this command.”) – couldn’t allocate a block of contiguous physical memory
    STATUS_NOT_FOUND (“Element not found.”) – couldn’t find (either ntoskrnl.exe OR ntkrnlpa.exe) OR kdcom.dll in the loaded module list.
    STATUS_INSUFFICIENT_RESOURCES (“Insufficient system resources exist to complete the requested service.”) – memory allocation failure
    STATUS_ACCESS_VIOLATION (“Invalid access to memory location.”) – MmMapLockedPagesSpecifyCache failed for for creating a locked view with alternate protection for kdcom.dll patching
    STATUS_PROCEDURE_NOT_FOUND (“The specified procedure could not be found.”) – couldn’t locate a required kdcom.dll export

    If you’re getting “element not found”, then that is most likely a failure to find either kdcom.dll or ntoskrnl.exe in the loaded module list. Are you starting the system with /KERNEL= or something of that sort?

    There are also a number of debug prints in vmxpatch.dll which will be enabled if a debugger is attached to vmware-vmx.exe. However, if you’re getting STATUS_NOT_FOUND, I suspect that kdvmware.sys will have been aborting before it tries to talk to vmxpatch.dll.

  7. Steve says:

    Ahah! Yes, I’m using a partially checked build right now!

  8. Steve says:

    I have it setup with a checked version of windows 2003 (partially checked build), and I renamed the kernel components to halmacpi.chk, ntkrnlmp.chk and ntkrpamp.chk.

    Is there a way I could recompile the driver on the vm side to look for specific variables instead of ntoskrnl.exe?? Maybe make a version that looks for a registry entry?


  9. Skywing says:

    Yeah, that’ll do it. The code for finding the kernel base is kind of fragile and uses the module name for comparisons, which falls flat on its face if you boot with /KERNEL=.