Debugging programs that block symbol server access

One of the rather frustrating things to debug in Windows is one of the programs or services that is in the path of symbol server requests. This is most often the case with a service running in a svchost group with a number of other services that are responsible for things like DNS or some internal support that WinInet relies on or soforth. This deadlock condition is also prone to happening if you are debugging something else that makes lots of calls out to WinInet, as WinInet has some cross-process state that allows the symbol server engine to get deadlocked waiting on the target to release a global mutex (or similar global synchronization / state).

If you naively try to simply attach a debugger and load symbols in such a situation, you’ll end up with a nasty surprise; the debugger will hang, and you’ll have to kill it (and whatever you were debugging) to recover. In the case of svchost groups, many of those services will not properly restart after just being abruptly killed, so you may even have to reboot.

Now, one obvious solution to this problem is to just turn off symbol server access at all and work without symbols. This is obviously a major pain, though – nobody wants to debug without symbols if you actually have access to them, right?

There are a couple of other things that you can do to debug things in this scenario, however, that are a bit less painful than forgoing symbols:

First, you can use the kernel debugger to debug user mode processes. I have often seen Microsoft employees recommend this on the newsgroups. While this works (the kernel debugger is not affected by the state of any services that you are poking on the target computer), it is most certainly a royal pain to do. Using the kernel debugger means that breakpoints will often affect all processes (unless used via ba), you’ll have to deal with parts of the program being paged out, and not to mention the fact that kernel debugger connections are typically much less responsive than a local user mode debugger.

Because of the inordinate amount of pain involved in using kd to debug a user mode process, I do not recommend ever going this route unless you absolutely positively have no other recourse for accurate debugging.

Therefore, I recommend a different procedure in this case:

  1. Disable symbol server entirely (remove all SRV* symbol server references from your symbol path) or make sure that you have loaded symbols for ntdll.dll.
  2. Attach to the process in question normally, but do not load symbols or issue any commands.
  3. Write a full user minidump out to disk somewhere. For example, .dump /ma c:\tmp.dmp.
  4. Detach from the process that blocks symbol server from working.
  5. Open the minidump you saved earlier, and issue a .reload /f command. This causes all symbols in the process to be downloaded from the symbol server if you do not already have them.
  6. Re-attach to the process that you wanted to debug, and set your symbol path to refer to the downstream store you used with symbol server, but without invoking symbol server. That is, if you had previously used SRV*C:\symbols*http://msdl.microsoft.com/download/symbols for your symbol path, set it to C:\symbols. This ensures that you will never try to hit the symbol server.
  7. Debug as normal.

After that, you’ll have everything in the process that could have symbols on the symbol server already downloaded into your downstream store. By turning off symbol server and just using the downstream store when you actually start your real debugging efforts, you’ll make sure that the debugger will never deadlock itself against the target. Best of all, you don’t need to download a very large symbol pack from Microsoft that might be missing files that have been hotfixed, since you are still (preloading) all of the symbols from the symbol server. This trick can, of course, also be used with your own internal symbol servers as well.

7 Responses to “Debugging programs that block symbol server access”

  1. dispensa says:

    Nice trick! Do all of the same locks happen if you do something with remote debugging (on the same box?) or cross-session debugging?

  2. Skywing says:

    Depends. For the svchost / DNS / Network case, yes. For the WinInet case, cross-session should work okay.

  3. Pavel Lebedinsky says:

    You can use .dump /m instead of /ma – this generates much smaller dump files that still have enough information to find correct symbols.

    If you’re just poking around and don’t need to set breakpoints etc. you can also attach without suspending the process (-pvr). This works kind of like local kd but for user mode.

  4. Ak says:

    Can you give an example for a WinInet intenral support?

    Because the DNS problem could be solved very easily using the hosts file, no?

  5. Skywing says:

    Ak: If you debug Internet Explorer and break in while something is making a WinInet call, you’ll tend to deadlock the debugger.

  6. jlanawalt says:

    Thank you for sharing this. I was pointed over to this tip from the my question about the same thing on MSDN forums at http://tinyurl.com/5gt2zn titled “Debugging and IE hang when loading symbols for netapi32.dll”

  7. Yuhong Bao says:

    There is a much simpler way of loading symbols from a symbol server than creating a minidump. You can use the /ip option of symchk to download symbols for all dlls loaded in a process.