I tend to prefer debugging with release builds instead of debug builds.

One of the things that I find myself espousing both at work and outside of work from time to time is the value of debugging using release builds of programs (for Windows applications, anyways). This may seem contradictory to some at first glance, as one would tend to believe that the debug build is in fact better for debugging (it is named the “debug build”, after all).

However, I tend to disagree with this sentiment, on several grounds:

  1. Debugging on debug builds only is an unrealistic situation. Most of the “interesting” problems that crop up in real life tend to be with release builds on customer sites or production environments. Many of the time, we do not have the luxury of being able to ship out a debug build to a customer or production environment.

    There is no doubt that debugging using the debug build can be easier, but I am of the opinion that it is disadvantageous to be unable to effectively debug release builds. Debugging with release builds all the time ensures that you can do this when you’ve really got no choice, or when it is not feasible to try and repro a problem using a debug build.

  2. Debug builds sometimes interfere with debugging. This is a highly counterintuitive concept initially, one that many people seem to be surprised at. To see what I mean, consider the scenario where one has a random memory corruption bug.

    This sort of problem is typically difficult and time consuming to track down, so one would want to use all available tools to help in this process. One most useful tool in the toolkit of any competent Windows debugger should be page heap, which is a special mode of the RTL heap (which implements the Win32 heap as exposed by APIs such as HeapAlloc).

    Page heap places a guard page at the end (or before, depending on its configuration) of every allocation. This guard page is marked inaccessible, such that any attempt to write to an allocation that exceeds the bounds of the allocated memory region will immediately fault with an access violation, instead of leaving the corruption to cause random failures at a later time. In effect, page heap allows one to catch the guility party “red handed” in many classes of heap corruption scenarios.

    Unfortunately, the debug build greatly diminishes the ability of page heap to operate. This is because when the debug version of the C runtime is used, any memory allocations that go through the CRT (such as new, malloc, and soforth) have special check and fill patterns placed before and after the allocation. These fill patterns are intended to be used to help detect memory corruption problems. When a memory block is returned using an API such as free, the CRT first checks the fill patterns to ensure that they are intact. If a discrepancy is found, the CRT will break into the debugger and notify the user that memory corruption has occured.

    If one has been following along thus far, it should not be too difficult to see how this conflicts with page heap. The problem lies in the fact that from the heap’s perspective, the debug CRT per-allocation metadata (including the check and fill patterns) are part of the user allocation, and so the special guard page is placed after (or before, if underrun protection is enabled) the fill patterns. This means that some classes of memory corruption bugs will overwrite the debug CRT metadata, but won’t trip page heap up, meaning that the only indication of memory corruption will be when the allocation is released, instead of when the corruption actually occured.

  3. Local variable and source line stepping are unreliable in release builds. Again, as with the first point, it is dangerous to get into a pattern of relying on these conveniences as they simply do not work correctly (or in the expected fashion) in release builds, after the optimizer has had its way with the program. If you get used to always relying on local variable and source line support, when used in conjunction with debug builds, then you’re going to be in for a rude awakening when you have to debug a release build. More than once at work I’ve been pulled in to help somebody out after they had gone down a wrong path when debugging something because the local variable display showed the wrong contents for a variable in a release build.

    The moral of the story here is to not rely on this information from the debugger, as it is only reliable for debug builds. Even then, local variable display will not work correctly unless you are stepping in source line mode, as within a source line (while stepping in assembly mode), local variables may not be initialized in the way that the debugger expects given the debug information.

Now, just to be clear, I’m not saying that anyone should abandon debug builds completely. There are a lot of valuable checks added by debug builds (assertions, the enhanced iterator validation in the VS2005 CRT, and stack variable corruption checks, just to name a few). However, it is important to be able to debug problems with release builds, and it seems to me that always relying on debug builds is detrimental to being able to do this. (Obviously, this can vary, but this is simply speaking on my personal experience.)

When I am debugging something, I typically only use assembly mode and line number information, if available (for manually matching up instructions with source code). Source code is still of course a useful time saver in many instances (if you have it), but I prefer not relying on the debugger to “get it right” with respect to such things, having been burned too many times in the past with incorrect results being returned in non-debug builds.

With a little bit of practice, you can get the same information that you would out of local variable display and the like with some basic reading of disassembly text and examination of the stack and register contents. As an added bonus, if you can do this in debug builds, you should by definition be able to do so in release builds as well, even when the debugger is unable to track locals correctly due to limitations in the debug information format.

Tags: ,

6 Responses to “I tend to prefer debugging with release builds instead of debug builds.”

  1. Steve says:

    It’s very true that assembly language is the only way to know what is really happening in the code. And I’ve made changes to code dynamically, even in drivers by adding some 0x90 in there too ;)

    I think everyone that debugs release code gets bitten by that bug eventually! However, I think that running the debug build is necessary during initial development. While I agree with you that the skills necessary to debug release builds need to be exerciced, I do think the debug version is more useful, at least in my driver world ;)

    By the way, I haven’t had a chance to test the new winDBg yet… Will soon though! Thanks!

    S

  2. I wholeheartedly agree with your comments and those made in http://blogs.msdn.com/jmstall/archive/2007/10/05/why-debugging-breaks-in-optimized-production-builds.aspx. Also, debugging in assembly mode can be a real eye-opener: for example, stepping through code that is built using class or templated libraries. It is too bad that working at this level can appear to be so terrifying, difficult, and last resort. A large part of the blame lies with the tools that are available; the tools need to work better with individual and to help rather than create more work.

  3. Jeff Curless says:

    I agree, lots of times I have helped co-workers solve problems by being able to understand the assembly. I always work with the debug build until I reach critical mass with what I am working on. Then I usually turn off debug, but keep optimizations off for awhile. Then finally I will turn on the optimizations and work with the full release version.

  4. Yuhong Bao says:

    In particular, you should be able to do dump debugging with release builds, for obvious reasons.

  5. […] usually do all of my testing with debug builds (contrary to Ken’s beliefs). Once I’m sure everything runs smoothly and I don’t get any assertions or erratic behavior, I […]

  6. […] usually do all of my testing with debug builds (contrary to Ken’s beliefs). Once I’m sure everything runs smoothly and I don’t get any assertions or erratic behavior, I […]

Leave a Reply