Beware of custom unhandled exception filters in DLLs

Previously, I had discussed some techniques for debugging unhandled exception filters.  There are some more gotchas relating to unhandled exception filters than just debugging them, though.

The problem with unhandled exception filters is that they are broken by design.  The API (SetUnhandledExceptionFilter) used to install them allows you to build a chain of unhandled exception filters as multiple images within a process install their own filter.  While this may seem fine in practice, it actually turns out to be a serious flaw.  The problem is that there is no support for removing these unhandled exception filters out of order.  If you do so, you often end up with a previous unhandled exception filter pointer used by some DLL that points to a now-unloaded DLL, because some DLL with an unhandled exception filter was unloaded, but the unhandled exception filter registered after it still has a pointer to the previous filter in the now unloaded DLL.

This turns out to (at best) cause your custom crash handling logic to appear to randomly fail to operate, and at worst, introduce serious security holes in your program.  You can read more about the security hole this introduces in the paper on, but the basic idea is that if unhandled exception filters are unregistered out of order, you have a “dangling” function pointer that points to no-mans-land.  If an attacker can fill your process address space with shell code and then cause an exception (perhaps an otherwise “harmless” null pointer dereference that would cause your program to crash), he or she can take control of your process and run arbitrary code.

Unfortunately, there isn’t a good way to fix this from an application perspective.  I would recommend just not ever calling the previous unhandled exception filter, as there is no way to know whether it points to the real code that registered it or malicious code that someone allocated all over your process address space (called “heap spraying” in exploitation terminology).

You still have to deal with the fact that someone else might later install an unhandled exception filter ahead of yours, though, and then cause the unhandled exception filter chain to be broken upstream of you.  There is no real good solution for this; you might investigate patching SetUnhandledExceptionFilter or UnhandledExceptionFilter to always call you, but you can imagine what would happen if two functions try to do this at the same time.

So, the moral of the story is as follows:

  1. Don’t trust unhandled exception filters, as the model is brittle and easily breaks in processes that load and unload DLLs frequently.
  2. If you must register an unhandled exception filter, do it in a DLL that is never unloaded (or even the main .exe) to prevent the unhandled exception filter from being used as an attack vector.
  3. Don’t try to call the previous unhandled exception filter pointer that you get back from SetUnhandledExceptionFilter, as this introduces a security risk.
  4. Don’t install an unhandled exception filter from within a plugin type DLL that is loaded in a third party application, and especially don’t install an unhandled exception filter in a plugin type DLL that gets unloaded on the fly.

Unfortunately, it turns out to be even harder than this to not get burned by unhandled exception filter chaining.  More on that in a future posting.

12 Responses to “Beware of custom unhandled exception filters in DLLs”

  1. […] In a previous posting, I discussed some of the pitfalls of unhandled exception filters (and how they can become a security problem for your application). I mentioned some guidelines you can use to help work around these problems and minimize the risk, but, as I alluded to earlier, the problem is actually worse than it might appear on the surface. […]

  2. Yuhong Bao says:

    How about allocating some memory, writing to it a jump instruction to your handler, and then passing the address of this memory to SetUnhandledExceptionFilter? Then if you determine

  3. Yuhong Bao says:

    How about allocating some memory, writing to it a jump instruction to your handler, and then passing the address of this memory to SetUnhandledExceptionFilter? Then when you unregister, write to the memory a jump instruction to the previous filter. DON’T FREE IT! Because the point of doing this is having the addresses the other exception filters have saved to be still be valid.

  4. Yuhong Bao says:

    To be fair about the security holes however, someone can easily set a window hook on another thread, and that hook will be execute in the context (including privileges!) of that thread. That is easier than exploiting the vulnerabilities created by custom unhandled exception filters.

  5. Skywing says:

    The security vulnerability in UnhandledExceptionFilter is that it leaves a dangling pointer that you could place code under at a later point in time.

    If you can already run code to call SetWindowsHookEx, then you don’t need to exploit a dangling UEF pointer as you can already run arbitrary code.

  6. Yuhong Bao says:


  7. Yuhong Bao says:

    BTW, this kind of problem is not unique to unhandled exception filters. Window subclasses had the same problem.

  8. Skywing says:

    There are some special circumstances around window subclasses that prevent this from being such a big problem for them.

    Primarily that when you create a subclass, you are in general only supposed to remove it when processing WM_DESTROY, which is when everyone removes their subclasses. Furthermore, WM_DESTROY is processed in the correct order by each subclass such as to not result in any “dangling pointers” as the entire hook chain is cleanly torn down.

    UEFs are different. There is no signal that is sent to all UEFs in the proper order to say “hey, you should unhook yourself now”. UEFs are removed completely “out of band” with respect to the hook chain signalling mechanism, and they are not all removed at once (simultaneously). Instead, whenever a module unloads, it typically removes its UEF, without regard as to whether someone else had a hook above it or not.

    If people did the same things with subclasses, then yes, this would be a bigger problem for them. But that is generally not the case, so it is typically much rarer to be seeing dangling wndproc pointers than as you have with UEFs.

  9. Adrian Tulloch says:

    I feel bad coming to a thread so very late, but, even 16 months later this is still important (and good!) stuff.

    My question is: are these concerns still current, or has MS06-051 essentially closed this hole?

  10. Yuhong Bao says:

    DOS interrupt handlers, and so many other things like this, also have this problem. The way DOS TSRs deals with this is refusing to unload if their interrupt handlers have been hooked by another TSR. Obviously DLLs can’t do this, but still…