Comments on: Frame pointer omission (FPO) optimization and consequences when debugging, part 1 http://www.nynaeve.net/?p=91 Adventures in Windows debugging and reverse engineering. Tue, 24 Aug 2010 12:05:39 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: Ksplice — safe enough? « JP’s Blog http://www.nynaeve.net/?p=91&cpage=1#comment-25982 Ksplice — safe enough? « JP’s Blog Thu, 01 May 2008 08:47:18 +0000 http://www.nynaeve.net/?p=91#comment-25982 [...] on Windows, it is still a problem that cannot be easily dismissed. Finally, optimizations such as Frame Pointer Omission can thwart attempts to perform a stack walk by following the [...] [...] on Windows, it is still a problem that cannot be easily dismissed. Finally, optimizations such as Frame Pointer Omission can thwart attempts to perform a stack walk by following the [...]

]]> By: Skywing http://www.nynaeve.net/?p=91&cpage=1#comment-656 Skywing Thu, 07 Dec 2006 02:47:39 +0000 http://www.nynaeve.net/?p=91#comment-656 Vladimir: Try using "/Oy-". Also, I've noticed that there are a lot of places where recent compilers (e.g. VS2005) will absolutely insist on using EBP when they don't really *need* to per-se, such as functions with array local variables. So, you might have a bit of trouble in getting CL 14 / VS2005 to omit code to use EBP as a frame in some cases (i.e. forcing CL to use direct ESP accesses). Vladimir: Try using “/Oy-”. Also, I’ve noticed that there are a lot of places where recent compilers (e.g. VS2005) will absolutely insist on using EBP when they don’t really *need* to per-se, such as functions with array local variables. So, you might have a bit of trouble in getting CL 14 / VS2005 to omit code to use EBP as a frame in some cases (i.e. forcing CL to use direct ESP accesses).

]]> By: Nynaeve » Blog Archive » Frame pointer omission (FPO) optimization and consequences when debugging, part 2 http://www.nynaeve.net/?p=91&cpage=1#comment-653 Nynaeve » Blog Archive » Frame pointer omission (FPO) optimization and consequences when debugging, part 2 Wed, 06 Dec 2006 17:28:25 +0000 http://www.nynaeve.net/?p=91#comment-653 [...] Frame pointer omission (FPO) and consequences when debugging, part 1. [...] [...] Frame pointer omission (FPO) and consequences when debugging, part 1. [...]

]]> By: Vladimir Scherbina http://www.nynaeve.net/?p=91&cpage=1#comment-524 Vladimir Scherbina Thu, 23 Nov 2006 21:30:02 +0000 http://www.nynaeve.net/?p=91#comment-524 Skywing, Did you managed /Oy to work? I did some tests in past and they failed. Simple code that compares dword value with zero looks identical in both cases: when compiling with /Oy and w/o /Oy: (this is what I have in both cases) ; HRESULT DllCanUnloadNow(void) .text:10005650 public DllCanUnloadNow .text:10005650 DllCanUnloadNow proc near .text:10005650 push ebp .text:10005651 mov ebp, esp .text:10005653 xor eax, eax .text:10005655 cmp dword_10010078, 0 .text:1000565C setnz al .text:1000565F pop ebp .text:10005660 retn .text:10005660 DllCanUnloadNow endp Skywing,

Did you managed /Oy to work? I did some tests in past and they failed.

Simple code that compares dword value with zero looks identical in both cases: when compiling with /Oy and w/o /Oy: (this is what I have in both cases)

; HRESULT DllCanUnloadNow(void)
.text:10005650 public DllCanUnloadNow
.text:10005650 DllCanUnloadNow proc near
.text:10005650 push ebp
.text:10005651 mov ebp, esp
.text:10005653 xor eax, eax
.text:10005655 cmp dword_10010078, 0
.text:1000565C setnz al
.text:1000565F pop ebp
.text:10005660 retn
.text:10005660 DllCanUnloadNow endp

]]> By: Pavel Lebedinsky http://www.nynaeve.net/?p=91&cpage=1#comment-503 Pavel Lebedinsky Wed, 22 Nov 2006 08:41:12 +0000 http://www.nynaeve.net/?p=91#comment-503 As of VS 2005, the default is to not invoke local destructors for SEH exceptions: http://msdn2.microsoft.com/en-us/library/1deeycx5.aspx /EHa still works but generally is not recommended. As of VS 2005, the default is to not invoke local destructors for SEH exceptions:

http://msdn2.microsoft.com/en-us/library/1deeycx5.aspx

/EHa still works but generally is not recommended.

]]> By: Skywing http://www.nynaeve.net/?p=91&cpage=1#comment-500 Skywing Wed, 22 Nov 2006 05:19:44 +0000 http://www.nynaeve.net/?p=91#comment-500 No. I would assume this is primarily because of their use in inline memcmp/strcmp/memcpy. No. I would assume this is primarily because of their use in inline memcmp/strcmp/memcpy.

]]> By: dispensa http://www.nynaeve.net/?p=91&cpage=1#comment-499 dispensa Wed, 22 Nov 2006 05:13:16 +0000 http://www.nynaeve.net/?p=91#comment-499 > The convention is to use the ebp register to access locals and stack arguments. Ebp is typically setup such that the first stack argument can be found at [ebp+08], with local variables typically at a negative displacement from ebp. I think it's probably fair to say it's more than convention; ebp (== base pointer) is one of the few registers whose segment default is SS (the only other one being esp?). The rest reference DS or CS. I know it's not a problem on Windows, since the segments (other than FS/GS) all match, but in principle, these are implicit 48-bit addresses, and EDI, for example, can't be used to reference the stack without an override. Now, with that completely theoretical assertion out of the way, do you run across a lot of generated code using register-indirect addressing into the stack with ESI/EDI/etc. as a base? > The convention is to use the ebp register to access locals and stack arguments. Ebp is typically setup such that the first stack argument can be found at [ebp+08], with local variables typically at a negative displacement from ebp.

I think it’s probably fair to say it’s more than convention; ebp (== base pointer) is one of the few registers whose segment default is SS (the only other one being esp?). The rest reference DS or CS.

I know it’s not a problem on Windows, since the segments (other than FS/GS) all match, but in principle, these are implicit 48-bit addresses, and EDI, for example, can’t be used to reference the stack without an override.

Now, with that completely theoretical assertion out of the way, do you run across a lot of generated code using register-indirect addressing into the stack with ESI/EDI/etc. as a base?

]]>