Win32 calling conventions: __cdecl in assembler

Continuing on the series about Win32 calling conventions, the next topic of discussion is how the various calling conventions look from an assembler level.

This is useful to know for a variety of reasons; if you are reverse engineering (or debugging) something, one of the first steps is figuring out the calling convention for a function you are working with, so that you know how to find the arguments for it, how it deals with the stack, and soforth.

For this post, I’ll concentrate primarily on __cdecl.  Future posts will cover the other major calling conventions.

As I have previously described, __cdecl is an entirely stack based calling convention, in which arguments are cleaned off the stack by the caller.  Given this, you can expect to see all of the arguments for a function placed onto the stack before a function call is main.  If you are using CL, then this is almost always done by using the “push” instruction to place arguments on the stack.

Consider the following simple example function:

__declspec(noinline)
int __cdecl CdeclFunction1(int a, int b, int c)
{
 return (a + b) * c;
}

First, we’ll take a look at what calls to a __cdecl function look like. For example, if we look at a call to the function described above like so:

CdeclFunction1(1, 2, 3);

… we’ll see something like this:

; 119  : 	int v = CdeclFunction1(1, 2, 3);

  00000	6a 03		 push	 3
  00002	6a 02		 push	 2
  00004	6a 01		 push	 1
  00006	e8 00 00 00 00	 call	 CdeclFunction1
  0000b	83 c4 0c	 add	 esp, 12

There are basically three different things going on here.

  1. Setting up arguments for the target function. This is what the three different “push” instructions do. Note that the arguments are pushed in reverse order – you’ll always see them in reverse order if they are placed on the stack via push.
  2. Making the actual function call itself. After all the arguments are in place, the “call” instruction is used to transfer execution to the target. Remember that on x86, the call instruction implicitly pushes the return address on the stack. After the function call returns, the return value of the function is stored in the eax (or edx:eax) registers, typically.
  3. Cleaning arguments off the stack after the function returns. This is the purpose of the “add esp, 0xc” instruction following the “call” instruction. Since the target function does not adjust the stack to remove arguments after the call, this is up to the calller. Sometimes, you may see multiple __cdecl function calls be made in rapid succession, with the compiler only cleaning arguments from the stack after all of the function calls have been made (turning many different “add esp” instructions into just one “add esp” instruction).

It is also worth looking at the implementation of the function to see what it does with the arguments passed in and how it sets up a return value. The assembler for CdeclFunction1 is as so:

CdeclFunction1 proc near

a= dword ptr  4
b= dword ptr  8
c= dword ptr  0Ch

mov     eax, [esp+8]    ; eax = b
mov     ecx, [esp+4]    ; ecx = a
add     eax, ecx        ; eax = eax + ecx
imul    eax, [esp+0Ch]  ; eax = eax * c
retn                    ; (return value = eax)
CdeclFunction1 endp

This function is fairly straightforward. Since __cdecl is stack based for argument passing, all of the parameters are on the stack. Recall that the “call” instruction pushes the return address onto the stack, so the stack will begin with the return value at [esp+0] and have the first argument at [esp+4]. A graphical view of the stack layout (relative to “esp”) of this function is thus:

+00000000  r              db 4 dup(?)      ; (Return address)
+00000004 a               dd ?
+00000008 b               dd ?
+0000000C c               dd ?

In this case, there is no frame pointer in use, so the function accesses all of the arguments directly relative to “esp”. The steps taken are:

  1. The function fills eax with the value of the second argument (b), located at [esp+8] according to our stack layout.
  2. Next, the function loads ecx with the value of the first argument (a), which is located at [esp+4].
  3. Next, the function adds to eax the value of the first argument (a), now stored in the ecx register.
  4. Finally, the function multiplies eax by the value of the third argument (c), located at [esp+c].
  5. After finishing with all of the computations needed to implement the function, it simply returns with a “retn” instruction. Since the caller cleans the stack, the “retn” intruction (with a stack adjustment) is not used here; __cdecl functions never use “retn <displacement>“, only “retn”. Additionally, because the result of the “mul” instruction happened to be stored in the eax register here, no extra instructions are needed to set up the return value, as it is already stored in the return value register (eax) at the end of the function.

Most __cdecl function calls are very similar to the one discussed above, although there will typically be much more code to the actual function and the function call (if there are many arguments), and the compiler may play some optimization tricks (such as deferring cleaning the stack across several function calls). The basic things to look for with a __cdecl function are:

  • All arguments are on the stack.
  • The return instruction is “retn” and not “retn <displacement>“, even when there are a non-zero number of arguments.
  • Shortly after the function call returns, the caller cleans the stack of arguments pushed. This may be deferred later, depending on how the compiler assembled the caller.

Note that if you have been paying attention, given the above criteria, you’ve probably noticed that a __cdecl function with zero arguments will look identical to an __stdcall function with zero arguments. If you don’t have symbols or decorated function names, there is no way to tell the two apart when there are no arguments, as the semantics are the same in that special case.

That’s all for a basic overview of __cdecl from an assembler perspective. Next time: more on the other calling conventions at an assembly level.

5 Responses to “Win32 calling conventions: __cdecl in assembler”

  1. […] Like __cdecl, __stdcall is completely stack-based. The semantics of __stdcall are very similar to __cdecl, except that the arguments are cleaned off the stack by the callee instead of the caller. Because the number of arguments removed from the stack is burned into the target function at compile time, there is no support for variadic functions (functions that take a variable number of arguments, such as printf) that use the __stdcall calling convention. The rules for register usage and return values are otherwise identical to __cdecl. […]

  2. Andrew Rogers says:

    The assembly is correctly annotated:

    CdeclFunction1 proc near

    a= dword ptr 4
    b= dword ptr 8
    c= dword ptr 0Ch

    mov eax, [esp+8] ; eax = b
    mov ecx, [esp+4] ; ecx = a
    add eax, ecx ; eax = eax + ecx
    imul eax, [esp+0Ch] ; eax = eax * c
    retn ; (return value = eax)

    CdeclFunction1 endp

    However, the textual description should then read:

    1. The function fills eax with the value of the second argument (b), located at [esp+8] according to our stack layout.

    2. Next, the function loads the value of the first argument (a) into ecx, which is located at [esp+4], and adds it to eax.

    3. Finally, it multiplies eax by the value of the third argument (c), located at [esp+c].

    In the following text I think it’s better to use “ret n” (n possibly in italics) to indicate a return with stack offset. The mnemonic “retn” is a synonym for “ret”, denoting a ‘near’ (same-segment) return, as opposed to a “retf” denoting a ‘far’ (inter-segment) return, so this could cause confusion:

    c.f.:


    __cdecl functions never use “retn”, only “ret”.

    and (from above):

    imul eax, [esp+0Ch] ; eax = eax * c
    retn ; (return value = eax)

    CdeclFunction1 endp

  3. Andrew Rogers says:

    BTW: other than my nit-picking comments, this is a well-explained and highly informative introduction to calling convetions – thank you!

  4. Skywing says:

    Good catch with the description being wrong there, and the confusing bit about ret.

    As for ret vs retn, depends on which disassembler you use, I suppose, as to which display name you’ll see. In any case, you are bound to run into a bit of inconsistency as to how programs represent that instruction; for example, the Microsoft disassemblers (at least the WinDbg one) tend to like to just show “ret”, while HIEW and IDA prefer the retn.

    I’ll fix the things you caught – thanks for the corrections.

  5. Vedang says:

    Thanks,
    came across __cdecl for the first time while going through some source code today, and was looking for an explanation.
    great article!