In Chapter 4, we discussed how the stack and the
call
instruction are used for function calls. Function calls can
appear differently in assembly code, and calling conventions govern the way the function call
occurs. These conventions include the order in which parameters are placed on the stack or in
registers, and whether the caller or the function called (the callee) is
responsible for cleaning up the stack when the function is complete.
The calling convention used depends on the compiler, among other factors. There are often subtle differences in how compilers implement these conventions, so it can be difficult to interface code that is compiled by different compilers. However, you need to follow certain conventions when using the Windows API, and these are uniformly implemented for compatibility (as discussed in Chapter 7).
We will use the pseudocode in Example 6-16 to describe each of the calling conventions.
Example 6-16. Pseudocode for a function call
int test(int x, int y, int z); int a, b, c, ret; ret = test(a, b, c);
The three most common calling conventions you will encounter are cdecl
, stdcall
, and fastcall
. We discuss the key differences between them in the following sections.
Although the same conventions can be implemented differently between compilers, we’ll focus on the most common ways they are used.
cdecl
is one of the most popular conventions and was
described in Chapter 4 when we introduced the stack and
function calls. In cdecl
, parameters are pushed onto the stack
from right to left, the caller cleans up the stack when the function is complete, and the return
value is stored in EAX. Example 6-17 shows an example of what the
disassembly would look like if the code in Example 6-16 were
compiled to use cdecl
.
Notice in the highlighted portion that the stack is cleaned up by the caller. In this
example, the parameters are pushed onto the stack from right to left, beginning with c
.
The popular stdcall
convention is similar to cdecl
, except stdcall
requires the
callee to clean up the stack when the function is complete. Therefore, the add
instruction highlighted in Example 6-17 would not be needed
if the stdcall
convention were used, since the function called
would be responsible for cleaning up the stack.
The test
function in Example 6-16 would be compiled differently under stdcall
, because it must be concerned with cleaning up the stack. Its
epilogue would need to take care of the cleanup.
stdcall
is the standard calling convention for the Windows
API. Any code calling these API functions will not need to clean up the stack, since that’s
the responsibility of the DLLs that implement the code for the API function.
The fastcall
calling convention varies the most across
compilers, but it generally works similarly in all cases. In fastcall
, the first few arguments (typically two) are passed in registers, with the most
commonly used registers being EDX and ECX (the Microsoft fastcall
convention). Additional arguments are loaded from right to left, and the calling function is usually
responsible for cleaning up the stack, if necessary. It is often more efficient to use fastcall
than other conventions, because the code doesn’t need to
involve the stack as much.
In addition to using the different calling conventions described so far, compilers may also
choose to use different instructions to perform the same operation, usually when the compiler
decides to move rather than push things onto the stack. Example 6-18
shows a C code example of a function call. The function adder
adds two arguments and returns the result. The main
function
calls adder
and prints the result using printf
.
Example 6-18. C code for a function call
int adder(int a, int b) { return a+b; } void main() { int x = 1; int y = 2; printf("the function returned the number %d ", adder(x,y)); }
The assembly code for the adder
function is
consistent across compilers and is displayed in Example 6-19. As you can see, this code adds arg_0
to arg_4
and stores the result in
EAX. (As discussed in Chapter 4, EAX stores the return
value.)
Example 6-19. Assembly code for the adder
function in Example 6-18
00401730 push ebp 00401731 mov ebp, esp 00401733 mov eax, [ebp+arg_0] 00401736 add eax, [ebp+arg_4] 00401739 pop ebp 0040173A retn
Table 6-1 displays different calling
conventions used by two different compilers: Microsoft Visual Studio and GNU Compiler Collection
(GCC). On the left, the parameters for adder
and printf
are pushed onto the stack before the call. On the right, the
parameters are moved onto the stack before the call. You should be prepared for both types of
calling conventions, because as an analyst, you won’t have control over the compiler. For
example, one instruction on the left does not correspond to any instruction on the right. This
instruction restores the stack pointer, which is not necessary on the right because the stack
pointer is never altered.
Remember that even when the same compiler is used, there can be differences in calling conventions depending on the various settings and options.
Table 6-1. Assembly Code for a Function Call with Two Different Calling Conventions