Function Calling Conventions

Many of us have heard of the different types of function calling conventions. This article will not only explain what each one is but will even show you how each one works by disassembling a program. Before we get to disassembling here is what you would need to continue:

IDA Freeware or Win32Dasm
Any C++ Compiler (Borland C++ 5.5 Preferable)
Basics of C++ and 32-bit Assembly

I have used IDA for disassembling but I have shown the listing using Win32dasm as well since most people haven’t got IDA. I have Used Borland C++ 5.5 only. I have not tested it using VC++ and in any case the keyword “pascal” is not supported by VC++. WinAPI is similar to pascal and is defined in windows.h. However if you scroll through windef.h you will see the following lines:

#elif (_MSC_VER >= 800) || defined(_STDCALL_SUPPORTED)
#define CALLBACK    __stdcall
#define WINAPI      __stdcall
#define WINAPIV     __cdecl
#define APIENTRY    WINAPI
#define APIPRIVATE  __stdcall
#define PASCAL      __stdcall

Both Pascal and WINAPI are considered as _stdcall. So let’s just work with Borland as of now.

Now, what are Function Calling Conventions? Which are the types of calling conventions?

Different methods of calling Functions are called Calling Conventions. Here are the most popular ones.

_cdecl
pascal
_stdcall
_fastcall

Here is what each one does:

_cdecl

Arguments Passed from Right to Left
Calling Function Clears the Stack
‘this’ pointer is passed via stack last in case of Programs using OOP
Functions using _cdecl are preceded by an “_”

Pascal

Arguments Passed from Left to Right
Called Function Clears the Stack

_stdcall

Arguments Passed from Right to Left
Called Function Clears the Stack
‘this’ pointer passed via Stack Last
Functions Using _stdcall are preceded by a ‘_’ and end with ‘@’

_fastcall

Passes Arguments Via Registers. In case of unavailability of registers arguments are passed via the Stack.
Functions using Fastcall precede with a ‘@’.

Good, now that you have understood this much, here is the program that will further be disassembled by IDA and Win32DASM. Here it is:

#include 

void pascal Func1(int a,int b)
{
  printf("The Result is:0x%X\n",a+b);
}

void _stdcall Func2(int a,int b,int &c)
{
    c=a-b;
}

void _fastcall Func3(int a,int b)
{
  printf("0x%X - 0x%X = 0x%X\n",a,b,a-b);
}

void _cdecl Func4(int a,int b)
{
  printf("%X  %X\n",a,b);
}

int Func5(int a,int b)
{
  return a+=b;
}

void main()
{
  Func1(0x1,0xA);
  int a;
  Func2(0xA,0x4,a);
  printf("%X\n",a);
  Func3(0xB,0xA);
  Func4(100,200);
  printf("0x%X\n",Func5(0xA,0x3));
}

Compile this with Borland C++ 5.5
Even though this is not required, some might like the output as well:

The Result is:0xB
6
0xB - 0xA = 0x1
64  C8
0xD

First, I’d explain the code disassembled with Win32DASM and then with IDA. Here is the Shortened Disassembled listing from Win32DASM.

* Referenced by a CALL at Address:
|:0040117C   
|                          ;    Func1()
:00401108 55                      push ebp
:00401109 8BEC                    mov ebp, esp  ;Opens Stack Frame
:0040110B 8B450C                  mov eax, dword ptr [ebp+0C]  ;Stores 1st Argument in EAX
:0040110E 034508                  add eax, dword ptr [ebp+08]; EAX+=2nd Argument
:00401111 50                      push eax ;Pushes the Result on the Stack

* Possible StringData Ref from Data Obj ->"The Result is:0x%X"
                                  |
:00401112 68E8A04000              push 0040A0E8
:00401117 E814320000              call 00404330 ; call _printf()
:0040111C 83C408                  add esp, 00000008 ;Clears 8 bytes off the stack
:0040111F 5D                      pop ebp ;Close Stack Frame
:00401120 C20800                  ret 0008 ;Returns and clears 8 bytes off the stack

;						OBSERVATION
; Looking at the Source we can say that Arguments are pushed from left to right
;ret 0008 tells us that the called function clears the stack.

; HENCE THIS FUNCTION USES PASCAL CONVENTION

* Referenced by a CALL at Address:
|:00401189   
|						; Func2()
:00401123 55                      push ebp
:00401124 8BEC                    mov ebp, esp ;Open Stack Frame
:00401126 8B4508                  mov eax, dword ptr [ebp+08] ; EAX=1st Argument
:00401129 2B450C                  sub eax, dword ptr [ebp+0C] ;EAX-=2nd Argument
:0040112C 8B5510                  mov edx, dword ptr [ebp+10]  ;Argument which is passed by
; reference is stored in EDX
:0040112F 8902                    mov dword ptr [edx], eax    ;Store Result into the address
; pointed by EDX ie. the variable passed by reference
:00401131 5D                      pop ebp  ;Close Stack Frame
:00401132 C20C00                  ret 000C ; Return and Clears 12 bytes off the Stack


;						OBSERVATIONS
; Looking at the Source "Arguments are passed from Right to Left"
; Stack is cleared by Called Function
; HENCE THIS FUNCTION USES _stdcall Convention


* Referenced by a CALL at Address:
|:004011A8   
|					; Func3()
:00401135 55                      push ebp
:00401136 8BEC                    mov ebp, esp ; Open Stack Frame		
:00401138 8BC8                    mov ecx, eax ; ECX and EAX are being used without
; saving them on the stack. This means that Arguments are passed Via Registers.
:0040113A 2BCA                    sub ecx, edx ; Should be easy by now
:0040113C 51                      push ecx     ;Push Result on Stack
:0040113D 52                      push edx     ;Pushes Argument
:0040113E 50                      push eax     ;Pushes Argument

* Possible StringData Ref from Data Obj ->"0x%X - 0x%X = 0x%X"
                                  |
:0040113F 68FCA04000              push 0040A0FC
:00401144 E8E7310000              call 00404330 ; call _printf()
:00401149 83C410                  add esp, 00000010; Clears 10 bytes.
				; This is because 4 arguments are passed
				; to printf of 4 bytes each Hence 0x4*0x4=0x10
:0040114C 5D                      pop ebp
:0040114D C3                      ret	;Return without Clearing Stack

;					OBSERVATIONS

;Looking at the Source Arguments are passed via Registers.
;Since Arguments are passed via registers clearing of stack is not required.
; Hence This Function uses fastcall Convention.

* Referenced by a CALL at Address:
|:004011B4   
|					; Func4
:0040114E 55                      push ebp
:0040114F 8BEC                    mov ebp, esp
:00401151 FF750C                  push [ebp+0C];Pushes both Arguments
:00401154 FF7508                  push [ebp+08]

* Possible StringData Ref from Data Obj ->"%X  %X"
                                  |
:00401157 6810A14000              push 0040A110
:0040115C E8CF310000              call 00404330 ; call _printf()
:00401161 83C40C                  add esp, 0000000C; Clear 12 bytes
:00401164 5D                      pop ebp
:00401165 C3                      ret ; Return

;					OBSERVATIONS
; Arguments are passed from Right to Left.
; Calling Function Clears the Stack.(look at :004011B9)
; HENCE THIS FUNCTION USES _cdecl Convention


* Referenced by a CALL at Address:
|:004011C0   
|					; Func5
:00401166 55                      push ebp
:00401167 8BEC                    mov ebp, esp
:00401169 8B450C                  mov eax, dword ptr [ebp+0C]
:0040116C 014508                  add dword ptr [ebp+08], eax
:0040116F 8B4508                  mov eax, dword ptr [ebp+08] ; Return
; Value is stored in EAX
:00401172 5D                      pop ebp
:00401173 C3                      ret ; This Function should be clear

;					OBSERVATIONS
; Arguments Passed from Right to Left.
; Calling Function Clears the Stack.
; HENCE THIS FUNCTION USES _cdecl Convention
                         ;  MAIN()
:00401174 55                      push ebp
:00401175 8BEC                    mov ebp, esp ;Opens Stack Frame
:00401177 51                      push ecx; Allocates 4 bytes using 
;Optimization for a local variable.It's 
;Equivalent is SUB ESP,0004
:00401178 6A01                    push 00000001 ; 1st Argument
:0040117A 6A0A                    push 0000000A ; Second Argument
:0040117C E887FFFFFF              call 00401108 ; call Func1()
:00401181 8D45FC                  lea eax, dword ptr [ebp-04] ;Local 
;Variable
:00401184 50                      push eax ; is passed by reference
:00401185 6A04                    push 00000004 ; 2nd Argument	
:00401187 6A0A                    push 0000000A ; 1st Argument
:00401189 E895FFFFFF              call 00401123 ; call Func2()
:0040118E FF75FC                  push [ebp-04] ;Pushes the Variable 
;which was previously passed to 
;Func2 by reference

* Possible StringData Ref from Data Obj ->"%X"
                                  |
:00401191 6818A14000              push 0040A118
:00401196 E895310000              call 00404330 ; call _printf()
:0040119B 83C408                  add esp, 00000008 ; Clear 8 bytes off 
; the Stack
:0040119E BA0A000000              mov edx, 0000000A ; Argument Passed 
; Via Register
:004011A3 B80B000000              mov eax, 0000000B ; Argument Passed 
; Via Register
:004011A8 E888FFFFFF              call 00401135     ; call Func3
:004011AD 68C8000000              push 000000C8	    ; 2nd Argument
:004011B2 6A64                    push 00000064     ; 1st Argument
:004011B4 E895FFFFFF              call 0040114E     ; call Func4
:004011B9 83C408                  add esp, 00000008 ; Calling Function 
; Clears Stack
:004011BC 6A03                    push 00000003     ; 2nd Argument
:004011BE 6A0A                    push 0000000A     ; 1st Argument
:004011C0 E8A1FFFFFF              call 00401166     ;call Func5
:004011C5 83C408                  add esp, 00000008 ; Calling Function 
; Clears Stack
:004011C8 50                      push eax          ;Pushes Return 
; Value Returned by Func5

* Possible StringData Ref from Data Obj ->"0x%X"
                                  |
:004011C9 681CA14000              push 0040A11C
:004011CE E85D310000              call 00404330 ; call _printf()
:004011D3 83C408                  add esp, 00000008 ; Clear 8 bytes
:004011D6 59                      pop ecx ;Clear Space previously 
; allocated for local variable. It's equivalent is ADD ESP,0004
:004011D7 5D                      pop ebp
:004011D8 C3                      ret ; Return

Win32DASM’s Listing does not show local variables and number of arguments though we can find that out through analysis. On the other hand IDA has a very simple listing and is very easy to understand complex programs. If you don’t believe me here is the Shortened Listing Given By IDA.

Func1		proc near		; CODE XREF: main+8 p

arg_0		= dword	ptr  8
arg_4		= dword	ptr  0Ch

		push	ebp
		mov	ebp, esp	; Open Stack Frame
		mov	eax, [ebp+arg_4] ; Stores 1st Argument
					     ; in EAX
		add	eax, [ebp+arg_0] ; Adds	EAX with
					     ; the Second Argument
		push	eax		     ; Pushes the Result
		push	offset aTheResultIs0xX ; aTheResultIs0xX db
						;  'The Result is:0x%X',0Ah,0
		call	_printf
		add	esp, 8		; Clear	8 bytes	off the
							;	Stack
					; for the arguments passed
					; to the printf	Func
		pop	ebp		; Close	Stack Frame
					; ie. Restore EBP
		retn	8		; Called Function Clears the Stack
Func1		endp

; Attributes: bp-based frame

Func2		proc near		; CODE XREF: main+15 p

arg_0		= dword	ptr  8
arg_4		= dword	ptr  0Ch
arg_8		= dword	ptr  10h

		push	ebp
		mov	ebp, esp	; Open Stack Frame
		mov	eax, [ebp+arg_0] ; EAX=Argument1
		sub	eax, [ebp+arg_4] ; EAX-=Argument2
		mov	edx, [ebp+arg_8] ; EDX=Argument	passed
					; by Reference
		mov	[edx], eax	; Store	the Result ie.EAX
					; into the address of the
					; variable passed by
					; reference
		pop	ebp		; Close	Stack Frame
		retn	0Ch		; Called Function clears the Stack
Func2		endp

; Attributes: bp-based frame

Func3		proc near		; CODE XREF: main+34 p
		push	ebp
		mov	ebp, esp
		mov	ecx, eax	; EAX and ECX are not initialised
					; before using.	INDICATES THAT
					; Arguments are	passed via
					; Registers
		sub	ecx, edx
		push	ecx
		push	edx
		push	eax		; char
		push	offset a0xX0xX0xX ; a0xX0xX0xX	    db
					;  '0x%X - 0x%X = 0x%X',0Ah,0
		call	_printf
		add	esp, 10h	; Called Function Clears Stack
		pop	ebp
		retn
Func3		endp


; Attributes: bp-based frame

Func4		proc near		; CODE XREF: main+40 p

arg_0		= byte ptr  8
arg_4		= dword	ptr  0Ch

		push	ebp
		mov	ebp, esp
		push	[ebp+arg_4]
		push	dword ptr [ebp+arg_0] ;	char
		push	offset aXX	; aXX		  db '%X  %X',0Ah,0
		call	_printf
		add	esp, 0Ch
		pop	ebp
		retn
Func4		endp

; Attributes: bp-based frame

Func5		proc near		; CODE XREF: main+4C p

arg_0		= dword	ptr  8
arg_4		= dword	ptr  0Ch

		push	ebp
		mov	ebp, esp	; Working should be clear by now
		mov	eax, [ebp+arg_4]
		add	[ebp+arg_0], eax
		mov	eax, [ebp+arg_0] ; Result stored in EAX	which
					; is where return value	is stored
		pop	ebp
		retn
Func5		endp


; int __cdecl main(int argc,const char **argv,const char *envp)
; Attributes: bp-based frame

main		proc near		; DATA XREF: .data:0040A0B8 o

var_4		= byte ptr -4
argc		= dword	ptr  8
argv		= dword	ptr  0Ch
envp		= dword	ptr  10h

		push	ebp
		mov	ebp, esp	; Open Stack Frame
		push	ecx		; Huh??? Where is the
					; SUB ESP,4h ?
					; Borland C++ is witty...
					; push ECX will	use
					; less bytes but it also
					; allocates 4 bytes on
					; the stack. So	usually
					; if a 4-byte Local Variable
					; is used instead of the
					; SUB Instruction, Borland
					; uses push ECX
		push	1		; First	Argument
		push	0Ah		; Second Argument
		call	Func1
		lea	eax, [ebp+var_4] ; Local Variable is
		push	eax		; passed by reference
		push	4		; Second Argument
		push	0Ah		; First	Argument
		call	Func2
		push	dword ptr [ebp+var_4] ;	Pushes the Variable
					;  which
					; was previously passed	to
					; Func2	by Reference
		push	offset asc_40A118 ; asc_40A118    db '%X',0Ah,0
		call	_printf
		add	esp, 8
		mov	edx, 0Ah	; Arguments are	passed
		mov	eax, 0Bh	; by Registers
		call	Func3
		push	0C8h
		push	64h
		call	Func4
		add	esp, 8		; Calling Function Clears Stack
		push	3
		push	0Ah
		call	Func5
		add	esp, 8		; Calling Function Clears Stack
		push	eax		; Returned Value is
					; pushed onto the Stack
		push	offset a0xX	; a0xX		  db '0x%X',0Ah,0
		call	_printf
		add	esp, 8
		pop	ecx
		pop	ebp		; Close	Stack Frame
		retn
main		endp

Ok, IDA automatically detects every commonly used library function like printf and substitute’s printf wherever required. That’s why I put the Listing of IDA Last so it would be easy to shift from Win32DASM’s listing which is a bit complicated to understand.

So here you have actually seen the function conventions work live!! Hope you enjoyed reading this ( and disassembling it! ) as much as I did writing it.

Author Information:

Sanchit Karve

http://www.freewebs.com/born2c0de/

born2c0de@hotmail.com