Stack caller & callee, x86/x64, Windows/Linux/OSX

Bare metal programming in PureBasic, for experienced users
User avatar
Keya
Addict
Addict
Posts: 1891
Joined: Thu Jun 04, 2015 7:10 am

Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by Keya »

hello i am trying to get my head around the stack in all its forms so i can feel comfortable with it. I dont want to feel intimidated by the stack! but i find that hard unless i can SEE whats going on!! lol :)

anyway i have put together a simple program to look at the conversation between caller and callee to see how different number of parameters affects the code on different systems, in particular what happens with stack registers

But first, the calling conventions!!!
Purebasic helpfile:
- On x86 processors, the available volatile registers are: eax, ecx And edx. All others must be always preserved.
- On x64 processors, the available volatile registers are: rax, rcx, rdx, r8, r9, xmm0, xmm1, xmm2 And xmm3. All others must be always preserved.
https://en.wikipedia.org/wiki/X86_calling_conventions
WINDOWS: RCX, RDX, R8, R9
The Microsoft x64 calling convention is followed on Windows and pre-boot UEFI (for long mode on x86-64). It uses registers RCX, RDX, R8, R9 for the first four integer or pointer arguments (in that order), and XMM0, XMM1, XMM2, XMM3 are used for floating point arguments. Additional arguments are pushed onto the stack (right to left). Integer return values (similar to x86) are returned in RAX if 64 bits or less. Floating point return values are returned in XMM0. Parameters less than 64 bits long are not zero extended; the high bits are not zeroed.

When compiling for the x64 architecture in a Windows context (whether using Microsoft or non-Microsoft tools), there is only one calling convention — the one described here, so that stdcall, thiscall, cdecl, fastcall, etc., are now all one and the same.

In the Microsoft x64 calling convention, it's the caller's responsibility to allocate 32 bytes of "shadow space" on the stack right before calling the function (regardless of the actual number of parameters used), and to pop the stack after the call. The shadow space is used to spill RCX, RDX, R8, and R9, but must be made available to all functions, even those with fewer than four parameters.

For example, a function taking 5 integer arguments will take the first to fourth in registers, and the fifth will be pushed on the top of the shadow space. So when the called function is entered, the stack will be composed of (in ascending order) the return address, followed by the shadow space (32 bytes) followed by the fifth parameter.
___

LINUX/OSX: RDI, RSI, RDX, RCX, R8, R9
The calling convention of the System V AMD64 ABI is followed on Solaris, Linux, FreeBSD, OS X, and other UNIX-like or POSIX-compliant operating systems. The first six integer or pointer arguments are passed in registers RDI, RSI, RDX, RCX, R8, and R9, while XMM0, XMM1, XMM2, XMM3, XMM4, XMM5, XMM6 and XMM7 are used for floating point arguments. For system calls, R10 is used instead of RCX. As in the Microsoft x64 calling convention, additional arguments are passed on the stack and the return value is stored in RAX.

Registers RBP, RBX, and R12–R15 are callee-save registers; all others must be saved by the caller if it wishes to preserve their values.

Unlike the Microsoft calling convention, a shadow space is not provided; on function entry, the return address is adjacent to the seventh integer argument on the stack.
So basically all the program is doing is looking at the code for a call to a simple function, and the function itself which simply makes reference to the parameters passed to it:

Code: Select all

Procedure.i Param2(p1.i, p2.i)   ;Callee
  mov rdx, p1 : mov rdx, p2
EndProcedure
 
Param2(1,2)    ;Caller
The program, which uses PB's native disassembler (Udis86 engine) which is really handy for this! :)

Code: Select all

DisableDebugger : EnableASM

CompilerIf #PB_Compiler_Processor = #PB_Processor_x86
  Macro rax : eax : EndMacro   
  Macro rdx : edx : EndMacro   
CompilerEndIf

Procedure.i Param0()
  !nop
EndProcedure

Procedure.i Param1(p1.i)
  mov rdx, p1    ;*ax is used for return, so we use *dx instead
EndProcedure

Procedure.i Param2(p1.i, p2.i)
  mov rdx, p1 : mov rdx, p2
EndProcedure

Procedure.i Param3(p1.i, p2.i, p3.i)
  mov rdx, p1 : mov rdx, p2 : mov rdx, p3
EndProcedure

Procedure.i Param4(p1.i, p2.i, p3.i, p4.i)
  mov rdx, p1 : mov rdx, p2 : mov rdx, p3 : mov rdx, p4
EndProcedure

Procedure.i Param5(p1.i, p2.i, p3.i, p4.i, p5.i)
  mov rdx, p1 : mov rdx, p2 : mov rdx, p3 : mov rdx, p4 : mov rdx, p5
EndProcedure

Procedure.i Param6(p1.i, p2.i, p3.i, p4.i, p5.i, p6.i)
  mov rdx, p1 : mov rdx, p2 : mov rdx, p3 : mov rdx, p4 : mov rdx, p5: mov rdx, p6
EndProcedure

Procedure.i Param7(p1.i, p2.i, p3.i, p4.i, p5.i, p6.i, p7.i)
  mov rdx, p1 : mov rdx, p2 : mov rdx, p3 : mov rdx, p4 : mov rdx, p5: mov rdx, p6: mov rdx, p7
EndProcedure

Goto ReadAllCodes   ;peace to all the anti-Goto'ers!

A0: 
Param0(): B0:

A1:
Param1(1): B1:

A2:
Param2(1,2): B2:

A3:
Param3(1,2,3): B3:

A4:
Param4(1,2,3,4): B4:

A5:
Param5(1,2,3,4,5): B5:

A6:
Param6(1,2,3,4,5,6): B6:

A7:
Param7(1,2,3,4,5,6,7)
!nop   ;required in x86 otherwise for some weird reason the disasm reports invalid for this one
B7:


Macro ReadCode(numparams, Code_Start, Code_End, CalleeProc)
  Text$ + "Caller:  (" + Str(numparams) + " parameters)" + #CRLF$
  If ExamineAssembly(?Code_Start, ?Code_End)
    While NextInstruction()
      sInstruct.s = Trim(InstructionString())
      If sInstruct = "nop"
      ElseIf Left(sInstruct,4) = "call"
        Text$ + "  call Callee()" + #CRLF$
      ElseIf (Left(sInstruct,4) = "mov " Or Left(sInstruct,4) = "push") And Left(Right(sInstruct,4),3) = " 0x"        
        Text$ + "  " + LSet(sInstruct,25," ") + ";#" + Right(sInstruct,1) + #CRLF$
      Else
        Text$ + "  " + sInstruct + #CRLF$
      EndIf
    Wend
  EndIf
  
  pcnt.i = 0
  Text$ + "Callee(): " + #CRLF$
  If ExamineAssembly(@CalleeProc(), @CalleeProc()+128)
    While NextInstruction()
      sInstruct.s = Trim(InstructionString())
      If Left(sInstruct,10) = "mov rax, [" Or Left(sInstruct,10) = "mov eax, [" Or Left(sInstruct,10) = "mov edx, [" Or Left(sInstruct,10) = "mov rdx, ["
        pcnt.i + 1
        Text$ + "  " + LSet(sInstruct,25," ") + ";#" + Str(pcnt) + #CRLF$
      Else
        Text$ + "  " + sInstruct + #CRLF$
      EndIf
      If Left(sInstruct,3) = "ret": Break: EndIf
    Wend
  EndIf
  Text$ + #CRLF$ 
EndMacro

ReadAllCodes:

CompilerIf #PB_Compiler_Processor = #PB_Processor_x64
  SysCPU.s="-x64"
CompilerElse
  SysCPU.s="-x86"
CompilerEndIf
CompilerIf #PB_Compiler_OS = #PB_OS_Linux
  SysOS.s="LNX"
CompilerElseIf #PB_Compiler_OS = #PB_OS_MacOS
  SysOS.s="MAC"
CompilerElse
  SysOS.s="WIN"
CompilerEndIf
Text$ = SysOS+SysCPU + ":   (PB v" + Str(#PB_Compiler_Version) + ")" + #CRLF$ + "========" + #CRLF$
ReadCode(0,A0,B0,Param0)
ReadCode(1,A1,B1,Param1)
ReadCode(2,A2,B2,Param2)
ReadCode(3,A3,B3,Param3)
ReadCode(4,A4,B4,Param4)
ReadCode(5,A5,B5,Param5)
ReadCode(6,A6,B6,Param6)
ReadCode(7,A7,B7,Param7)
SetClipboardText(Text$)
MessageRequester("Saved to clipboard", Text$)
;CreateFile(0, #PB_Compiler_FilePath+"~Stack_"+SysOS+SysCPU+".txt")
; WriteData(0, @Text$, Len(Text$))
;CloseFile(0)
Last edited by Keya on Sun Nov 22, 2015 6:08 pm, edited 5 times in total.
User avatar
Keya
Addict
Addict
Posts: 1891
Joined: Thu Jun 04, 2015 7:10 am

Re: Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by Keya »

Output from x86 & x64 Windows, Linux, OSX...

WIN-x86: (PB v540)

Code: Select all

Caller:  (0 parameters)
  call Callee()
Callee(): 
  nop
  xor eax, eax
  ret

Caller:  (1 parameters)
  push dword 0x1           ;#1
  call Callee()
Callee(): 
  mov edx, [esp+0x4]       ;#1
  xor eax, eax
  ret 0x4

Caller:  (2 parameters)
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
Callee(): 
  mov edx, [esp+0x4]       ;#1
  mov edx, [esp+0x8]       ;#2
  xor eax, eax
  ret 0x8

Caller:  (3 parameters)
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
Callee(): 
  mov edx, [esp+0x4]       ;#1
  mov edx, [esp+0x8]       ;#2
  mov edx, [esp+0xc]       ;#3
  xor eax, eax
  ret 0xc

Caller:  (4 parameters)
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
Callee(): 
  mov edx, [esp+0x4]       ;#1
  mov edx, [esp+0x8]       ;#2
  mov edx, [esp+0xc]       ;#3
  mov edx, [esp+0x10]      ;#4
  xor eax, eax
  ret 0x10

Caller:  (5 parameters)
  push dword 0x5           ;#5
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
Callee(): 
  mov edx, [esp+0x4]       ;#1
  mov edx, [esp+0x8]       ;#2
  mov edx, [esp+0xc]       ;#3
  mov edx, [esp+0x10]      ;#4
  mov edx, [esp+0x14]      ;#5
  xor eax, eax
  ret 0x14

Caller:  (6 parameters)
  push dword 0x6           ;#6
  push dword 0x5           ;#5
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
Callee(): 
  mov edx, [esp+0x4]       ;#1
  mov edx, [esp+0x8]       ;#2
  mov edx, [esp+0xc]       ;#3
  mov edx, [esp+0x10]      ;#4
  mov edx, [esp+0x14]      ;#5
  mov edx, [esp+0x18]      ;#6
  xor eax, eax
  ret 0x18

Caller:  (7 parameters)
  push dword 0x7           ;#7
  push dword 0x6           ;#6
  push dword 0x5           ;#5
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
Callee(): 
  mov edx, [esp+0x4]       ;#1
  mov edx, [esp+0x8]       ;#2
  mov edx, [esp+0xc]       ;#3
  mov edx, [esp+0x10]      ;#4
  mov edx, [esp+0x14]      ;#5
  mov edx, [esp+0x18]      ;#6
  mov edx, [esp+0x1c]      ;#7
  xor eax, eax
  ret 0x1c
LNX-x86: (PB v540)

Code: Select all

Caller:  (0 parameters)
  call Callee()
Callee(): 
  xor eax, eax
  push eax
  push eax
  push eax
  nop
  xor eax, eax
  add esp, 0xc
  ret

Caller:  (1 parameters)
  sub esp, 0xc
  push dword 0x1           ;#1
  call Callee()
  add esp, 0xc
Callee(): 
  xor eax, eax
  push eax
  push eax
  push eax
  mov edx, [esp+0x10]      ;#1
  xor eax, eax
  add esp, 0xc
  ret 0x4

Caller:  (2 parameters)
  sub esp, 0x8
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
  add esp, 0x8
Callee(): 
  xor eax, eax
  push eax
  push eax
  push eax
  mov edx, [esp+0x10]      ;#1
  mov edx, [esp+0x14]      ;#2
  xor eax, eax
  add esp, 0xc
  ret 0x8

Caller:  (3 parameters)
  sub esp, 0x4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
  add esp, 0x4
Callee(): 
  xor eax, eax
  push eax
  push eax
  push eax
  mov edx, [esp+0x10]      ;#1
  mov edx, [esp+0x14]      ;#2
  mov edx, [esp+0x18]      ;#3
  xor eax, eax
  add esp, 0xc
  ret 0xc

Caller:  (4 parameters)
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
Callee(): 
  xor eax, eax
  push eax
  push eax
  push eax
  mov edx, [esp+0x10]      ;#1
  mov edx, [esp+0x14]      ;#2
  mov edx, [esp+0x18]      ;#3
  mov edx, [esp+0x1c]      ;#4
  xor eax, eax
  add esp, 0xc
  ret 0x10

Caller:  (5 parameters)
  sub esp, 0xc
  push dword 0x5           ;#5
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
  add esp, 0xc
Callee(): 
  xor eax, eax
  push eax
  push eax
  push eax
  mov edx, [esp+0x10]      ;#1
  mov edx, [esp+0x14]      ;#2
  mov edx, [esp+0x18]      ;#3
  mov edx, [esp+0x1c]      ;#4
  mov edx, [esp+0x20]      ;#5
  xor eax, eax
  add esp, 0xc
  ret 0x14

Caller:  (6 parameters)
  sub esp, 0x8
  push dword 0x6           ;#6
  push dword 0x5           ;#5
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
  add esp, 0x8
Callee(): 
  xor eax, eax
  push eax
  push eax
  push eax
  mov edx, [esp+0x10]      ;#1
  mov edx, [esp+0x14]      ;#2
  mov edx, [esp+0x18]      ;#3
  mov edx, [esp+0x1c]      ;#4
  mov edx, [esp+0x20]      ;#5
  mov edx, [esp+0x24]      ;#6
  xor eax, eax
  add esp, 0xc
  ret 0x18

Caller:  (7 parameters)
  sub esp, 0x4
  push dword 0x7           ;#7
  push dword 0x6           ;#6
  push dword 0x5           ;#5
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  call Callee()
  add esp, 0x4
Callee(): 
  xor eax, eax
  push eax
  push eax
  push eax
  mov edx, [esp+0x10]      ;#1
  mov edx, [esp+0x14]      ;#2
  mov edx, [esp+0x18]      ;#3
  mov edx, [esp+0x1c]      ;#4
  mov edx, [esp+0x20]      ;#5
  mov edx, [esp+0x24]      ;#6
  mov edx, [esp+0x28]      ;#7
  xor eax, eax
  add esp, 0xc
  ret 0x1c
WIN-x64: (PB v540)

Code: Select all

Caller:  (0 parameters)
  call Callee()
Callee(): 
  sub rsp, 0x28
  nop
  xor rax, rax
  add rsp, 0x28
  ret

Caller:  (1 parameters)
  mov rcx, 0x1             ;#1
  call Callee()
Callee(): 
  mov [rsp+0x8], rcx
  sub rsp, 0x28
  mov rdx, [rsp+0x30]      ;#1
  xor rax, rax
  add rsp, 0x28
  ret

Caller:  (2 parameters)
  mov rdx, 0x2             ;#2
  mov rcx, 0x1             ;#1
  call Callee()
Callee(): 
  mov [rsp+0x8], rcx
  mov [rsp+0x10], rdx
  sub rsp, 0x28
  mov rdx, [rsp+0x30]      ;#1
  mov rdx, [rsp+0x38]      ;#2
  xor rax, rax
  add rsp, 0x28
  ret

Caller:  (3 parameters)
  mov r8, 0x3              ;#3
  mov rdx, 0x2             ;#2
  mov rcx, 0x1             ;#1
  call Callee()
Callee(): 
  mov [rsp+0x8], rcx
  mov [rsp+0x10], rdx
  mov [rsp+0x18], r8
  sub rsp, 0x28
  mov rdx, [rsp+0x30]      ;#1
  mov rdx, [rsp+0x38]      ;#2
  mov rdx, [rsp+0x40]      ;#3
  xor rax, rax
  add rsp, 0x28
  ret

Caller:  (4 parameters)
  mov r9, 0x4              ;#4
  mov r8, 0x3              ;#3
  mov rdx, 0x2             ;#2
  mov rcx, 0x1             ;#1
  call Callee()
Callee(): 
  mov [rsp+0x8], rcx
  mov [rsp+0x10], rdx
  mov [rsp+0x18], r8
  mov [rsp+0x20], r9
  sub rsp, 0x28
  mov rdx, [rsp+0x30]      ;#1
  mov rdx, [rsp+0x38]      ;#2
  mov rdx, [rsp+0x40]      ;#3
  mov rdx, [rsp+0x48]      ;#4
  xor rax, rax
  add rsp, 0x28
  ret

Caller:  (5 parameters)
  sub rsp, 0x8
  push dword 0x5           ;#5
  mov r9, 0x4              ;#4
  mov r8, 0x3              ;#3
  mov rdx, 0x2             ;#2
  mov rcx, 0x1             ;#1
  sub rsp, 0x20
  call Callee()
  add rsp, 0x30
Callee(): 
  mov [rsp+0x8], rcx
  mov [rsp+0x10], rdx
  mov [rsp+0x18], r8
  mov [rsp+0x20], r9
  sub rsp, 0x28
  mov rdx, [rsp+0x30]      ;#1
  mov rdx, [rsp+0x38]      ;#2
  mov rdx, [rsp+0x40]      ;#3
  mov rdx, [rsp+0x48]      ;#4
  mov rdx, [rsp+0x50]      ;#5
  xor rax, rax
  add rsp, 0x28
  ret

Caller:  (6 parameters)
  push dword 0x6           ;#6
  push dword 0x5           ;#5
  mov r9, 0x4              ;#4
  mov r8, 0x3              ;#3
  mov rdx, 0x2             ;#2
  mov rcx, 0x1             ;#1
  sub rsp, 0x20
  call Callee()
  add rsp, 0x30
Callee(): 
  mov [rsp+0x8], rcx
  mov [rsp+0x10], rdx
  mov [rsp+0x18], r8
  mov [rsp+0x20], r9
  sub rsp, 0x28
  mov rdx, [rsp+0x30]      ;#1
  mov rdx, [rsp+0x38]      ;#2
  mov rdx, [rsp+0x40]      ;#3
  mov rdx, [rsp+0x48]      ;#4
  mov rdx, [rsp+0x50]      ;#5
  mov rdx, [rsp+0x58]      ;#6
  xor rax, rax
  add rsp, 0x28
  ret

Caller:  (7 parameters)
  sub rsp, 0x8
  push dword 0x7           ;#7
  push dword 0x6           ;#6
  push dword 0x5           ;#5
  mov r9, 0x4              ;#4
  mov r8, 0x3              ;#3
  mov rdx, 0x2             ;#2
  mov rcx, 0x1             ;#1
  sub rsp, 0x20
  call Callee()
  add rsp, 0x40
Callee(): 
  mov [rsp+0x8], rcx
  mov [rsp+0x10], rdx
  mov [rsp+0x18], r8
  mov [rsp+0x20], r9
  sub rsp, 0x28
  mov rdx, [rsp+0x30]      ;#1
  mov rdx, [rsp+0x38]      ;#2
  mov rdx, [rsp+0x40]      ;#3
  mov rdx, [rsp+0x48]      ;#4
  mov rdx, [rsp+0x50]      ;#5
  mov rdx, [rsp+0x58]      ;#6
  mov rdx, [rsp+0x60]      ;#7
  xor rax, rax
  add rsp, 0x28
  ret
LNX-x64: (PB v540)

Code: Select all

Caller:  (0 parameters)
  call Callee()
Callee(): 
  sub rsp, 0x28
  nop
  xor rax, rax
  add rsp, 0x28
  ret

Caller:  (1 parameters)
  push dword 0x1           ;#1
  pop rdi
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rdx, [rsp+0x28]      ;#1
  xor rax, rax
  add rsp, 0x38
  ret

Caller:  (2 parameters)
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  pop rdi
  pop rsi
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  xor rax, rax
  add rsp, 0x38
  ret

Caller:  (3 parameters)
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  pop rdi
  pop rsi
  pop rdx
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rax, rdx
  mov [rsp+0x38], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  mov rdx, [rsp+0x38]      ;#3
  xor rax, rax
  add rsp, 0x48
  ret

Caller:  (4 parameters)
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  pop rdi
  pop rsi
  pop rdx
  pop rcx
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rax, rdx
  mov [rsp+0x38], rax
  mov rax, rcx
  mov [rsp+0x40], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  mov rdx, [rsp+0x38]      ;#3
  mov rdx, [rsp+0x40]      ;#4
  xor rax, rax
  add rsp, 0x48
  ret

Caller:  (5 parameters)
  push dword 0x5           ;#5
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  pop rdi
  pop rsi
  pop rdx
  pop rcx
  pop r8
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  push rax
  push rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rax, rdx
  mov [rsp+0x38], rax
  mov rax, rcx
  mov [rsp+0x40], rax
  mov rax, r8
  mov [rsp+0x48], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  mov rdx, [rsp+0x38]      ;#3
  mov rdx, [rsp+0x40]      ;#4
  mov rdx, [rsp+0x48]      ;#5
  xor rax, rax
  add rsp, 0x58
  ret

Caller:  (6 parameters)
  push dword 0x6           ;#6
  push dword 0x5           ;#5
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  pop rdi
  pop rsi
  pop rdx
  pop rcx
  pop r8
  pop r9
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  push rax
  push rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rax, rdx
  mov [rsp+0x38], rax
  mov rax, rcx
  mov [rsp+0x40], rax
  mov rax, r8
  mov [rsp+0x48], rax
  mov rax, r9
  mov [rsp+0x50], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  mov rdx, [rsp+0x38]      ;#3
  mov rdx, [rsp+0x40]      ;#4
  mov rdx, [rsp+0x48]      ;#5
  mov rdx, [rsp+0x50]      ;#6
  xor rax, rax
  add rsp, 0x58
  ret

Caller:  (7 parameters)
  sub rsp, 0x8
  push dword 0x7           ;#7
  push dword 0x6           ;#6
  push dword 0x5           ;#5
  push dword 0x4           ;#4
  push dword 0x3           ;#3
  push dword 0x2           ;#2
  push dword 0x1           ;#1
  pop rdi
  pop rsi
  pop rdx
  pop rcx
  pop r8
  pop r9
  call Callee()
  add rsp, 0x10
Callee(): 
  mov rax, 0x8
  sub rsp, 0x8
  mov qword [rsp], 0x0
  invalid  ;disasm engine bug
MAC-x64: (PB v540)

Code: Select all

Caller:  (0 parameters)
  call Callee()
Callee(): 
  sub rsp, 0x28
  nop
  xor rax, rax
  add rsp, 0x28
  ret

Caller:  (1 parameters)
  mov rax, 0x1             ;#1
  push rax
  pop rdi
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rdx, [rsp+0x28]      ;#1
  xor rax, rax
  add rsp, 0x38
  ret

Caller:  (2 parameters)
  mov rax, 0x2             ;#2
  push rax
  mov rax, 0x1             ;#1
  push rax
  pop rdi
  pop rsi
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  xor rax, rax
  add rsp, 0x38
  ret

Caller:  (3 parameters)
  mov rax, 0x3             ;#3
  push rax
  mov rax, 0x2             ;#2
  push rax
  mov rax, 0x1             ;#1
  push rax
  pop rdi
  pop rsi
  pop rdx
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rax, rdx
  mov [rsp+0x38], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  mov rdx, [rsp+0x38]      ;#3
  xor rax, rax
  add rsp, 0x48
  ret

Caller:  (4 parameters)
  mov rax, 0x4             ;#4
  push rax
  mov rax, 0x3             ;#3
  push rax
  mov rax, 0x2             ;#2
  push rax
  mov rax, 0x1             ;#1
  push rax
  pop rdi
  pop rsi
  pop rdx
  pop rcx
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rax, rdx
  mov [rsp+0x38], rax
  mov rax, rcx
  mov [rsp+0x40], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  mov rdx, [rsp+0x38]      ;#3
  mov rdx, [rsp+0x40]      ;#4
  xor rax, rax
  add rsp, 0x48
  ret

Caller:  (5 parameters)
  mov rax, 0x5             ;#5
  push rax
  mov rax, 0x4             ;#4
  push rax
  mov rax, 0x3             ;#3
  push rax
  mov rax, 0x2             ;#2
  push rax
  mov rax, 0x1             ;#1
  push rax
  pop rdi
  pop rsi
  pop rdx
  pop rcx
  pop r8
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  push rax
  push rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rax, rdx
  mov [rsp+0x38], rax
  mov rax, rcx
  mov [rsp+0x40], rax
  mov rax, r8
  mov [rsp+0x48], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  mov rdx, [rsp+0x38]      ;#3
  mov rdx, [rsp+0x40]      ;#4
  mov rdx, [rsp+0x48]      ;#5
  xor rax, rax
  add rsp, 0x58
  ret

Caller:  (6 parameters)
  mov rax, 0x6             ;#6
  push rax
  mov rax, 0x5             ;#5
  push rax
  mov rax, 0x4             ;#4
  push rax
  mov rax, 0x3             ;#3
  push rax
  mov rax, 0x2             ;#2
  push rax
  mov rax, 0x1             ;#1
  push rax
  pop rdi
  pop rsi
  pop rdx
  pop rcx
  pop r8
  pop r9
  call Callee()
Callee(): 
  xor rax, rax
  push rax
  push rax
  push rax
  push rax
  push rax
  push rax
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x28], rax
  mov rax, rsi
  mov [rsp+0x30], rax
  mov rax, rdx
  mov [rsp+0x38], rax
  mov rax, rcx
  mov [rsp+0x40], rax
  mov rax, r8
  mov [rsp+0x48], rax
  mov rax, r9
  mov [rsp+0x50], rax
  mov rdx, [rsp+0x28]      ;#1
  mov rdx, [rsp+0x30]      ;#2
  mov rdx, [rsp+0x38]      ;#3
  mov rdx, [rsp+0x40]      ;#4
  mov rdx, [rsp+0x48]      ;#5
  mov rdx, [rsp+0x50]      ;#6
  xor rax, rax
  add rsp, 0x58
  ret

Caller:  (7 parameters)
  sub rsp, 0x8
  mov rax, 0x7             ;#7
  push rax
  mov rax, 0x6             ;#6
  push rax
  mov rax, 0x5             ;#5
  push rax
  mov rax, 0x4             ;#4
  push rax
  mov rax, 0x3             ;#3
  push rax
  mov rax, 0x2             ;#2
  push rax
  mov rax, 0x1             ;#1
  push rax
  pop rdi
  pop rsi
  pop rdx
  pop rcx
  pop r8
  pop r9
  call Callee()
  add rsp, 0x10
Callee(): 
  mov rax, 0x8
  ClearLoop:
  sub rsp, 0x8
  mov qword [rsp+ret_addr], 0x0
  dec rax
  jne ClearLoop
  sub rsp, 0x28
  mov rax, rdi
  mov [rsp+0x20], rax
  mov rax, rsi
  mov [rsp+0x28], rax
  mov rax, rdx
  mov [rsp+0x30], rax
  mov rax, rcx
  mov [rsp+0x38], rax
  mov rax, r8
  mov [rsp+0x40], rax
  mov rax, r9
  mov [rsp+0x48], rax
  mov rax, [rsp+0x68]
  mov [rsp+0x50], rax
  mov rdx, [rsp+0x20]    ;#1
  mov rdx, [rsp+0x28]    ;#2
  mov rdx, [rsp+0x30]    ;#3
  mov rdx, [rsp+0x38]    ;#4
  mov rdx, [rsp+0x40]    ;#5
  mov rdx, [rsp+0x48]    ;#6
  mov rdx, [rsp+0x50]    ;#7
  xor rax, rax
  add rsp, 0x68
  ret
User avatar
Keya
Addict
Addict
Posts: 1891
Joined: Thu Jun 04, 2015 7:10 am

Re: Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by Keya »

perhaps easier to see side-by-side

0 parameters:

Code: Select all

WIN-x86                   LNX-x86                   WIN-x64                   LNX-x64                   MAC-x64
  call Callee()             call Callee()             call Callee()             call Callee()             call Callee()
Callee():                 Callee():                 Callee():                 Callee():                 Callee():
  nop                       xor eax, eax              sub rsp, 0x28             sub rsp, 0x28             sub rsp, 0x28
  xor eax, eax              push eax                  nop                       nop                       nop
  ret                       push eax                  xor rax, rax              xor rax, rax              xor rax, rax
                            push eax                  add rsp, 0x28             add rsp, 0x28             add rsp, 0x28
                            nop                       ret                       ret                       ret
                            xor eax, eax
                            add esp, 0xc
                            ret
1 parameter:

Code: Select all

WIN-x86                   LNX-x86                   WIN-x64                   LNX-x64                   MAC-x64
  push dword 0x1            sub esp, 0xc              mov rcx, 0x1              push dword 0x1            mov rax, 0x1
  call Callee()             push dword 0x1            call Callee()             pop rdi                   push rax
Callee():                   call Callee()           Callee():                   call Callee()             pop rdi
  mov edx, [esp+0x4]        add esp, 0xc              mov [rsp+0x8], rcx      Callee():                   call Callee()
  xor eax, eax            Callee():                   sub rsp, 0x28             xor rax, rax            Callee():
  ret 0x4                   xor eax, eax              mov rdx, [rsp+0x30]       push rax                  xor rax, rax
                            push eax                  xor rax, rax              push rax                  push rax
                            push eax                  add rsp, 0x28             sub rsp, 0x28             push rax
                            push eax                  ret                       mov rax, rdi              sub rsp, 0x28
                            mov edx, [esp+0x10]                                 mov [rsp+0x28], rax       mov rax, rdi
                            xor eax, eax                                        mov rdx, [rsp+0x28]       mov [rsp+0x28], rax
                            add esp, 0xc                                        xor rax, rax              mov rdx, [rsp+0x28]
                            ret 0x4                                             add rsp, 0x38             xor rax, rax
                                                                                ret                       add rsp, 0x38
                                                                                                          ret
2 parameters:

Code: Select all

WIN-x86                   LNX-x86                   WIN-x64                   LNX-x64                   MAC-x64
  push dword 0x2            sub esp, 0x8              mov rdx, 0x2              push dword 0x2            mov rax, 0x2
  push dword 0x1            push dword 0x2            mov rcx, 0x1              push dword 0x1            push rax
  call Callee()             push dword 0x1            call Callee()             pop rdi                   mov rax, 0x1
Callee():                   call Callee()           Callee():                   pop rsi                   push rax
  mov edx, [esp+0x4]        add esp, 0x8              mov [rsp+0x8], rcx        call Callee()             pop rdi
  mov edx, [esp+0x8]      Callee():                   mov [rsp+0x10], rdx     Callee():                   pop rsi
  xor eax, eax              xor eax, eax              sub rsp, 0x28             xor rax, rax              call Callee()
  ret 0x8                   push eax                  mov rdx, [rsp+0x30]       push rax                Callee():
                            push eax                  mov rdx, [rsp+0x38]       push rax                  xor rax, rax
                            push eax                  xor rax, rax              sub rsp, 0x28             push rax
                            mov edx, [esp+0x10]       add rsp, 0x28             mov rax, rdi              push rax
                            mov edx, [esp+0x14]       ret                       mov [rsp+0x28], rax       sub rsp, 0x28
                            xor eax, eax                                        mov rax, rsi              mov rax, rdi
                            add esp, 0xc                                        mov [rsp+0x30], rax       mov [rsp+0x28], rax
                            ret 0x8                                             mov rdx, [rsp+0x28]       mov rax, rsi
                                                                                mov rdx, [rsp+0x30]       mov [rsp+0x30], rax
                                                                                xor rax, rax              mov rdx, [rsp+0x28]
                                                                                add rsp, 0x38             mov rdx, [rsp+0x30]
                                                                                ret                       xor rax, rax
                                                                                                          add rsp, 0x38
                                                                                                          ret
3 parameters:

Code: Select all

WIN-x86                   LNX-x86                   WIN-x64                   LNX-x64                   MAC-x64
  push dword 0x3            sub esp, 0x4              mov r8, 0x3               push dword 0x3            mov rax, 0x3
  push dword 0x2            push dword 0x3            mov rdx, 0x2              push dword 0x2            push rax
  push dword 0x1            push dword 0x2            mov rcx, 0x1              push dword 0x1            mov rax, 0x2
  call Callee()             push dword 0x1            call Callee()             pop rdi                   push rax
Callee():                   call Callee()           Callee():                   pop rsi                   mov rax, 0x1
  mov edx, [esp+0x4]        add esp, 0x4              mov [rsp+0x8], rcx        pop rdx                   push rax
  mov edx, [esp+0x8]      Callee():                   mov [rsp+0x10], rdx       call Callee()             pop rdi
  mov edx, [esp+0xc]        xor eax, eax              mov [rsp+0x18], r8      Callee():                   pop rsi
  xor eax, eax              push eax                  sub rsp, 0x28             xor rax, rax              pop rdx
  ret 0xc                   push eax                  mov rdx, [rsp+0x30]       push rax                  call Callee()
                            push eax                  mov rdx, [rsp+0x38]       push rax                Callee():
                            mov edx, [esp+0x10]       mov rdx, [rsp+0x40]       push rax                  xor rax, rax
                            mov edx, [esp+0x14]       xor rax, rax              push rax                  push rax
                            mov edx, [esp+0x18]       add rsp, 0x28             sub rsp, 0x28             push rax
                            xor eax, eax              ret                       mov rax, rdi              push rax
                            add esp, 0xc                                        mov [rsp+0x28], rax       push rax
                            ret 0xc                                             mov rax, rsi              sub rsp, 0x28
                                                                                mov [rsp+0x30], rax       mov rax, rdi
                                                                                mov rax, rdx              mov [rsp+0x28], rax
                                                                                mov [rsp+0x38], rax       mov rax, rsi
                                                                                mov rdx, [rsp+0x28]       mov [rsp+0x30], rax
                                                                                mov rdx, [rsp+0x30]       mov rax, rdx
                                                                                mov rdx, [rsp+0x38]       mov [rsp+0x38], rax
                                                                                xor rax, rax              mov rdx, [rsp+0x28]
                                                                                add rsp, 0x48             mov rdx, [rsp+0x30]
                                                                                ret                       mov rdx, [rsp+0x38]
                                                                                                          xor rax, rax
                                                                                                          add rsp, 0x48
                                                                                                          ret
4 parameters:

Code: Select all

WIN-x86                   LNX-x86                   WIN-x64                   LNX-x64                   MAC-x64
  push dword 0x4            push dword 0x4            mov r9, 0x4               push dword 0x4            mov rax, 0x4
  push dword 0x3            push dword 0x3            mov r8, 0x3               push dword 0x3            push rax
  push dword 0x2            push dword 0x2            mov rdx, 0x2              push dword 0x2            mov rax, 0x3
  push dword 0x1            push dword 0x1            mov rcx, 0x1              push dword 0x1            push rax
  call Callee()             call Callee()             call Callee()             pop rdi                   mov rax, 0x2
Callee():                 Callee():                 Callee():                   pop rsi                   push rax
  mov edx, [esp+0x4]        xor eax, eax              mov [rsp+0x8], rcx        pop rdx                   mov rax, 0x1
  mov edx, [esp+0x8]        push eax                  mov [rsp+0x10], rdx       pop rcx                   push rax
  mov edx, [esp+0xc]        push eax                  mov [rsp+0x18], r8        call Callee()             pop rdi
  mov edx, [esp+0x10]       push eax                  mov [rsp+0x20], r9      Callee():                   pop rsi
  xor eax, eax              mov edx, [esp+0x10]       sub rsp, 0x28             xor rax, rax              pop rdx
  ret 0x10                  mov edx, [esp+0x14]       mov rdx, [rsp+0x30]       push rax                  pop rcx
                            mov edx, [esp+0x18]       mov rdx, [rsp+0x38]       push rax                  call Callee()
                            mov edx, [esp+0x1c]       mov rdx, [rsp+0x40]       push rax                Callee():
                            xor eax, eax              mov rdx, [rsp+0x48]       push rax                  xor rax, rax
                            add esp, 0xc              xor rax, rax              sub rsp, 0x28             push rax
                            ret 0x10                  add rsp, 0x28             mov rax, rdi              push rax
                                                      ret                       mov [rsp+0x28], rax       push rax
                                                                                mov rax, rsi              push rax
                                                                                mov [rsp+0x30], rax       sub rsp, 0x28
                                                                                mov rax, rdx              mov rax, rdi
                                                                                mov [rsp+0x38], rax       mov [rsp+0x28], rax
                                                                                mov rax, rcx              mov rax, rsi
                                                                                mov [rsp+0x40], rax       mov [rsp+0x30], rax
                                                                                mov rdx, [rsp+0x28]       mov rax, rdx
                                                                                mov rdx, [rsp+0x30]       mov [rsp+0x38], rax
                                                                                mov rdx, [rsp+0x38]       mov rax, rcx
                                                                                mov rdx, [rsp+0x40]       mov [rsp+0x40], rax
                                                                                xor rax, rax              mov rdx, [rsp+0x28]
                                                                                add rsp, 0x48             mov rdx, [rsp+0x30]
                                                                                ret                       mov rdx, [rsp+0x38]
                                                                                                          mov rdx, [rsp+0x40]
                                                                                                          xor rax, rax
                                                                                                          add rsp, 0x48
                                                                                                          ret
5 parameters:

Code: Select all

WIN-x86                   LNX-x86                   WIN-x64                   LNX-x64                   MAC-x64
  push dword 0x5            sub esp, 0xc              sub rsp, 0x8              push dword 0x5            mov rax, 0x5
  push dword 0x4            push dword 0x5            push dword 0x5            push dword 0x4            push rax
  push dword 0x3            push dword 0x4            mov r9, 0x4               push dword 0x3            mov rax, 0x4
  push dword 0x2            push dword 0x3            mov r8, 0x3               push dword 0x2            push rax
  push dword 0x1            push dword 0x2            mov rdx, 0x2              push dword 0x1            mov rax, 0x3
  call Callee()             push dword 0x1            mov rcx, 0x1              pop rdi                   push rax
Callee():                   call Callee()             sub rsp, 0x20             pop rsi                   mov rax, 0x2
  mov edx, [esp+0x4]        add esp, 0xc              call Callee()             pop rdx                   push rax
  mov edx, [esp+0x8]      Callee():                   add rsp, 0x30             pop rcx                   mov rax, 0x1
  mov edx, [esp+0xc]        xor eax, eax            Callee():                   pop r8                    push rax
  mov edx, [esp+0x10]       push eax                  mov [rsp+0x8], rcx        call Callee()             pop rdi
  mov edx, [esp+0x14]       push eax                  mov [rsp+0x10], rdx     Callee():                   pop rsi
  xor eax, eax              push eax                  mov [rsp+0x18], r8        xor rax, rax              pop rdx
  ret 0x14                  mov edx, [esp+0x10]       mov [rsp+0x20], r9        push rax                  pop rcx
                            mov edx, [esp+0x14]       sub rsp, 0x28             push rax                  pop r8
                            mov edx, [esp+0x18]       mov rdx, [rsp+0x30]       push rax                  call Callee()
                            mov edx, [esp+0x1c]       mov rdx, [rsp+0x38]       push rax                Callee():
                            mov edx, [esp+0x20]       mov rdx, [rsp+0x40]       push rax                  xor rax, rax
                            xor eax, eax              mov rdx, [rsp+0x48]       push rax                  push rax
                            add esp, 0xc              mov rdx, [rsp+0x50]       sub rsp, 0x28             push rax
                            ret 0x14                  xor rax, rax              mov rax, rdi              push rax
                                                      add rsp, 0x28             mov [rsp+0x28], rax       push rax
                                                      ret                       mov rax, rsi              push rax
                                                                                mov [rsp+0x30], rax       push rax
                                                                                mov rax, rdx              sub rsp, 0x28
                                                                                mov [rsp+0x38], rax       mov rax, rdi
                                                                                mov rax, rcx              mov [rsp+0x28], rax
                                                                                mov [rsp+0x40], rax       mov rax, rsi
                                                                                mov rax, r8               mov [rsp+0x30], rax
                                                                                mov [rsp+0x48], rax       mov rax, rdx
                                                                                mov rdx, [rsp+0x28]       mov [rsp+0x38], rax
                                                                                mov rdx, [rsp+0x30]       mov rax, rcx
                                                                                mov rdx, [rsp+0x38]       mov [rsp+0x40], rax
                                                                                mov rdx, [rsp+0x40]       mov rax, r8
                                                                                mov rdx, [rsp+0x48]       mov [rsp+0x48], rax
                                                                                xor rax, rax              mov rdx, [rsp+0x28]
                                                                                add rsp, 0x58             mov rdx, [rsp+0x30]
                                                                                ret                       mov rdx, [rsp+0x38]
                                                                                                          mov rdx, [rsp+0x40]
                                                                                                          mov rdx, [rsp+0x48]
                                                                                                          xor rax, rax
                                                                                                          add rsp, 0x58
                                                                                                          ret
6 parameters:

Code: Select all

WIN-x86                   LNX-x86                   WIN-x64                   LNX-x64                   MAC-x64
  push dword 0x6            sub esp, 0x8              push dword 0x6            push dword 0x6            mov rax, 0x6
  push dword 0x5            push dword 0x6            push dword 0x5            push dword 0x5            push rax
  push dword 0x4            push dword 0x5            mov r9, 0x4               push dword 0x4            mov rax, 0x5
  push dword 0x3            push dword 0x4            mov r8, 0x3               push dword 0x3            push rax
  push dword 0x2            push dword 0x3            mov rdx, 0x2              push dword 0x2            mov rax, 0x4
  push dword 0x1            push dword 0x2            mov rcx, 0x1              push dword 0x1            push rax
  call Callee()             push dword 0x1            sub rsp, 0x20             pop rdi                   mov rax, 0x3
Callee():                   call Callee()             call Callee()             pop rsi                   push rax
  mov edx, [esp+0x4]        add esp, 0x8              add rsp, 0x30             pop rdx                   mov rax, 0x2
  mov edx, [esp+0x8]      Callee():                 Callee():                   pop rcx                   push rax
  mov edx, [esp+0xc]        xor eax, eax              mov [rsp+0x8], rcx        pop r8                    mov rax, 0x1
  mov edx, [esp+0x10]       push eax                  mov [rsp+0x10], rdx       pop r9                    push rax
  mov edx, [esp+0x14]       push eax                  mov [rsp+0x18], r8        call Callee()             pop rdi
  mov edx, [esp+0x18]       push eax                  mov [rsp+0x20], r9      Callee():                   pop rsi
  xor eax, eax              mov edx, [esp+0x10]       sub rsp, 0x28             xor rax, rax              pop rdx
  ret 0x18                  mov edx, [esp+0x14]       mov rdx, [rsp+0x30]       push rax                  pop rcx
                            mov edx, [esp+0x18]       mov rdx, [rsp+0x38]       push rax                  pop r8
                            mov edx, [esp+0x1c]       mov rdx, [rsp+0x40]       push rax                  pop r9
                            mov edx, [esp+0x20]       mov rdx, [rsp+0x48]       push rax                  call Callee()
                            mov edx, [esp+0x24]       mov rdx, [rsp+0x50]       push rax                Callee():
                            xor eax, eax              mov rdx, [rsp+0x58]       push rax                  xor rax, rax
                            add esp, 0xc              xor rax, rax              sub rsp, 0x28             push rax
                            ret 0x18                  add rsp, 0x28             mov rax, rdi              push rax
                                                      ret                       mov [rsp+0x28], rax       push rax
                                                                                mov rax, rsi              push rax
                                                                                mov [rsp+0x30], rax       push rax
                                                                                mov rax, rdx              push rax
                                                                                mov [rsp+0x38], rax       sub rsp, 0x28
                                                                                mov rax, rcx              mov rax, rdi
                                                                                mov [rsp+0x40], rax       mov [rsp+0x28], rax
                                                                                mov rax, r8               mov rax, rsi
                                                                                mov [rsp+0x48], rax       mov [rsp+0x30], rax
                                                                                mov rax, r9               mov rax, rdx
                                                                                mov [rsp+0x50], rax       mov [rsp+0x38], rax
                                                                                mov rdx, [rsp+0x28]       mov rax, rcx
                                                                                mov rdx, [rsp+0x30]       mov [rsp+0x40], rax
                                                                                mov rdx, [rsp+0x38]       mov rax, r8
                                                                                mov rdx, [rsp+0x40]       mov [rsp+0x48], rax
                                                                                mov rdx, [rsp+0x48]       mov rax, r9
                                                                                mov rdx, [rsp+0x50]       mov [rsp+0x50], rax
                                                                                xor rax, rax              mov rdx, [rsp+0x28]
                                                                                add rsp, 0x58             mov rdx, [rsp+0x30]
                                                                                ret                       mov rdx, [rsp+0x38]
                                                                                                          mov rdx, [rsp+0x40]
                                                                                                          mov rdx, [rsp+0x48]
                                                                                                          mov rdx, [rsp+0x50]
                                                                                                          xor rax, rax
                                                                                                          add rsp, 0x58
                                                                                                          ret
7 parameters:

Code: Select all

WIN-x86                   LNX-x86                   WIN-x64                   LNX-x64                   MAC-x64
  push dword 0x7            sub esp, 0x4              sub rsp, 0x8              sub rsp, 0x8              sub rsp, 0x8
  push dword 0x6            push dword 0x7            push dword 0x7            push dword 0x7            mov rax, 0x7
  push dword 0x5            push dword 0x6            push dword 0x6            push dword 0x6            push rax
  push dword 0x4            push dword 0x5            push dword 0x5            push dword 0x5            mov rax, 0x6
  push dword 0x3            push dword 0x4            mov r9, 0x4               push dword 0x4            push rax
  push dword 0x2            push dword 0x3            mov r8, 0x3               push dword 0x3            mov rax, 0x5
  push dword 0x1            push dword 0x2            mov rdx, 0x2              push dword 0x2            push rax
  call Callee()             push dword 0x1            mov rcx, 0x1              push dword 0x1            mov rax, 0x4
Callee():                   call Callee()             sub rsp, 0x20             pop rdi                   push rax
  mov edx, [esp+0x4]        add esp, 0x4              call Callee()             pop rsi                   mov rax, 0x3
  mov edx, [esp+0x8]      Callee():                   add rsp, 0x40             pop rdx                   push rax
  mov edx, [esp+0xc]        xor eax, eax            Callee():                   pop rcx                   mov rax, 0x2
  mov edx, [esp+0x10]       push eax                  mov [rsp+0x8], rcx        pop r8                    push rax
  mov edx, [esp+0x14]       push eax                  mov [rsp+0x10], rdx       pop r9                    mov rax, 0x1
  mov edx, [esp+0x18]       push eax                  mov [rsp+0x18], r8        call Callee()             push rax
  mov edx, [esp+0x1c]       mov edx, [esp+0x10]       mov [rsp+0x20], r9        add rsp, 0x10             pop rdi
  xor eax, eax              mov edx, [esp+0x14]       sub rsp, 0x28           Callee():                   pop rsi
  ret 0x1c                  mov edx, [esp+0x18]       mov rdx, [rsp+0x30]       mov rax, 0x8              pop rdx
                            mov edx, [esp+0x1c]       mov rdx, [rsp+0x38]       sub rsp, 0x8              pop rcx
                            mov edx, [esp+0x20]       mov rdx, [rsp+0x40]       mov qword [rsp], 0x0      pop r8
                            mov edx, [esp+0x24]       mov rdx, [rsp+0x48]       invalid                   pop r9
                            mov edx, [esp+0x28]       mov rdx, [rsp+0x50]                                 call Callee()
                            xor eax, eax              mov rdx, [rsp+0x58]                                 add rsp, 0x10
                            add esp, 0xc              mov rdx, [rsp+0x60]                               Callee():
                            ret 0x1c                  xor rax, rax                                        mov rax, 0x8
                                                      add rsp, 0x28                                       ClearLoop:
                                                      ret                                                 sub rsp, 0x8
                                                                                                          mov qword [rsp+ret_addr], 0x0
                                                                                                          dec rax
                                                                                                          jne ClearLoop
                                                                                                          sub rsp, 0x28
                                                                                                          mov rax, rdi
                                                                                                          mov [rsp+0x20], rax
                                                                                                          mov rax, rsi
                                                                                                          mov [rsp+0x28], rax
                                                                                                          mov rax, rdx
                                                                                                          mov [rsp+0x30], rax
                                                                                                          mov rax, rcx
                                                                                                          mov [rsp+0x38], rax
                                                                                                          mov rax, r8
                                                                                                          mov [rsp+0x40], rax
                                                                                                          mov rax, r9
                                                                                                          mov [rsp+0x48], rax
                                                                                                          mov rax, [rsp+0x68]
                                                                                                          mov [rsp+0x50], rax
                                                                                                          mov rdx, [rsp+0x20]
                                                                                                          mov rdx, [rsp+0x30]
                                                                                                          mov rdx, [rsp+0x38]
                                                                                                          mov rdx, [rsp+0x40]
                                                                                                          mov rdx, [rsp+0x48]
                                                                                                          mov rdx, [rsp+0x50]
                                                                                                          xor rax, rax
                                                                                                          add rsp, 0x68
                                                                                                          ret
wilbert
PureBasic Expert
PureBasic Expert
Posts: 3870
Joined: Sun Aug 08, 2004 5:21 am
Location: Netherlands

Re: Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by wilbert »

Things change of course when you mix integer with floating point function arguments. :)
On OSX stack alignment is also important.
Windows (x64)
Raspberry Pi OS (Arm64)
ker2x
User
User
Posts: 38
Joined: Sat May 10, 2008 7:52 am
Location: SecondLife

Re: Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by ker2x »

Sticky \o/
User avatar
Crusiatus Black
Enthusiast
Enthusiast
Posts: 389
Joined: Mon May 12, 2008 1:25 pm
Location: The Netherlands
Contact:

Re: Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by Crusiatus Black »

Really interesting!
Image
Bas Groothedde,
Imagine Programming

I live in a philosophical paradoxal randome filled with enigma's!
User avatar
Keya
Addict
Addict
Posts: 1891
Joined: Thu Jun 04, 2015 7:10 am

Re: Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by Keya »

wilbert wrote:On OSX stack alignment is also important.
uhh ohhhh... thankyou for mentioning this because it looks like that's the issue i've just run into!!! I had everything finally going well on Windows after getting the structure elements aligned, but still not working on Mac and stack alignment is the only reason I can think of, especially being a memory access error that doesn't make any other sense, and well, the instruction is the aligned movdqa afterall! I dont suppose you have a little example to accommodate these shenanigans?!? (something like this maybe? looks similar to what PB is compiling with sub esp/rsp). I was doing push's, but I'm guessing for OSX i need something like "sub esp, n", but im guessing the compiler needs to determine that n value, hrm. I cant quite get my head around it at this stage, more coffee and flooding my brain with hexadecimals in the meantime, lol

something like this maybe!?

Code: Select all

Define oldsp.l
!mov [v_oldsp], esp
!and esp, -16         ;align stack

;... do stack-aligned work

!mov esp, [v_oldsp]   ;restore stack
MessageRequester("OK","OK")
Last edited by Keya on Fri Oct 07, 2016 11:54 pm, edited 2 times in total.
User avatar
Keya
Addict
Addict
Posts: 1891
Joined: Thu Jun 04, 2015 7:10 am

Re: Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by Keya »

btw, strange problem... the following works/aligns fine in Windows/fasm, but is not aligning in OSX/yasm :(

Code: Select all

MessageRequester("Mod", Str(Mod(?testaligned,16)))  ;returns 0 if 16-aligned
End

! align 16
testaligned:
! db 0
Strange because yasm supports the align statement, and everything seems ok in the assembly PB creates:

Code: Select all

; !align 16
align 16
; testaligned:
l_testaligned:
; ! db 0
 db     0
wilbert
PureBasic Expert
PureBasic Expert
Posts: 3870
Joined: Sun Aug 08, 2004 5:21 am
Location: Netherlands

Re: Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by wilbert »

The requirement for stack alignment is something the OS demands.
If you call an api function from your own asm code, at that time the stack needs to be aligned.
If you only jump around in your own code, it shouldn't matter.
If you import a procedure and use it like any other PB procedure in your PB code, it also shouldn't be a problem as PB takes care of the alignment.

In this case it looks like an alignment problem of Nasm. Yasm (x64) works fine but Nasm (x86) doesn't.
Nasm wrote:A final caveat: ALIGN and ALIGNB work relative to the beginning of the section, not the beginning of the address space in the final executable. Aligning to a 16-byte boundary when the section you're in is only guaranteed to be aligned to a 4-byte boundary, for example, is a waste of effort.
It's probably the section which isn't aligned to a 16 byte boundary.

Yasm mentions the same for ALIGNB but for yasm, ALIGN is more intelligent and does adjust the section alignment to be the maximum specified alignment.
As a workaround you could simply replace Nasm with Yasm.
Might be good for Fred to do (replace Nasm with Yasm for x86). I don't see any reason why we should have two different assemblers on OSX.
Windows (x64)
Raspberry Pi OS (Arm64)
User avatar
tj1010
Enthusiast
Enthusiast
Posts: 622
Joined: Mon Feb 25, 2013 5:51 pm
Location: US or Estonia
Contact:

Re: Stack caller & callee, x86/x64, Windows/Linux/OSX

Post by tj1010 »

Doing this without proxy handlers or pointers or some loader is actually impossible. A loader could at least use debug API to monitor RETN, RETF and RET or inline patch them by using free bytes in a execute ELF or PE section.
The truth hurts.
Post Reply