Nintendo 64 ABI and Calling Convention

The Nintendo 64 uses a MIPS architecture NEC VR4300i, and applications created for the system are all compiled with C or C++. For the assembly developer looking to hack games that use C, knowing the calling convention used is one of the most important things you can know.

The MIPS o32 ABI determines 3 things: The register names and uses, the allocation of the stack, and how to effectively call functions using those.

Four registers are used for passing arguments to a function: a0, a1, a2, and a3. These arguments can hold any value up to 32 bits each, which includes pointer addresses.

myFunc(0, 1, 0x12345678, &myPointer);

li a0, 0
li a1, 1
li a2, 0x12345678
la a3, myPointer ; Loads an address value
jal myFunc
; make sure 'li' and 'la' instructions
; don't fall into a delay slot,
; otherwise only half of the value
; will load.
nop

If a function returns a value, it will be stored in v0.

myFunc(myReturningFunc());

jal myReturningFunc
nop
move a0, v0 ; move v0 to a0
jal myFunc
nop

The ABI specifies four argument registers, so the natural thing to wonder is where a to put a fifth argument, sixth argument, and so on. The answer is to put them on the Stack. The general layout of a stack frame (the view of the stack from any arbitrary function) is as shown:

Let's see how this looks in practice:

myFunc(0, 1, 2, 3, 4, 5, 6, 7, 8);

li a0, 0
li a1, 1
li a2, 2
li a3, 3
li t0, 4
sw t0, 0x10(sp)
li t0, 5
sw t0, 0x14(sp)
li t0, 6
sw t0, 0x18(sp)
li t0, 7
sw t0, 0x1C(sp)
li t0, 8
sw t0, 0x20(sp)
jal myFunc
nop

Now the question is where the top of the stack is, and how to avoid it. This is also explained in code:

void myFunc(int a, int b, int c, int d, int e, int f, int g, int h) {...

glabel myFunc
    addiu sp, -0x30
    sw ra, 0x2C(sp)
    ...
    lw ra, 0x2C(sp)
    jr ra
    addiu sp, 0x30

The addiu instruction determines how much gets allocated to the stack, and the return address is placed at the top of this region.

When a function takes floating point arguments, the ABI assumes you're using float registers more often and gives each of them their own purpose so you dont have to use general purpose registers too much.

float three_input_adder(float a, float b, float c) {
    return a + b + c;
}
void myFunc(void) {
    float x = three_input_adder(1.0f, 3.0f, 4.0f);
}

glabel three_input_adder
    lwc1 f4, 0x10(sp)
    add.s f0, f12, f14
    add.s f0, f0, f4 ; return value gets stored in f0
    jr ra
     nop
glabel myFunc
    addiu sp, -0x18
    sw ra, 0x14(sp)
    
    li.s f12, 1.0 ; f12 holds the first argument
    li.s f14, 3.0 ; f14 holds the second argument
    li.s f4,  4.0
    swc1 f4, 0x10(sp) ; subsequent float arguments go on the stack
    jal three_input_adder
     nop
    
    mfc1 t0, f0 ; f0 has the result
    
    lw ra, 0x14(sp)
    jr ra
     addiu sp, 0x18