LithosAnanake/docs/lithosananke/riscv64-stack-fix/README.md

# Kernel Stack Trampoline Fix — riscv64

**Date**: 2026-08-02  
**Branch**: `master`  
**Commit**: `7366275`

---

## Problem

The riscv64 kernel crashed partway through VM bootstrap during the 2026-08-01
DoE campaign run, well after amd64 and aarch64 had both booted cleanly to
`ok>` in the same session. The crash was a wild jump: execution landed at an
address nowhere inside the kernel's own linked image, then almost immediately
faulted trying to load from a small, near-null address.

```
Registering FORTH-79 Standard word set...
Registering FORTH-79 arithmetic words...
FORTH-79 Standard word set registration complete
*** EXCEPTION (riscv64) ***
  scause = 0x000000000000000d
  sepc   = 0x00000000be27c3c2
  stval  = 0x00000000000074d8
```

`scause = 0xd` is a supervisor-mode load page fault. `sepc` (~0xbe27c3c2) sat
near the top of QEMU's 1 GB RAM window — nowhere close to the kernel's own
`.text`/`.data`/`.bss` footprint (`0x400000`–`0xcdf9a0`, confirmed via `nm`/
`readelf` on the exact build that produced the log). `stval` (0x74d8, ~30 KB)
stayed fixed across rebuilds even as `sepc` drifted — the signature of a
corrupted return address rather than a fixed logic bug at one source line.

## Investigation

Two dead ends were ruled out before the real cause surfaced:

1. **ELF relocation / buffer overlap in the split-build loader.** The
   codebase has a `!MONOLITHIC_BUILD` path (`elf_load_kernel()` in
   `elf_loader.c`) that loads `kernel.elf` from the ESP into a UEFI
   `AllocateAnyPages` buffer. This looked like a plausible overlap candidate
   at first, but `Makefile.starkernel` defaults to `MONOLITHIC=1`, meaning
   `kernel_main` is linked directly into the loader and that code path never
   runs. Confirmed by checking the built binary for the investigation's own
   diagnostic strings — absent, because the compilation unit was built with
   `-DMONOLITHIC_BUILD` and the code was preprocessed out.
2. **`RAW_LOG()` silently no-ops on every non-amd64 arch.** All of
   `load_kernel_from_esp()`'s narration (and an early ConOut-based
   diagnostic) never reached the captured serial log, on riscv64, in any
   build. `RAW_LOG` was defined only for `ARCH_AMD64`; everywhere else it
   expanded to `((void)0)`. This made straightforward diagnosis impossible
   until a real raw-UART writer was added for riscv64 (see Secondary Fix).

With real output available, `console_println()` checkpoints were bisected
through `vm_init_with_host()` and `sk_vm_bootstrap_parity()`. Every checkpoint
inside `vm_init_with_host()` printed successfully, including the very last
line of the function — but the checkpoint immediately after its *return*
never appeared. Re-running with a trivial, unrelated local-variable addition
shifted the crash earlier and changed the exception type entirely (from a
load fault to an instruction-fetch fault at `sepc == stval`, i.e. the CPU
tried to *execute* an unmapped address). A crash whose location and even
category shift under unrelated code changes is the standard signature of
stack corruption, not a fixed bug at a specific call site.

## Root Cause

`kernel_main.c` documented the gap directly, once read closely:

> On amd64, `kernel_entry.S` switches the stack from UEFI's default to a
> 2 MiB zero-initialised BSS stack and tail-calls this function as
> `kernel_main_impl`. On aarch64 and riscv64 the assembly trampoline is not
> yet implemented and the UEFI loader calls `kernel_main` directly.

amd64 has carried a dedicated stack-switch trampoline
(`src/starkernel/arch/amd64/kernel_entry.S`) since early on, with its own
comment explaining why: *"UEFI hands control to kernel_main() running on a
small UEFI-provided stack (typically 128 KB–256 KB). The FORTH interpreter +
DOE experiment loop can easily exceed that depth."* riscv64 never had the
equivalent. It ran the entire VM bootstrap — 27 chained word-registration
modules, physics/SSM init, Tripod capsule birth — directly on whatever small
stack EDK2's riscv64 firmware handed the loader, overflowing it and
corrupting a return address. aarch64 has the same theoretical exposure but
happens not to trip it, apparently because its firmware's default boot stack
is larger — incidental, not a guarantee.

## Fix

**`src/starkernel/arch/riscv64/kernel_entry.S`** (new) — RISC-V stack-switch
trampoline mirroring amd64's:

```asm
    .section .bss
    .align 4
    .global g_kernel_stack
g_kernel_stack:
    .space 0x200000
g_kernel_stack_top:
    .global g_kernel_stack_top

    .section .text
    .extern kernel_main_impl
    .global kernel_main
kernel_main:
    ld   t0, BOOT_INFO_KERNEL_STACK_BASE_OFFSET(a0)
    bnez t0, .Ldynamic_stack
    la   t0, g_kernel_stack_top
    j    .Lstack_ready
.Ldynamic_stack:
    ld   t1, BOOT_INFO_KERNEL_STACK_SIZE_OFFSET(a0)
    add  t0, t0, t1
.Lstack_ready:
    andi t0, t0, -16
    mv   sp, t0
    mv   s0, zero
    tail kernel_main_impl
```

**`src/starkernel/kernel_main.c`** — riscv64 now builds `kernel_main_impl`
(invoked via the trampoline) instead of `kernel_main` directly, same pattern
as amd64:

```c
#if defined(__x86_64__) || defined(__riscv)
void kernel_main_impl(BootInfo *boot_info) {
#else
void kernel_main(BootInfo *boot_info) {
#endif
```

**`Makefile.starkernel`** — wires the new file into the riscv64 loader build:

```makefile
ifeq ($(ARCH),riscv64)
LOADER_ASM += $(KERNEL_SRC)/arch/$(ARCH)/kernel_entry.S
endif
```

## Secondary Fix

`RAW_LOG()` in `src/starkernel/boot/uefi_loader.c` was a no-op on every arch
except amd64, silencing all of the loader's own diagnostic messages on
riscv64 (and aarch64). Added a real raw-UART writer for riscv64, targeting
QEMU virt's 16550-compatible `uart8250` at MMIO `0x10000000` (matches
`Domain0 Region03` in the OpenSBI boot banner) — the same byte-polling
protocol as the existing amd64 port-I/O version, just MMIO instead of `outb`/
`inb`. Existing and future `RAW_LOG()` call sites in the loader now actually
produce output on riscv64.

## Result

| Criterion | Before | After |
|---|---|---|
| `sepc` after word registration | Wild address outside kernel image (`0xbe27c3c2`-ish, drifting per build) | N/A — no longer occurs |
| Exception | Load or instruction-fetch page fault, non-deterministic between builds | None |
| `[Hera] ok>` (riscv64) | Never reached | ✅ reached, Tripod fleet (Hera/Hermes/Artemis) runs to completion |
| `PASS: E2E msg flow` | Never logged | ✅ |
| `PARITY:MAMA_INIT` / `Init: Mama birth OK` | Never logged | ✅ |
| Loader `RAW_LOG()` output on riscv64 | Silent no-op | ✅ visible in serial log |

## Acceptance Testing

All three architectures must boot to `ok>`, run the Tripod fleet, and reach
`[Hera] ok>`. Per repo convention, run in this exact order, one QEMU instance
at a time, always `clean` before `qemu`:

```bash
make -f Makefile.starkernel ARCH=amd64   clean qemu
make -f Makefile.starkernel ARCH=aarch64 clean qemu
make -f Makefile.starkernel ARCH=riscv64 clean qemu
```

Verify each log:

```bash
grep "ok>"                 logs/<session>/<arch>/qemu-<arch>-*.log
grep "PASS: E2E msg flow"  logs/<session>/<arch>/qemu-<arch>-*.log
grep "PARITY:MAMA_INIT"    logs/<session>/<arch>/qemu-<arch>-*.log
grep "EXCEPTION"           logs/<session>/<arch>/qemu-<arch>-*.log   # must be empty
```

All three architectures confirmed clean on 2026-08-02 (see
`logs/20260802-064050/riscv64/`, `logs/20260802-064424/amd64/`,
`logs/20260802-064656/aarch64/`, committed alongside this fix).