LithosAnanake/docs/lithosananke/riscv64-stack-fix/README.md

Kernel Stack Trampoline Fix — riscv64

Date: 2026-08-02
Branch: master
Commit: 7366275

---

Problem

The riscv64 kernel crashed partway through VM bootstrap during the 2026-08-01 DoE campaign run, well after amd64 and aarch64 had both booted cleanly to ok> in the same session. The crash was a wild jump: execution landed at an address nowhere inside the kernel's own linked image, then almost immediately faulted trying to load from a small, near-null address.

Registering FORTH-79 Standard word set...
Registering FORTH-79 arithmetic words...
FORTH-79 Standard word set registration complete
*** EXCEPTION (riscv64) ***
  scause = 0x000000000000000d
  sepc   = 0x00000000be27c3c2
  stval  = 0x00000000000074d8

scause = 0xd is a supervisor-mode load page fault. sepc (~0xbe27c3c2) sat near the top of QEMU's 1 GB RAM window — nowhere close to the kernel's own .text/.data/.bss footprint (0x400000–0xcdf9a0, confirmed via nm/ readelf on the exact build that produced the log). stval (0x74d8, ~30 KB) stayed fixed across rebuilds even as sepc drifted — the signature of a corrupted return address rather than a fixed logic bug at one source line.

Investigation

Two dead ends were ruled out before the real cause surfaced:

1. ELF relocation / buffer overlap in the split-build loader. The codebase has a !MONOLITHIC_BUILD path (elf_load_kernel() in elf_loader.c) that loads kernel.elf from the ESP into a UEFI AllocateAnyPages buffer. This looked like a plausible overlap candidate at first, but Makefile.starkernel defaults to MONOLITHIC=1, meaning kernel_main is linked directly into the loader and that code path never runs. Confirmed by checking the built binary for the investigation's own diagnostic strings — absent, because the compilation unit was built with -DMONOLITHIC_BUILD and the code was preprocessed out.
2. RAW_LOG() silently no-ops on every non-amd64 arch. All of load_kernel_from_esp()'s narration (and an early ConOut-based diagnostic) never reached the captured serial log, on riscv64, in any build. RAW_LOG was defined only for ARCH_AMD64; everywhere else it expanded to ((void)0). This made straightforward diagnosis impossible until a real raw-UART writer was added for riscv64 (see Secondary Fix).

With real output available, console_println() checkpoints were bisected through vm_init_with_host() and sk_vm_bootstrap_parity(). Every checkpoint inside vm_init_with_host() printed successfully, including the very last line of the function — but the checkpoint immediately after its return never appeared. Re-running with a trivial, unrelated local-variable addition shifted the crash earlier and changed the exception type entirely (from a load fault to an instruction-fetch fault at sepc == stval, i.e. the CPU tried to execute an unmapped address). A crash whose location and even category shift under unrelated code changes is the standard signature of stack corruption, not a fixed bug at a specific call site.

Root Cause

kernel_main.c documented the gap directly, once read closely:

On amd64, kernel_entry.S switches the stack from UEFI's default to a 2 MiB zero-initialised BSS stack and tail-calls this function as kernel_main_impl. On aarch64 and riscv64 the assembly trampoline is not yet implemented and the UEFI loader calls kernel_main directly.

amd64 has carried a dedicated stack-switch trampoline (src/starkernel/arch/amd64/kernel_entry.S) since early on, with its own comment explaining why: "UEFI hands control to kernel_main() running on a small UEFI-provided stack (typically 128 KB–256 KB). The FORTH interpreter + DOE experiment loop can easily exceed that depth." riscv64 never had the equivalent. It ran the entire VM bootstrap — 27 chained word-registration modules, physics/SSM init, Tripod capsule birth — directly on whatever small stack EDK2's riscv64 firmware handed the loader, overflowing it and corrupting a return address. aarch64 has the same theoretical exposure but happens not to trip it, apparently because its firmware's default boot stack is larger — incidental, not a guarantee.

Fix

src/starkernel/arch/riscv64/kernel_entry.S (new) — RISC-V stack-switch trampoline mirroring amd64's:

    .section .bss
    .align 4
    .global g_kernel_stack
g_kernel_stack:
    .space 0x200000
g_kernel_stack_top:
    .global g_kernel_stack_top

    .section .text
    .extern kernel_main_impl
    .global kernel_main
kernel_main:
    ld   t0, BOOT_INFO_KERNEL_STACK_BASE_OFFSET(a0)
    bnez t0, .Ldynamic_stack
    la   t0, g_kernel_stack_top
    j    .Lstack_ready
.Ldynamic_stack:
    ld   t1, BOOT_INFO_KERNEL_STACK_SIZE_OFFSET(a0)
    add  t0, t0, t1
.Lstack_ready:
    andi t0, t0, -16
    mv   sp, t0
    mv   s0, zero
    tail kernel_main_impl

src/starkernel/kernel_main.c — riscv64 now builds kernel_main_impl (invoked via the trampoline) instead of kernel_main directly, same pattern as amd64:

#if defined(__x86_64__) || defined(__riscv)
void kernel_main_impl(BootInfo *boot_info) {
#else
void kernel_main(BootInfo *boot_info) {
#endif

Makefile.starkernel — wires the new file into the riscv64 loader build:

ifeq ($(ARCH),riscv64)
LOADER_ASM += $(KERNEL_SRC)/arch/$(ARCH)/kernel_entry.S
endif

Secondary Fix

RAW_LOG() in src/starkernel/boot/uefi_loader.c was a no-op on every arch except amd64, silencing all of the loader's own diagnostic messages on riscv64 (and aarch64). Added a real raw-UART writer for riscv64, targeting QEMU virt's 16550-compatible uart8250 at MMIO 0x10000000 (matches Domain0 Region03 in the OpenSBI boot banner) — the same byte-polling protocol as the existing amd64 port-I/O version, just MMIO instead of outb/ inb. Existing and future RAW_LOG() call sites in the loader now actually produce output on riscv64.

Result

Criterion	Before	After
sepc after word registration	Wild address outside kernel image (0xbe27c3c2-ish, drifting per build)	N/A — no longer occurs
Exception	Load or instruction-fetch page fault, non-deterministic between builds	None
[Hera] ok> (riscv64)	Never reached	✅ reached, Tripod fleet (Hera/Hermes/Artemis) runs to completion
PASS: E2E msg flow	Never logged	✅
PARITY:MAMA_INIT / Init: Mama birth OK	Never logged	✅
Loader RAW_LOG() output on riscv64	Silent no-op	✅ visible in serial log

Acceptance Testing

All three architectures must boot to ok>, run the Tripod fleet, and reach [Hera] ok>. Per repo convention, run in this exact order, one QEMU instance at a time, always clean before qemu:

make -f Makefile.starkernel ARCH=amd64   clean qemu
make -f Makefile.starkernel ARCH=aarch64 clean qemu
make -f Makefile.starkernel ARCH=riscv64 clean qemu

Verify each log:

grep "ok>"                 logs/<session>/<arch>/qemu-<arch>-*.log
grep "PASS: E2E msg flow"  logs/<session>/<arch>/qemu-<arch>-*.log
grep "PARITY:MAMA_INIT"    logs/<session>/<arch>/qemu-<arch>-*.log
grep "EXCEPTION"           logs/<session>/<arch>/qemu-<arch>-*.log   # must be empty

All three architectures confirmed clean on 2026-08-02 (see logs/20260802-064050/riscv64/, logs/20260802-064424/amd64/, logs/20260802-064656/aarch64/, committed alongside this fix).