diff --git a/docs/lithosananke/riscv64-stack-fix/README.md b/docs/lithosananke/riscv64-stack-fix/README.md new file mode 100644 index 0000000..fd7d4e8 --- /dev/null +++ b/docs/lithosananke/riscv64-stack-fix/README.md @@ -0,0 +1,183 @@ +# Kernel Stack Trampoline Fix — riscv64 + +**Date**: 2026-08-02 +**Branch**: `master` +**Commit**: `7366275` + +--- + +## Problem + +The riscv64 kernel crashed partway through VM bootstrap during the 2026-08-01 +DoE campaign run, well after amd64 and aarch64 had both booted cleanly to +`ok>` in the same session. The crash was a wild jump: execution landed at an +address nowhere inside the kernel's own linked image, then almost immediately +faulted trying to load from a small, near-null address. + +``` +Registering FORTH-79 Standard word set... +Registering FORTH-79 arithmetic words... +FORTH-79 Standard word set registration complete +*** EXCEPTION (riscv64) *** + scause = 0x000000000000000d + sepc = 0x00000000be27c3c2 + stval = 0x00000000000074d8 +``` + +`scause = 0xd` is a supervisor-mode load page fault. `sepc` (~0xbe27c3c2) sat +near the top of QEMU's 1 GB RAM window — nowhere close to the kernel's own +`.text`/`.data`/`.bss` footprint (`0x400000`–`0xcdf9a0`, confirmed via `nm`/ +`readelf` on the exact build that produced the log). `stval` (0x74d8, ~30 KB) +stayed fixed across rebuilds even as `sepc` drifted — the signature of a +corrupted return address rather than a fixed logic bug at one source line. + +## Investigation + +Two dead ends were ruled out before the real cause surfaced: + +1. **ELF relocation / buffer overlap in the split-build loader.** The + codebase has a `!MONOLITHIC_BUILD` path (`elf_load_kernel()` in + `elf_loader.c`) that loads `kernel.elf` from the ESP into a UEFI + `AllocateAnyPages` buffer. This looked like a plausible overlap candidate + at first, but `Makefile.starkernel` defaults to `MONOLITHIC=1`, meaning + `kernel_main` is linked directly into the loader and that code path never + runs. Confirmed by checking the built binary for the investigation's own + diagnostic strings — absent, because the compilation unit was built with + `-DMONOLITHIC_BUILD` and the code was preprocessed out. +2. **`RAW_LOG()` silently no-ops on every non-amd64 arch.** All of + `load_kernel_from_esp()`'s narration (and an early ConOut-based + diagnostic) never reached the captured serial log, on riscv64, in any + build. `RAW_LOG` was defined only for `ARCH_AMD64`; everywhere else it + expanded to `((void)0)`. This made straightforward diagnosis impossible + until a real raw-UART writer was added for riscv64 (see Secondary Fix). + +With real output available, `console_println()` checkpoints were bisected +through `vm_init_with_host()` and `sk_vm_bootstrap_parity()`. Every checkpoint +inside `vm_init_with_host()` printed successfully, including the very last +line of the function — but the checkpoint immediately after its *return* +never appeared. Re-running with a trivial, unrelated local-variable addition +shifted the crash earlier and changed the exception type entirely (from a +load fault to an instruction-fetch fault at `sepc == stval`, i.e. the CPU +tried to *execute* an unmapped address). A crash whose location and even +category shift under unrelated code changes is the standard signature of +stack corruption, not a fixed bug at a specific call site. + +## Root Cause + +`kernel_main.c` documented the gap directly, once read closely: + +> On amd64, `kernel_entry.S` switches the stack from UEFI's default to a +> 2 MiB zero-initialised BSS stack and tail-calls this function as +> `kernel_main_impl`. On aarch64 and riscv64 the assembly trampoline is not +> yet implemented and the UEFI loader calls `kernel_main` directly. + +amd64 has carried a dedicated stack-switch trampoline +(`src/starkernel/arch/amd64/kernel_entry.S`) since early on, with its own +comment explaining why: *"UEFI hands control to kernel_main() running on a +small UEFI-provided stack (typically 128 KB–256 KB). The FORTH interpreter + +DOE experiment loop can easily exceed that depth."* riscv64 never had the +equivalent. It ran the entire VM bootstrap — 27 chained word-registration +modules, physics/SSM init, Tripod capsule birth — directly on whatever small +stack EDK2's riscv64 firmware handed the loader, overflowing it and +corrupting a return address. aarch64 has the same theoretical exposure but +happens not to trip it, apparently because its firmware's default boot stack +is larger — incidental, not a guarantee. + +## Fix + +**`src/starkernel/arch/riscv64/kernel_entry.S`** (new) — RISC-V stack-switch +trampoline mirroring amd64's: + +```asm + .section .bss + .align 4 + .global g_kernel_stack +g_kernel_stack: + .space 0x200000 +g_kernel_stack_top: + .global g_kernel_stack_top + + .section .text + .extern kernel_main_impl + .global kernel_main +kernel_main: + ld t0, BOOT_INFO_KERNEL_STACK_BASE_OFFSET(a0) + bnez t0, .Ldynamic_stack + la t0, g_kernel_stack_top + j .Lstack_ready +.Ldynamic_stack: + ld t1, BOOT_INFO_KERNEL_STACK_SIZE_OFFSET(a0) + add t0, t0, t1 +.Lstack_ready: + andi t0, t0, -16 + mv sp, t0 + mv s0, zero + tail kernel_main_impl +``` + +**`src/starkernel/kernel_main.c`** — riscv64 now builds `kernel_main_impl` +(invoked via the trampoline) instead of `kernel_main` directly, same pattern +as amd64: + +```c +#if defined(__x86_64__) || defined(__riscv) +void kernel_main_impl(BootInfo *boot_info) { +#else +void kernel_main(BootInfo *boot_info) { +#endif +``` + +**`Makefile.starkernel`** — wires the new file into the riscv64 loader build: + +```makefile +ifeq ($(ARCH),riscv64) +LOADER_ASM += $(KERNEL_SRC)/arch/$(ARCH)/kernel_entry.S +endif +``` + +## Secondary Fix + +`RAW_LOG()` in `src/starkernel/boot/uefi_loader.c` was a no-op on every arch +except amd64, silencing all of the loader's own diagnostic messages on +riscv64 (and aarch64). Added a real raw-UART writer for riscv64, targeting +QEMU virt's 16550-compatible `uart8250` at MMIO `0x10000000` (matches +`Domain0 Region03` in the OpenSBI boot banner) — the same byte-polling +protocol as the existing amd64 port-I/O version, just MMIO instead of `outb`/ +`inb`. Existing and future `RAW_LOG()` call sites in the loader now actually +produce output on riscv64. + +## Result + +| Criterion | Before | After | +|---|---|---| +| `sepc` after word registration | Wild address outside kernel image (`0xbe27c3c2`-ish, drifting per build) | N/A — no longer occurs | +| Exception | Load or instruction-fetch page fault, non-deterministic between builds | None | +| `[Hera] ok>` (riscv64) | Never reached | ✅ reached, Tripod fleet (Hera/Hermes/Artemis) runs to completion | +| `PASS: E2E msg flow` | Never logged | ✅ | +| `PARITY:MAMA_INIT` / `Init: Mama birth OK` | Never logged | ✅ | +| Loader `RAW_LOG()` output on riscv64 | Silent no-op | ✅ visible in serial log | + +## Acceptance Testing + +All three architectures must boot to `ok>`, run the Tripod fleet, and reach +`[Hera] ok>`. Per repo convention, run in this exact order, one QEMU instance +at a time, always `clean` before `qemu`: + +```bash +make -f Makefile.starkernel ARCH=amd64 clean qemu +make -f Makefile.starkernel ARCH=aarch64 clean qemu +make -f Makefile.starkernel ARCH=riscv64 clean qemu +``` + +Verify each log: + +```bash +grep "ok>" logs///qemu--*.log +grep "PASS: E2E msg flow" logs///qemu--*.log +grep "PARITY:MAMA_INIT" logs///qemu--*.log +grep "EXCEPTION" logs///qemu--*.log # must be empty +``` + +All three architectures confirmed clean on 2026-08-02 (see +`logs/20260802-064050/riscv64/`, `logs/20260802-064424/amd64/`, +`logs/20260802-064656/aarch64/`, committed alongside this fix).