Document the riscv64 kernel stack trampoline fix
Write-up of the 2026-08-02 riscv64 boot crash investigation and fix
(commit 7366275), in the same style as docs/lithosananke/amd64-isr-fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
757ce97dc1
commit
0fa9bd730e
@@ -0,0 +1,183 @@
|
||||
# Kernel Stack Trampoline Fix — riscv64
|
||||
|
||||
**Date**: 2026-08-02
|
||||
**Branch**: `master`
|
||||
**Commit**: `7366275`
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
The riscv64 kernel crashed partway through VM bootstrap during the 2026-08-01
|
||||
DoE campaign run, well after amd64 and aarch64 had both booted cleanly to
|
||||
`ok>` in the same session. The crash was a wild jump: execution landed at an
|
||||
address nowhere inside the kernel's own linked image, then almost immediately
|
||||
faulted trying to load from a small, near-null address.
|
||||
|
||||
```
|
||||
Registering FORTH-79 Standard word set...
|
||||
Registering FORTH-79 arithmetic words...
|
||||
FORTH-79 Standard word set registration complete
|
||||
*** EXCEPTION (riscv64) ***
|
||||
scause = 0x000000000000000d
|
||||
sepc = 0x00000000be27c3c2
|
||||
stval = 0x00000000000074d8
|
||||
```
|
||||
|
||||
`scause = 0xd` is a supervisor-mode load page fault. `sepc` (~0xbe27c3c2) sat
|
||||
near the top of QEMU's 1 GB RAM window — nowhere close to the kernel's own
|
||||
`.text`/`.data`/`.bss` footprint (`0x400000`–`0xcdf9a0`, confirmed via `nm`/
|
||||
`readelf` on the exact build that produced the log). `stval` (0x74d8, ~30 KB)
|
||||
stayed fixed across rebuilds even as `sepc` drifted — the signature of a
|
||||
corrupted return address rather than a fixed logic bug at one source line.
|
||||
|
||||
## Investigation
|
||||
|
||||
Two dead ends were ruled out before the real cause surfaced:
|
||||
|
||||
1. **ELF relocation / buffer overlap in the split-build loader.** The
|
||||
codebase has a `!MONOLITHIC_BUILD` path (`elf_load_kernel()` in
|
||||
`elf_loader.c`) that loads `kernel.elf` from the ESP into a UEFI
|
||||
`AllocateAnyPages` buffer. This looked like a plausible overlap candidate
|
||||
at first, but `Makefile.starkernel` defaults to `MONOLITHIC=1`, meaning
|
||||
`kernel_main` is linked directly into the loader and that code path never
|
||||
runs. Confirmed by checking the built binary for the investigation's own
|
||||
diagnostic strings — absent, because the compilation unit was built with
|
||||
`-DMONOLITHIC_BUILD` and the code was preprocessed out.
|
||||
2. **`RAW_LOG()` silently no-ops on every non-amd64 arch.** All of
|
||||
`load_kernel_from_esp()`'s narration (and an early ConOut-based
|
||||
diagnostic) never reached the captured serial log, on riscv64, in any
|
||||
build. `RAW_LOG` was defined only for `ARCH_AMD64`; everywhere else it
|
||||
expanded to `((void)0)`. This made straightforward diagnosis impossible
|
||||
until a real raw-UART writer was added for riscv64 (see Secondary Fix).
|
||||
|
||||
With real output available, `console_println()` checkpoints were bisected
|
||||
through `vm_init_with_host()` and `sk_vm_bootstrap_parity()`. Every checkpoint
|
||||
inside `vm_init_with_host()` printed successfully, including the very last
|
||||
line of the function — but the checkpoint immediately after its *return*
|
||||
never appeared. Re-running with a trivial, unrelated local-variable addition
|
||||
shifted the crash earlier and changed the exception type entirely (from a
|
||||
load fault to an instruction-fetch fault at `sepc == stval`, i.e. the CPU
|
||||
tried to *execute* an unmapped address). A crash whose location and even
|
||||
category shift under unrelated code changes is the standard signature of
|
||||
stack corruption, not a fixed bug at a specific call site.
|
||||
|
||||
## Root Cause
|
||||
|
||||
`kernel_main.c` documented the gap directly, once read closely:
|
||||
|
||||
> On amd64, `kernel_entry.S` switches the stack from UEFI's default to a
|
||||
> 2 MiB zero-initialised BSS stack and tail-calls this function as
|
||||
> `kernel_main_impl`. On aarch64 and riscv64 the assembly trampoline is not
|
||||
> yet implemented and the UEFI loader calls `kernel_main` directly.
|
||||
|
||||
amd64 has carried a dedicated stack-switch trampoline
|
||||
(`src/starkernel/arch/amd64/kernel_entry.S`) since early on, with its own
|
||||
comment explaining why: *"UEFI hands control to kernel_main() running on a
|
||||
small UEFI-provided stack (typically 128 KB–256 KB). The FORTH interpreter +
|
||||
DOE experiment loop can easily exceed that depth."* riscv64 never had the
|
||||
equivalent. It ran the entire VM bootstrap — 27 chained word-registration
|
||||
modules, physics/SSM init, Tripod capsule birth — directly on whatever small
|
||||
stack EDK2's riscv64 firmware handed the loader, overflowing it and
|
||||
corrupting a return address. aarch64 has the same theoretical exposure but
|
||||
happens not to trip it, apparently because its firmware's default boot stack
|
||||
is larger — incidental, not a guarantee.
|
||||
|
||||
## Fix
|
||||
|
||||
**`src/starkernel/arch/riscv64/kernel_entry.S`** (new) — RISC-V stack-switch
|
||||
trampoline mirroring amd64's:
|
||||
|
||||
```asm
|
||||
.section .bss
|
||||
.align 4
|
||||
.global g_kernel_stack
|
||||
g_kernel_stack:
|
||||
.space 0x200000
|
||||
g_kernel_stack_top:
|
||||
.global g_kernel_stack_top
|
||||
|
||||
.section .text
|
||||
.extern kernel_main_impl
|
||||
.global kernel_main
|
||||
kernel_main:
|
||||
ld t0, BOOT_INFO_KERNEL_STACK_BASE_OFFSET(a0)
|
||||
bnez t0, .Ldynamic_stack
|
||||
la t0, g_kernel_stack_top
|
||||
j .Lstack_ready
|
||||
.Ldynamic_stack:
|
||||
ld t1, BOOT_INFO_KERNEL_STACK_SIZE_OFFSET(a0)
|
||||
add t0, t0, t1
|
||||
.Lstack_ready:
|
||||
andi t0, t0, -16
|
||||
mv sp, t0
|
||||
mv s0, zero
|
||||
tail kernel_main_impl
|
||||
```
|
||||
|
||||
**`src/starkernel/kernel_main.c`** — riscv64 now builds `kernel_main_impl`
|
||||
(invoked via the trampoline) instead of `kernel_main` directly, same pattern
|
||||
as amd64:
|
||||
|
||||
```c
|
||||
#if defined(__x86_64__) || defined(__riscv)
|
||||
void kernel_main_impl(BootInfo *boot_info) {
|
||||
#else
|
||||
void kernel_main(BootInfo *boot_info) {
|
||||
#endif
|
||||
```
|
||||
|
||||
**`Makefile.starkernel`** — wires the new file into the riscv64 loader build:
|
||||
|
||||
```makefile
|
||||
ifeq ($(ARCH),riscv64)
|
||||
LOADER_ASM += $(KERNEL_SRC)/arch/$(ARCH)/kernel_entry.S
|
||||
endif
|
||||
```
|
||||
|
||||
## Secondary Fix
|
||||
|
||||
`RAW_LOG()` in `src/starkernel/boot/uefi_loader.c` was a no-op on every arch
|
||||
except amd64, silencing all of the loader's own diagnostic messages on
|
||||
riscv64 (and aarch64). Added a real raw-UART writer for riscv64, targeting
|
||||
QEMU virt's 16550-compatible `uart8250` at MMIO `0x10000000` (matches
|
||||
`Domain0 Region03` in the OpenSBI boot banner) — the same byte-polling
|
||||
protocol as the existing amd64 port-I/O version, just MMIO instead of `outb`/
|
||||
`inb`. Existing and future `RAW_LOG()` call sites in the loader now actually
|
||||
produce output on riscv64.
|
||||
|
||||
## Result
|
||||
|
||||
| Criterion | Before | After |
|
||||
|---|---|---|
|
||||
| `sepc` after word registration | Wild address outside kernel image (`0xbe27c3c2`-ish, drifting per build) | N/A — no longer occurs |
|
||||
| Exception | Load or instruction-fetch page fault, non-deterministic between builds | None |
|
||||
| `[Hera] ok>` (riscv64) | Never reached | ✅ reached, Tripod fleet (Hera/Hermes/Artemis) runs to completion |
|
||||
| `PASS: E2E msg flow` | Never logged | ✅ |
|
||||
| `PARITY:MAMA_INIT` / `Init: Mama birth OK` | Never logged | ✅ |
|
||||
| Loader `RAW_LOG()` output on riscv64 | Silent no-op | ✅ visible in serial log |
|
||||
|
||||
## Acceptance Testing
|
||||
|
||||
All three architectures must boot to `ok>`, run the Tripod fleet, and reach
|
||||
`[Hera] ok>`. Per repo convention, run in this exact order, one QEMU instance
|
||||
at a time, always `clean` before `qemu`:
|
||||
|
||||
```bash
|
||||
make -f Makefile.starkernel ARCH=amd64 clean qemu
|
||||
make -f Makefile.starkernel ARCH=aarch64 clean qemu
|
||||
make -f Makefile.starkernel ARCH=riscv64 clean qemu
|
||||
```
|
||||
|
||||
Verify each log:
|
||||
|
||||
```bash
|
||||
grep "ok>" logs/<session>/<arch>/qemu-<arch>-*.log
|
||||
grep "PASS: E2E msg flow" logs/<session>/<arch>/qemu-<arch>-*.log
|
||||
grep "PARITY:MAMA_INIT" logs/<session>/<arch>/qemu-<arch>-*.log
|
||||
grep "EXCEPTION" logs/<session>/<arch>/qemu-<arch>-*.log # must be empty
|
||||
```
|
||||
|
||||
All three architectures confirmed clean on 2026-08-02 (see
|
||||
`logs/20260802-064050/riscv64/`, `logs/20260802-064424/amd64/`,
|
||||
`logs/20260802-064656/aarch64/`, committed alongside this fix).
|
||||
Reference in New Issue
Block a user