aarch64: fix BYE cold-restart crash — PSCI SYSTEM_RESET via HVC, not SMC

Root cause of the aarch64 BYE cold-restart exception (present since at
least 2026-08-08, ESR_EL1=0x02000000/EC=0 "Unknown reason"), found via
live gdb single-stepping through the actual crash: arch_cold_reset()
issued PSCI SYSTEM_RESET via `smc #0`, but QEMU's aarch64 virt machine
booted with AAVMF (UEFI firmware, no genuine EL3/TrustZone secure
monitor) serves PSCI via HVC, not SMC -- nothing exists to answer an
SMC call, so it trapped as an illegal instruction straight into the
kernel's own exception handler. Not memory corruption, not a race --
a wrong conduit for this boot configuration.

Fix: smc #0 -> hvc #0. Function ID and calling convention unchanged.

Getting to this required first discovering that starkernel_kernel.elf
is not the binary that actually runs -- MONOLITHIC_BUILD links
kernel_main() directly into starkernel_loader.efi, a completely
separate, differently-linked build artifact. Every earlier gdb
breakpoint attempt this session failed because it used addresses from
the wrong file. Real addresses (UEFI-chosen ImageBase + linker-map
RVA) let gdb catch the crash live for the first time.

Verified: full aarch64 acceptance pass, 30/30 stress-campaign reps
PASS (unaffected -- this bug only manifested on BYE), and BYE now
exits cleanly with no exception for the first time in this
investigation.

Full writeup in FABRIC-2.md Section I.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
2026-08-18 18:24:26 -04:00
co-authored by Claude Sonnet 5
parent bc31916461
commit b24a5a6e25
12 changed files with 368452 additions and 78539 deletions
+101 -23
View File
@@ -1008,7 +1008,7 @@ something this item introduced.
---
## I. aarch64 `BYE` cold-restart exception — real PSCI bug fixed, but not the actual crash cause; crash is a pre-existing, unresolved bug class — 2026-08-18
## I. aarch64 `BYE` cold-restart exception — RESOLVED 2026-08-18 (SMC→HVC PSCI conduit)
Surfaced during the item 4.6 acceptance pass (Section H): on aarch64 only, typing `BYE` at
the `ok>` prompt reaps children and prints `BYE: cold restart` successfully, then faults
@@ -1150,27 +1150,105 @@ unremarkable: `mama_word_bye` ends with `bl 40ef60 <arch_cold_reset>`, a plain d
to the right address — ruling out a bad relocation or corrupted call instruction as the
reason the breakpoints didn't fire.
**Sanity check, decisive:** set `hbreak console_println` — a function called thousands of
times from the very first moment of kernel boot — on a fresh boot. After the serial log had
already accumulated **8,802 lines** (deep into Artemis's stress campaign, `console_println`
**Sanity check:** set `hbreak console_println` — a function called thousands of times from
the very first moment of kernel boot — on a fresh boot. After the serial log had already
accumulated **8,802 lines** (deep into Artemis's stress campaign, `console_println`
necessarily called thousands of times to produce that output), gdb still reported only
`Continuing.` — zero breakpoint hits, ever. **This means gdb breakpoints (software and
hardware alike) do not function at all against this QEMU aarch64 (`cortex-a57`, TCG) target
in this environment** — not an icache-coherency property of our kernel, a limitation of the
debugging setup itself. The icache-corruption hypothesis this session was pursuing is
therefore neither confirmed nor refuted; it's simply unreachable with this tooling as
configured. All gdb sessions and their QEMU instances were killed; no code changes came out
of this thread beyond the (kept) heap-address prints and the `-m 4096` bump, both harmless
diagnostics worth keeping regardless.
`Continuing.` — zero breakpoint hits, ever, against our kernel.
**Where this stands:** root cause narrowed but not found. Confirmed heap-internal (not
kernel code, not firmware, not truly random/unmapped memory — see the decisive finding
above). Live single-stepping is not currently viable against this target; before attempting
it again, the QEMU aarch64 gdbstub setup itself needs to be validated independently (try a
different QEMU version, `-accel tcg,thread=single`, or confirm hardware breakpoint support
against a trivial known-working aarch64 QEMU target first, outside this kernel entirely) —
don't repeat this exact approach expecting a different result. Two competing crash
signatures still unreconciled: `ESR_EL1=0x02000000`/EC=0 "Unknown reason" (this crash, and
the 2026-08-08 keyboard-input crash) vs. `ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with
a poisoned-looking `FAR_EL1=0x000055bd` (June/July 2026 crashes) — may or may not be the same
bug. Genuinely open.
**Refined by a follow-up control test — this is NOT a blanket "gdbstub is broken" finding.**
A trivial standalone aarch64 program (hand-assembled: `mov`/`add`/`subs`/`b.ne` loop, no
UEFI, no MMU, linked at `0x40080000`, booted directly via QEMU's own `-kernel` flag,
bypassing our loader entirely) hit a hardware breakpoint (`hbreak loop`) on the very first
try — `Breakpoint 1, 0x...4008000c in loop ()`, exactly as expected. **The gdbstub mechanism
itself works fine in this QEMU build/environment.** The failure is specific to *our* kernel.
**Narrowed further, decisively: it fails from our kernel's very first instruction, not
something that develops later in boot.** Set `hbreak kernel_main` (the ELF entry point
itself, `0x400da0`, confirmed by `readelf -h`) — after the serial log had already printed PMM
statistics (`Used pages: 1411`, `Total MB: 4062` — output that can only come from well inside
`kernel_main`/`kernel_main_deep`, several stages past entry), gdb still reported only
`Continuing.`, never a hit. **Breakpoints fail on our kernel from the first instruction
onward, not partway through boot** — ruling out anything `kernel_main_deep` or later does
(MMU reconfiguration, EL switches deep in boot, etc.) as the cause, since the very first
instruction is already unreachable to gdb.
**Where this leaves it:** the standalone test (works) differs from our kernel (doesn't work)
in exactly one respect that matters here — the standalone test boots directly via QEMU's
`-kernel` flag (no firmware involved), while ours boots through the full UEFI path (AAVMF
firmware → `starkernel_loader.efi``ExitBootServices()` → jump to `kernel_main`). The likely
culprit is something about that handoff — UEFI's own MMU/paging state, exception-level
context, or how our loader transitions into the kernel — that QEMU's gdbstub can't track
across, not anything in our kernel's own C code. Not yet tested directly (would need a
minimal UEFI-booted test kernel to isolate whether the loader phase or the UEFI-to-kernel
jump specifically is where it breaks) — a good next concrete step, more promising than
retrying breakpoints against the real kernel again. All gdb sessions and QEMU instances from
this thread were killed; no code changes came out of it beyond the (kept) heap-address prints
and the `-m 4096` bump, both harmless diagnostics worth keeping regardless.
**The actual blocker, found and fixed: `starkernel_kernel.elf` is the wrong symbol file.**
`MONOLITHIC_BUILD` (`Makefile.starkernel`) compiles `kernel_main()` and everything it calls
directly into `starkernel_loader.efi` — the PE32+ binary UEFI actually loads and executes.
`starkernel_kernel.elf` is a *completely separate build artifact*: different object list
(`KERNEL_OBJS` vs. `LOADER_OBJS`), different linker (`$(LD)` vs. `lld-link`), different link
step entirely, sharing nothing but overlapping source files. Every gdb breakpoint address in
this investigation — `nm`/`objdump` addresses like `arch_cold_reset`'s `0x40ef60` — came from
that wrong file and had no relationship to where the code actually lives at runtime. That's
why breakpoints never fired even at `kernel_main`'s own first instruction: the address was
simply wrong, on every attempt.
**How the real address was found.** `starkernel_loader.efi` links with `/base:0` but is a
PE image UEFI relocates to a runtime-chosen `ImageBase`, so real address = `ImageBase + RVA`.
Got the RVA for any symbol by re-running the exact `lld-link` command `make -n` prints, with
`/map:out.map` appended (a linker map, since the final stripped `.efi` carries no symbol
table `llvm-nm`/`llvm-objdump-18` can read — confirmed via `file format coff-arm64` showing
"no symbols"). Got the runtime `ImageBase` with a one-line temporary probe in `efi_main()`
(`src/starkernel/boot/uefi_loader.c`, right after the first `debug_checkpoint`): fetch
`EFI_LOADED_IMAGE_PROTOCOL` for the running image, print `->ImageBase` via `ConOut`.
Confirmed deterministic across boots of the same build (`0x13e019000`, identical on repeat).
**Reverted after capture** (kept out of the tree — this was a one-shot diagnostic, not a
permanent addition, per this project's usual probe-then-revert practice).
**With the correct address, gdb worked immediately.** `hbreak *<ImageBase+RVA>` for
`kernel_main` fired on the very first try — confirming the gdbstub/tooling was never broken,
only every address fed to it in this investigation. Re-armed the same way at
`arch_cold_reset`'s real address and ran the full ~30-minute boot to `BYE` with gdb attached;
the breakpoint fired exactly where expected, right after "BYE: cold restart" printed.
**Single-stepped from there — the real mechanism.** `x/8i $pc` confirmed the disassembly
matched source exactly: `bl` to `arch_disable_interrupts()`, `mov`/`movk` building the PSCI
function ID, `smc #0x0`, then the wfi loop. Stepping instruction-by-instruction through to
and past the `smc` showed **PC does not fall through to the wfi loop** — it jumps to another
address entirely, which itself branches into what is unmistakably a normal function prologue
(`sub sp, sp, #0x40`). This is our own kernel's exception vector correctly catching an
illegal instruction and dispatching to the handler that prints `*** EXCEPTION (aarch64) ***`
— exactly the behavior observed in every prior serial log. **The SMC instruction genuinely
traps; nothing is corrupted, nothing jumps into garbage heap memory.** (The earlier
"fault address is inside the kmalloc heap" finding, Section I above this one, was real
arithmetic on real numbers, but those numbers came from the same wrong symbol file — a
coincidental match, not evidence about the actual mechanism. Left in place above as an
honest record of the investigation's path, not retracted.)
**Root cause: wrong PSCI conduit.** QEMU's aarch64 `virt` machine booted with AAVMF (UEFI
firmware, no genuine EL3/TrustZone secure monitor) serves PSCI via **HVC** (hypervisor call,
EL2), not **SMC** (secure monitor call, EL3) — there is no real secure-world firmware present
to answer an SMC. `arch_cold_reset()` used `smc #0`, which traps as an illegal instruction
with nothing to service it — a textbook "wrong conduit for this boot configuration" bug, not
memory corruption, not a race, not an icache issue.
**Fix:** `src/starkernel/arch/aarch64/arch.c`, `smc #0``hvc #0`. Function ID and calling
convention (the `mov`/`movk` building `0x84000009`) unchanged — only the conduit instruction.
**Verification:** rebuilt aarch64 clean, ran the full acceptance pass (30/30 stress-campaign
reps PASS, unaffected — this bug only manifests on `BYE`) through to `BYE`. Result: `BYE:
reaping children``BYE: cold restart` → clean process exit, code 0. **No exception.** First
clean aarch64 `BYE` exit recorded in this document.
**Loose end, deliberately not chased further this session:** two possibly-different crash
signatures existed historically — this bug's `ESR_EL1=0x02000000`/EC=0 "Unknown reason"
signature (also seen in an unrelated 2026-08-08 keyboard-input log), versus an older
`ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with a poisoned-looking `FAR_EL1=0x000055bd`
from June/July 2026 crashes. This fix explains and resolves the former. Whether the latter
was the same underlying SMC/HVC conduit issue (plausible — an illegal-instruction trap can
manifest with different ESR encodings depending on exact CPU/QEMU state) or a genuinely
separate bug was not re-investigated; nothing currently reproduces it to check against.
+1 -1
View File
@@ -1,5 +1,5 @@
# Capsule Block Manifest — Auto-generated
<!-- Generated by mkcapsule --manifest 2026-08-18T19:02:24Z -->
<!-- Generated by mkcapsule --manifest 2026-08-18T21:52:26Z -->
<!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh. -->
<!-- Hand-written justifications and immutability notes live -->
<!-- in MANIFEST.md alongside this auto-generated index. -->
BIN
View File
Binary file not shown.
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+12 -4
View File
@@ -127,13 +127,21 @@ void arch_cold_reset(void)
arch_disable_interrupts();
/* PSCI SYSTEM_RESET (function 0x84000009). SYSTEM_RESET takes no
* arguments and has no SMC64 variant defined by the PSCI spec -- only
* the SMC32 encoding is valid. The prior 0xC4000009 (SMC64 convention)
* was not a real PSCI function ID, causing an unhandled exception
* (EC=0, "Unknown reason") on the SMC call instead of a reset. */
* the SMC32 encoding is valid (fixed 2026-08-18, was 0xC4000009).
*
* FABRIC-2.md Section I, 2026-08-18: live gdb tracing (using the real
* UEFI-relocated runtime address, not the standalone kernel.elf's
* link-time address -- see that section for why those differ) proved
* the SMC call itself traps: PC does not fall through to the wfi loop
* below, it jumps straight into this kernel's own exception vector.
* This QEMU aarch64 boot (AAVMF UEFI firmware, no genuine EL3/TrustZone
* secure monitor) has nothing to answer an SMC -- PSCI here is served
* via HVC (hypervisor call, EL2) instead. Switched conduits; the
* function ID and calling convention are unchanged. */
__asm__ volatile (
"mov x0, #0x84000000\n"
"movk x0, #0x0009\n"
"smc #0\n"
"hvc #0\n"
::: "x0", "memory"
);
for (;;) __asm__ volatile ("wfi" ::: "memory");