aarch64: fix BYE cold-restart crash — PSCI SYSTEM_RESET via HVC, not SMC
Root cause of the aarch64 BYE cold-restart exception (present since at least 2026-08-08, ESR_EL1=0x02000000/EC=0 "Unknown reason"), found via live gdb single-stepping through the actual crash: arch_cold_reset() issued PSCI SYSTEM_RESET via `smc #0`, but QEMU's aarch64 virt machine booted with AAVMF (UEFI firmware, no genuine EL3/TrustZone secure monitor) serves PSCI via HVC, not SMC -- nothing exists to answer an SMC call, so it trapped as an illegal instruction straight into the kernel's own exception handler. Not memory corruption, not a race -- a wrong conduit for this boot configuration. Fix: smc #0 -> hvc #0. Function ID and calling convention unchanged. Getting to this required first discovering that starkernel_kernel.elf is not the binary that actually runs -- MONOLITHIC_BUILD links kernel_main() directly into starkernel_loader.efi, a completely separate, differently-linked build artifact. Every earlier gdb breakpoint attempt this session failed because it used addresses from the wrong file. Real addresses (UEFI-chosen ImageBase + linker-map RVA) let gdb catch the crash live for the first time. Verified: full aarch64 acceptance pass, 30/30 stress-campaign reps PASS (unaffected -- this bug only manifested on BYE), and BYE now exits cleanly with no exception for the first time in this investigation. Full writeup in FABRIC-2.md Section I. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
bc31916461
commit
b24a5a6e25
+101
-23
@@ -1008,7 +1008,7 @@ something this item introduced.
|
||||
|
||||
---
|
||||
|
||||
## I. aarch64 `BYE` cold-restart exception — real PSCI bug fixed, but not the actual crash cause; crash is a pre-existing, unresolved bug class — 2026-08-18
|
||||
## I. aarch64 `BYE` cold-restart exception — RESOLVED 2026-08-18 (SMC→HVC PSCI conduit)
|
||||
|
||||
Surfaced during the item 4.6 acceptance pass (Section H): on aarch64 only, typing `BYE` at
|
||||
the `ok>` prompt reaps children and prints `BYE: cold restart` successfully, then faults
|
||||
@@ -1150,27 +1150,105 @@ unremarkable: `mama_word_bye` ends with `bl 40ef60 <arch_cold_reset>`, a plain d
|
||||
to the right address — ruling out a bad relocation or corrupted call instruction as the
|
||||
reason the breakpoints didn't fire.
|
||||
|
||||
**Sanity check, decisive:** set `hbreak console_println` — a function called thousands of
|
||||
times from the very first moment of kernel boot — on a fresh boot. After the serial log had
|
||||
already accumulated **8,802 lines** (deep into Artemis's stress campaign, `console_println`
|
||||
**Sanity check:** set `hbreak console_println` — a function called thousands of times from
|
||||
the very first moment of kernel boot — on a fresh boot. After the serial log had already
|
||||
accumulated **8,802 lines** (deep into Artemis's stress campaign, `console_println`
|
||||
necessarily called thousands of times to produce that output), gdb still reported only
|
||||
`Continuing.` — zero breakpoint hits, ever. **This means gdb breakpoints (software and
|
||||
hardware alike) do not function at all against this QEMU aarch64 (`cortex-a57`, TCG) target
|
||||
in this environment** — not an icache-coherency property of our kernel, a limitation of the
|
||||
debugging setup itself. The icache-corruption hypothesis this session was pursuing is
|
||||
therefore neither confirmed nor refuted; it's simply unreachable with this tooling as
|
||||
configured. All gdb sessions and their QEMU instances were killed; no code changes came out
|
||||
of this thread beyond the (kept) heap-address prints and the `-m 4096` bump, both harmless
|
||||
diagnostics worth keeping regardless.
|
||||
`Continuing.` — zero breakpoint hits, ever, against our kernel.
|
||||
|
||||
**Where this stands:** root cause narrowed but not found. Confirmed heap-internal (not
|
||||
kernel code, not firmware, not truly random/unmapped memory — see the decisive finding
|
||||
above). Live single-stepping is not currently viable against this target; before attempting
|
||||
it again, the QEMU aarch64 gdbstub setup itself needs to be validated independently (try a
|
||||
different QEMU version, `-accel tcg,thread=single`, or confirm hardware breakpoint support
|
||||
against a trivial known-working aarch64 QEMU target first, outside this kernel entirely) —
|
||||
don't repeat this exact approach expecting a different result. Two competing crash
|
||||
signatures still unreconciled: `ESR_EL1=0x02000000`/EC=0 "Unknown reason" (this crash, and
|
||||
the 2026-08-08 keyboard-input crash) vs. `ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with
|
||||
a poisoned-looking `FAR_EL1=0x000055bd` (June/July 2026 crashes) — may or may not be the same
|
||||
bug. Genuinely open.
|
||||
**Refined by a follow-up control test — this is NOT a blanket "gdbstub is broken" finding.**
|
||||
A trivial standalone aarch64 program (hand-assembled: `mov`/`add`/`subs`/`b.ne` loop, no
|
||||
UEFI, no MMU, linked at `0x40080000`, booted directly via QEMU's own `-kernel` flag,
|
||||
bypassing our loader entirely) hit a hardware breakpoint (`hbreak loop`) on the very first
|
||||
try — `Breakpoint 1, 0x...4008000c in loop ()`, exactly as expected. **The gdbstub mechanism
|
||||
itself works fine in this QEMU build/environment.** The failure is specific to *our* kernel.
|
||||
|
||||
**Narrowed further, decisively: it fails from our kernel's very first instruction, not
|
||||
something that develops later in boot.** Set `hbreak kernel_main` (the ELF entry point
|
||||
itself, `0x400da0`, confirmed by `readelf -h`) — after the serial log had already printed PMM
|
||||
statistics (`Used pages: 1411`, `Total MB: 4062` — output that can only come from well inside
|
||||
`kernel_main`/`kernel_main_deep`, several stages past entry), gdb still reported only
|
||||
`Continuing.`, never a hit. **Breakpoints fail on our kernel from the first instruction
|
||||
onward, not partway through boot** — ruling out anything `kernel_main_deep` or later does
|
||||
(MMU reconfiguration, EL switches deep in boot, etc.) as the cause, since the very first
|
||||
instruction is already unreachable to gdb.
|
||||
|
||||
**Where this leaves it:** the standalone test (works) differs from our kernel (doesn't work)
|
||||
in exactly one respect that matters here — the standalone test boots directly via QEMU's
|
||||
`-kernel` flag (no firmware involved), while ours boots through the full UEFI path (AAVMF
|
||||
firmware → `starkernel_loader.efi` → `ExitBootServices()` → jump to `kernel_main`). The likely
|
||||
culprit is something about that handoff — UEFI's own MMU/paging state, exception-level
|
||||
context, or how our loader transitions into the kernel — that QEMU's gdbstub can't track
|
||||
across, not anything in our kernel's own C code. Not yet tested directly (would need a
|
||||
minimal UEFI-booted test kernel to isolate whether the loader phase or the UEFI-to-kernel
|
||||
jump specifically is where it breaks) — a good next concrete step, more promising than
|
||||
retrying breakpoints against the real kernel again. All gdb sessions and QEMU instances from
|
||||
this thread were killed; no code changes came out of it beyond the (kept) heap-address prints
|
||||
and the `-m 4096` bump, both harmless diagnostics worth keeping regardless.
|
||||
|
||||
**The actual blocker, found and fixed: `starkernel_kernel.elf` is the wrong symbol file.**
|
||||
`MONOLITHIC_BUILD` (`Makefile.starkernel`) compiles `kernel_main()` and everything it calls
|
||||
directly into `starkernel_loader.efi` — the PE32+ binary UEFI actually loads and executes.
|
||||
`starkernel_kernel.elf` is a *completely separate build artifact*: different object list
|
||||
(`KERNEL_OBJS` vs. `LOADER_OBJS`), different linker (`$(LD)` vs. `lld-link`), different link
|
||||
step entirely, sharing nothing but overlapping source files. Every gdb breakpoint address in
|
||||
this investigation — `nm`/`objdump` addresses like `arch_cold_reset`'s `0x40ef60` — came from
|
||||
that wrong file and had no relationship to where the code actually lives at runtime. That's
|
||||
why breakpoints never fired even at `kernel_main`'s own first instruction: the address was
|
||||
simply wrong, on every attempt.
|
||||
|
||||
**How the real address was found.** `starkernel_loader.efi` links with `/base:0` but is a
|
||||
PE image UEFI relocates to a runtime-chosen `ImageBase`, so real address = `ImageBase + RVA`.
|
||||
Got the RVA for any symbol by re-running the exact `lld-link` command `make -n` prints, with
|
||||
`/map:out.map` appended (a linker map, since the final stripped `.efi` carries no symbol
|
||||
table `llvm-nm`/`llvm-objdump-18` can read — confirmed via `file format coff-arm64` showing
|
||||
"no symbols"). Got the runtime `ImageBase` with a one-line temporary probe in `efi_main()`
|
||||
(`src/starkernel/boot/uefi_loader.c`, right after the first `debug_checkpoint`): fetch
|
||||
`EFI_LOADED_IMAGE_PROTOCOL` for the running image, print `->ImageBase` via `ConOut`.
|
||||
Confirmed deterministic across boots of the same build (`0x13e019000`, identical on repeat).
|
||||
**Reverted after capture** (kept out of the tree — this was a one-shot diagnostic, not a
|
||||
permanent addition, per this project's usual probe-then-revert practice).
|
||||
|
||||
**With the correct address, gdb worked immediately.** `hbreak *<ImageBase+RVA>` for
|
||||
`kernel_main` fired on the very first try — confirming the gdbstub/tooling was never broken,
|
||||
only every address fed to it in this investigation. Re-armed the same way at
|
||||
`arch_cold_reset`'s real address and ran the full ~30-minute boot to `BYE` with gdb attached;
|
||||
the breakpoint fired exactly where expected, right after "BYE: cold restart" printed.
|
||||
|
||||
**Single-stepped from there — the real mechanism.** `x/8i $pc` confirmed the disassembly
|
||||
matched source exactly: `bl` to `arch_disable_interrupts()`, `mov`/`movk` building the PSCI
|
||||
function ID, `smc #0x0`, then the wfi loop. Stepping instruction-by-instruction through to
|
||||
and past the `smc` showed **PC does not fall through to the wfi loop** — it jumps to another
|
||||
address entirely, which itself branches into what is unmistakably a normal function prologue
|
||||
(`sub sp, sp, #0x40`). This is our own kernel's exception vector correctly catching an
|
||||
illegal instruction and dispatching to the handler that prints `*** EXCEPTION (aarch64) ***`
|
||||
— exactly the behavior observed in every prior serial log. **The SMC instruction genuinely
|
||||
traps; nothing is corrupted, nothing jumps into garbage heap memory.** (The earlier
|
||||
"fault address is inside the kmalloc heap" finding, Section I above this one, was real
|
||||
arithmetic on real numbers, but those numbers came from the same wrong symbol file — a
|
||||
coincidental match, not evidence about the actual mechanism. Left in place above as an
|
||||
honest record of the investigation's path, not retracted.)
|
||||
|
||||
**Root cause: wrong PSCI conduit.** QEMU's aarch64 `virt` machine booted with AAVMF (UEFI
|
||||
firmware, no genuine EL3/TrustZone secure monitor) serves PSCI via **HVC** (hypervisor call,
|
||||
EL2), not **SMC** (secure monitor call, EL3) — there is no real secure-world firmware present
|
||||
to answer an SMC. `arch_cold_reset()` used `smc #0`, which traps as an illegal instruction
|
||||
with nothing to service it — a textbook "wrong conduit for this boot configuration" bug, not
|
||||
memory corruption, not a race, not an icache issue.
|
||||
|
||||
**Fix:** `src/starkernel/arch/aarch64/arch.c`, `smc #0` → `hvc #0`. Function ID and calling
|
||||
convention (the `mov`/`movk` building `0x84000009`) unchanged — only the conduit instruction.
|
||||
|
||||
**Verification:** rebuilt aarch64 clean, ran the full acceptance pass (30/30 stress-campaign
|
||||
reps PASS, unaffected — this bug only manifests on `BYE`) through to `BYE`. Result: `BYE:
|
||||
reaping children` → `BYE: cold restart` → clean process exit, code 0. **No exception.** First
|
||||
clean aarch64 `BYE` exit recorded in this document.
|
||||
|
||||
**Loose end, deliberately not chased further this session:** two possibly-different crash
|
||||
signatures existed historically — this bug's `ESR_EL1=0x02000000`/EC=0 "Unknown reason"
|
||||
signature (also seen in an unrelated 2026-08-08 keyboard-input log), versus an older
|
||||
`ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with a poisoned-looking `FAR_EL1=0x000055bd`
|
||||
from June/July 2026 crashes. This fix explains and resolves the former. Whether the latter
|
||||
was the same underlying SMC/HVC conduit issue (plausible — an illegal-instruction trap can
|
||||
manifest with different ESR encodings depending on exact CPU/QEMU state) or a genuinely
|
||||
separate bug was not re-investigated; nothing currently reproduces it to check against.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# Capsule Block Manifest — Auto-generated
|
||||
<!-- Generated by mkcapsule --manifest 2026-08-18T19:02:24Z -->
|
||||
<!-- Generated by mkcapsule --manifest 2026-08-18T21:52:26Z -->
|
||||
<!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh. -->
|
||||
<!-- Hand-written justifications and immutability notes live -->
|
||||
<!-- in MANIFEST.md alongside this auto-generated index. -->
|
||||
|
||||
Binary file not shown.
+78620
-78511
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -127,13 +127,21 @@ void arch_cold_reset(void)
|
||||
arch_disable_interrupts();
|
||||
/* PSCI SYSTEM_RESET (function 0x84000009). SYSTEM_RESET takes no
|
||||
* arguments and has no SMC64 variant defined by the PSCI spec -- only
|
||||
* the SMC32 encoding is valid. The prior 0xC4000009 (SMC64 convention)
|
||||
* was not a real PSCI function ID, causing an unhandled exception
|
||||
* (EC=0, "Unknown reason") on the SMC call instead of a reset. */
|
||||
* the SMC32 encoding is valid (fixed 2026-08-18, was 0xC4000009).
|
||||
*
|
||||
* FABRIC-2.md Section I, 2026-08-18: live gdb tracing (using the real
|
||||
* UEFI-relocated runtime address, not the standalone kernel.elf's
|
||||
* link-time address -- see that section for why those differ) proved
|
||||
* the SMC call itself traps: PC does not fall through to the wfi loop
|
||||
* below, it jumps straight into this kernel's own exception vector.
|
||||
* This QEMU aarch64 boot (AAVMF UEFI firmware, no genuine EL3/TrustZone
|
||||
* secure monitor) has nothing to answer an SMC -- PSCI here is served
|
||||
* via HVC (hypervisor call, EL2) instead. Switched conduits; the
|
||||
* function ID and calling convention are unchanged. */
|
||||
__asm__ volatile (
|
||||
"mov x0, #0x84000000\n"
|
||||
"movk x0, #0x0009\n"
|
||||
"smc #0\n"
|
||||
"hvc #0\n"
|
||||
::: "x0", "memory"
|
||||
);
|
||||
for (;;) __asm__ volatile ("wfi" ::: "memory");
|
||||
|
||||
Reference in New Issue
Block a user