Stage 3 follow-on: fix stack-ownership corruption + DoE switch columns (FABRIC-3.md §XXVIII.1)
Build / build-amd64-iso (push) Waiting to run
Build / build-aarch64-iso (push) Waiting to run
Build / build-riscv64-img (push) Waiting to run

DoE CSV gained 6 switch-signal columns (switch_count_cumulative,
switch_current_slot, switch_*_readiness, switch_ticks_since), and verifying
them with a boot-time HB-ON probe surfaced a real livelock: the preemption
checkpoint could fire inside a VM-EXEC-nested execute_colon_word() call and
switch away from a stack it didn't own, parking a borrowed region of the
caller's stack under the wrong VM's saved-context pointer. The trampoline
bounce was the visible (safe) half of this; the corruption was the quiet
half, live in every prior "clean" Stage 3 boot without ever showing up in
the log.

Fixed by gating the checkpoint on being at the outermost vm_interpret()
call (g_vm_interpret_depth / sk_vm_at_outermost_interpret(), vm_core.c),
per Bob's decision. Also fixed two related bugs found in the same pass:
g_switch_back_to was a single global stale after first entry, now per-VM
state (native_switch_back_to); note_switch_performed() fired on resume
instead of switch-out, now called before the switch.

Verified on all 3 architectures: steady log growth (no freeze), zero
leaked QEMU processes, DoE columns internally consistent, Hermes/Artemis
confirmed genuinely executing (not just trampoline-bouncing). Temporary
HB-ON boot probe reverted after capture.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UNhH1mhi52i6Qihh7ZV5S
This commit is contained in:
Robert Allan James
2026-09-13 23:58:59 -04:00
co-authored by Claude Sonnet 5
parent 986d042aa7
commit 862d7d9c48
19 changed files with 79695 additions and 19 deletions
+106
View File
@@ -3698,3 +3698,109 @@ switching and zero fault indicators, interactive console commands computed corre
(WIREBIND-scope extension) remains a ratified-decision-only step, not attempted -- the async
unclean-detach UAF risk it names is unchanged by anything built in Stages 0-3.
## XXVIII.1 -- Stage 3 follow-on: DoE CSV columns exposed real stack-ownership corruption in the
switch checkpoint, root-caused and fixed (2026-09-13)
**Correction to §XXVIII's own "Stage 3 CLOSED" claim above:** it overclaimed. "Zero fault
indicators" was true of what was *visible* -- Stage 3 emits nothing per switch by default, so a
switch storm and a healthy idle REPL produce an identical serial log. The corruption below was
already live in every one of those "clean" Stage 3 boots; nothing in that verification pass could
have shown it.
**Motivation:** wanted `1 1 + .` -> `2` -- a real trace of switching activity for
correlation/tuning. Added 6 columns to the DoE CSV (`doe_log.c`): `switch_count_cumulative`,
`switch_current_slot`, `switch_hera_readiness`, `switch_hermes_readiness`,
`switch_artemis_readiness`, `switch_ticks_since` -- all read-only exposure of state
`capsule_vm_switch_signal.c` already tracked internally for the switch decision itself. To
actually see rows emit without interactive input (foreground-only QEMU, no way to type `HB-ON`),
added a temporary boot-time `vm_interpret(mama, "HB-ON")` probe in `kernel_main.c`.
**The probe surfaced a real livelock, not a probe artifact.** Booted amd64: DoE rows emitted
correctly at first, then the serial log froze solid (`wc -l` identical across repeated checks,
15+ seconds) while `ps aux` showed the QEMU process at 102% CPU -- actively spinning, not blocked
on I/O. Bob's own real-time observation prompted the check: "is the serial moving? framebuffer
frozen static view. i wont say stuck."
**Root cause, found by reading `switch.c` and `vm_core.c` together, not by further boot attempts:**
`sk_vm_context_switch(from, to)` unconditionally saves whatever the *current* stack pointer
happens to be into `from->native_stack_saved_sp` -- it has no notion of whether `from` actually
owns the physical stack it is currently standing on. The Stage 3 checkpoint lives inside
`execute_colon_word()`, whose `vm` argument is *not always* the VM that owns the current native
stack: `VM-EXEC`/`VM-CALL` (`mama_word_vm_exec`/`mama_word_vm_call`, `mama_forth_words.c`)
dispatch into a *different* VM's dictionary via a nested `vm_interpret()` call, still physically
running on the caller's own stack (Hera's, in the common case -- her own `MSG-TICK` idle-pump
calling `VM-EXEC` into Hermes or Artemis). When the checkpoint fires inside that nested call, `vm`
is the nested VM (e.g. Hermes), but the stack is Hera's. A real switch there parks a *borrowed*
region of Hera's own stack under Hermes's saved-context pointer -- a stack-ownership violation.
If Hera is later independently resumed at her own (enclosing, higher-address) saved SP, she
resumes the same call chain and pushes new frames straight over the region "Hermes" believes she
owns: silent corruption. Confirmed reachable, not theoretical -- `execute_colon_word` is the code
pointer for every colon word (`vm_create_word(..., execute_colon_word)`), reached both by Hera's
own top-level REPL dispatch (outermost, genuinely owns the stack) and, nested inside that exact
call chain, by her own `MSG-TICK`'s `VM-EXEC` (does not own the stack) -- both firing sites are
real code paths, not a hypothetical interleaving.
**This subsumes the livelock, doesn't sit beside it as a separate bug.** The livelock is the
*safe* half of this defect: first-ever entry into Hermes/Artemis (no prior parked context) lands
in `sk_vm_switch_entry()`'s placeholder trampoline (deliberately a no-op yield-back, per Stage
2's own documented, not-yet-superseded contract), which immediately bounces back -- readiness
re-accumulates, the checkpoint re-fires, bounce again, a two-body ping-pong at full CPU with zero
FORTH progress. The *unsafe* half -- switching away from a nested nonzero-depth frame -- produces
quiet memory corruption instead of a loud freeze, and was not distinguished from the livelock
until traced through the code.
**Fix: gate the checkpoint on being at the outermost `vm_interpret()` call, per Bob's explicit
decision ("only allow the checkpoint at the outermost frame").** Added `g_vm_interpret_depth`, a
file-scope counter in `vm_core.c` incremented/decremented around `vm_interpret()`'s body, in the
same save/restore position as the existing `g_log_attrib_vm` (which already brackets nested
`vm_interpret()` calls for a different reason -- log-source attribution). Exposed via
`sk_vm_at_outermost_interpret()` (`depth <= 1`). The Stage 3 checkpoint in `execute_colon_word()`
now only calls `sk_vm_switch_signal_take_pending()` when this holds; when nested, the pending
switch is left untouched (not consumed) so the next checkpoint at the outermost frame -- whichever
VM that turns out to be -- picks it up instead of losing it. Framing worth keeping: Stage 2's
switch primitive is correct under its actual documented contract (manual, cooperative, top-level
use only); Stage 3 called it from a nested dispatch point without checking that contract -- a
Stage 3 integration defect, not a flaw in Stage 2 itself.
**Two more bugs found and fixed in the same pass, same root cause (no per-context ownership
tracking), both explicitly authorized before touching code:**
- `g_switch_back_to` was a single global captured once at a VM's *first-ever* trampoline entry,
then read from a local baked into that trampoline's own stack frame forever after. A later
switch-in from a *different* VM than the original caller would still bounce back to the
original caller, silently stranding whoever actually switched in. Fixed by moving it to
per-VM state -- `native_switch_back_to` (new `VM` field, alongside the other Stage 1/2
native-stack fields it belongs with), written unconditionally by `sk_vm_context_switch()` on
every switch-in (first-ever or resumed), read fresh every trampoline loop iteration instead of
once.
- `sk_vm_switch_signal_note_switch_performed()` was called *after* `sk_vm_context_switch()`
returned -- which, for a real switch, is on *resume*, an arbitrary number of ticks later, not
on the actual switch-out. Moved to fire immediately before the switch call, so
`switch_count_cumulative`/`switch_ticks_since` (the new DoE columns this whole follow-on was
for) attribute to the correct event.
**Verification, per this project's discipline:** all 3 architectures rebuilt clean after the
`sk_vm_at_outermost_interpret()` gate alone, boot-tested with the `HB-ON` probe still active --
log growth confirmed steady (no freeze) on amd64 (9324 frozen before the fix; 9035-10618 lines,
continuously growing, after), aarch64, and riscv64, all reaching natural completion with clean
QMP-independent exits (`-no-reboot` + outer `timeout`) and zero leaked QEMU processes. DoE columns
sampled directly: `switch_current_slot` matches the emitting VM's own registered slot in every
row checked, that VM's own readiness column correctly reset to 0 on its own rows,
`switch_count_cumulative` monotonically increasing across the run (spot-checked every 200th row).
`[Hermes]`-tagged DoE rows confirmed present (she does become the outermost-attributed VM and
genuinely executes FORTH, not just trampoline-bounces) with sane values. Rebuilt and re-verified
clean on all 3 architectures again after the two follow-on bug fixes (`native_switch_back_to`,
`note_switch_performed()` ordering) together. Finally, the temporary `HB-ON` boot-time probe
reverted from `kernel_main.c`, and all 3 architectures rebuilt and re-verified clean at rest
(idle `ok>`, no DoE spam, correct console banner) with the probe gone.
**Still open, not addressed this pass:** the trampoline's placeholder behavior itself (Stage 2's
original "no production behavior defined yet -- immediately yield back, forever") is now safe to
enter (no corruption) but still wasteful whenever a VM with no queued work crosses the readiness
threshold -- it bounces in and immediately back out via the fixed, correct mechanism, at real but
bounded cost (nothing like the old unbounded livelock). This is the same eligibility question
raised earlier in this follow-on and left open pending a proper answer: a message-arrival-hook
based "does this VM actually have pending work" signal was scoped in conversation (a per-VM
counter/flag, written at the FORTH-side message-enqueue site, read the same ISR-safe way
readiness already is) but not yet implemented -- superseded in priority by this stack-ownership
finding, which had to be fixed first. Revisit once this correction lands.