Stage 3 follow-on: fix stack-ownership corruption + DoE switch columns (FABRIC-3.md §XXVIII.1)
DoE CSV gained 6 switch-signal columns (switch_count_cumulative, switch_current_slot, switch_*_readiness, switch_ticks_since), and verifying them with a boot-time HB-ON probe surfaced a real livelock: the preemption checkpoint could fire inside a VM-EXEC-nested execute_colon_word() call and switch away from a stack it didn't own, parking a borrowed region of the caller's stack under the wrong VM's saved-context pointer. The trampoline bounce was the visible (safe) half of this; the corruption was the quiet half, live in every prior "clean" Stage 3 boot without ever showing up in the log. Fixed by gating the checkpoint on being at the outermost vm_interpret() call (g_vm_interpret_depth / sk_vm_at_outermost_interpret(), vm_core.c), per Bob's decision. Also fixed two related bugs found in the same pass: g_switch_back_to was a single global stale after first entry, now per-VM state (native_switch_back_to); note_switch_performed() fired on resume instead of switch-out, now called before the switch. Verified on all 3 architectures: steady log growth (no freeze), zero leaked QEMU processes, DoE columns internally consistent, Hermes/Artemis confirmed genuinely executing (not just trampoline-bouncing). Temporary HB-ON boot probe reverted after capture. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016UNhH1mhi52i6Qihh7ZV5S
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
986d042aa7
commit
862d7d9c48
+106
@@ -3698,3 +3698,109 @@ switching and zero fault indicators, interactive console commands computed corre
|
||||
(WIREBIND-scope extension) remains a ratified-decision-only step, not attempted -- the async
|
||||
unclean-detach UAF risk it names is unchanged by anything built in Stages 0-3.
|
||||
|
||||
## XXVIII.1 -- Stage 3 follow-on: DoE CSV columns exposed real stack-ownership corruption in the
|
||||
switch checkpoint, root-caused and fixed (2026-09-13)
|
||||
|
||||
**Correction to §XXVIII's own "Stage 3 CLOSED" claim above:** it overclaimed. "Zero fault
|
||||
indicators" was true of what was *visible* -- Stage 3 emits nothing per switch by default, so a
|
||||
switch storm and a healthy idle REPL produce an identical serial log. The corruption below was
|
||||
already live in every one of those "clean" Stage 3 boots; nothing in that verification pass could
|
||||
have shown it.
|
||||
|
||||
**Motivation:** wanted `1 1 + .` -> `2` -- a real trace of switching activity for
|
||||
correlation/tuning. Added 6 columns to the DoE CSV (`doe_log.c`): `switch_count_cumulative`,
|
||||
`switch_current_slot`, `switch_hera_readiness`, `switch_hermes_readiness`,
|
||||
`switch_artemis_readiness`, `switch_ticks_since` -- all read-only exposure of state
|
||||
`capsule_vm_switch_signal.c` already tracked internally for the switch decision itself. To
|
||||
actually see rows emit without interactive input (foreground-only QEMU, no way to type `HB-ON`),
|
||||
added a temporary boot-time `vm_interpret(mama, "HB-ON")` probe in `kernel_main.c`.
|
||||
|
||||
**The probe surfaced a real livelock, not a probe artifact.** Booted amd64: DoE rows emitted
|
||||
correctly at first, then the serial log froze solid (`wc -l` identical across repeated checks,
|
||||
15+ seconds) while `ps aux` showed the QEMU process at 102% CPU -- actively spinning, not blocked
|
||||
on I/O. Bob's own real-time observation prompted the check: "is the serial moving? framebuffer
|
||||
frozen static view. i wont say stuck."
|
||||
|
||||
**Root cause, found by reading `switch.c` and `vm_core.c` together, not by further boot attempts:**
|
||||
`sk_vm_context_switch(from, to)` unconditionally saves whatever the *current* stack pointer
|
||||
happens to be into `from->native_stack_saved_sp` -- it has no notion of whether `from` actually
|
||||
owns the physical stack it is currently standing on. The Stage 3 checkpoint lives inside
|
||||
`execute_colon_word()`, whose `vm` argument is *not always* the VM that owns the current native
|
||||
stack: `VM-EXEC`/`VM-CALL` (`mama_word_vm_exec`/`mama_word_vm_call`, `mama_forth_words.c`)
|
||||
dispatch into a *different* VM's dictionary via a nested `vm_interpret()` call, still physically
|
||||
running on the caller's own stack (Hera's, in the common case -- her own `MSG-TICK` idle-pump
|
||||
calling `VM-EXEC` into Hermes or Artemis). When the checkpoint fires inside that nested call, `vm`
|
||||
is the nested VM (e.g. Hermes), but the stack is Hera's. A real switch there parks a *borrowed*
|
||||
region of Hera's own stack under Hermes's saved-context pointer -- a stack-ownership violation.
|
||||
If Hera is later independently resumed at her own (enclosing, higher-address) saved SP, she
|
||||
resumes the same call chain and pushes new frames straight over the region "Hermes" believes she
|
||||
owns: silent corruption. Confirmed reachable, not theoretical -- `execute_colon_word` is the code
|
||||
pointer for every colon word (`vm_create_word(..., execute_colon_word)`), reached both by Hera's
|
||||
own top-level REPL dispatch (outermost, genuinely owns the stack) and, nested inside that exact
|
||||
call chain, by her own `MSG-TICK`'s `VM-EXEC` (does not own the stack) -- both firing sites are
|
||||
real code paths, not a hypothetical interleaving.
|
||||
|
||||
**This subsumes the livelock, doesn't sit beside it as a separate bug.** The livelock is the
|
||||
*safe* half of this defect: first-ever entry into Hermes/Artemis (no prior parked context) lands
|
||||
in `sk_vm_switch_entry()`'s placeholder trampoline (deliberately a no-op yield-back, per Stage
|
||||
2's own documented, not-yet-superseded contract), which immediately bounces back -- readiness
|
||||
re-accumulates, the checkpoint re-fires, bounce again, a two-body ping-pong at full CPU with zero
|
||||
FORTH progress. The *unsafe* half -- switching away from a nested nonzero-depth frame -- produces
|
||||
quiet memory corruption instead of a loud freeze, and was not distinguished from the livelock
|
||||
until traced through the code.
|
||||
|
||||
**Fix: gate the checkpoint on being at the outermost `vm_interpret()` call, per Bob's explicit
|
||||
decision ("only allow the checkpoint at the outermost frame").** Added `g_vm_interpret_depth`, a
|
||||
file-scope counter in `vm_core.c` incremented/decremented around `vm_interpret()`'s body, in the
|
||||
same save/restore position as the existing `g_log_attrib_vm` (which already brackets nested
|
||||
`vm_interpret()` calls for a different reason -- log-source attribution). Exposed via
|
||||
`sk_vm_at_outermost_interpret()` (`depth <= 1`). The Stage 3 checkpoint in `execute_colon_word()`
|
||||
now only calls `sk_vm_switch_signal_take_pending()` when this holds; when nested, the pending
|
||||
switch is left untouched (not consumed) so the next checkpoint at the outermost frame -- whichever
|
||||
VM that turns out to be -- picks it up instead of losing it. Framing worth keeping: Stage 2's
|
||||
switch primitive is correct under its actual documented contract (manual, cooperative, top-level
|
||||
use only); Stage 3 called it from a nested dispatch point without checking that contract -- a
|
||||
Stage 3 integration defect, not a flaw in Stage 2 itself.
|
||||
|
||||
**Two more bugs found and fixed in the same pass, same root cause (no per-context ownership
|
||||
tracking), both explicitly authorized before touching code:**
|
||||
- `g_switch_back_to` was a single global captured once at a VM's *first-ever* trampoline entry,
|
||||
then read from a local baked into that trampoline's own stack frame forever after. A later
|
||||
switch-in from a *different* VM than the original caller would still bounce back to the
|
||||
original caller, silently stranding whoever actually switched in. Fixed by moving it to
|
||||
per-VM state -- `native_switch_back_to` (new `VM` field, alongside the other Stage 1/2
|
||||
native-stack fields it belongs with), written unconditionally by `sk_vm_context_switch()` on
|
||||
every switch-in (first-ever or resumed), read fresh every trampoline loop iteration instead of
|
||||
once.
|
||||
- `sk_vm_switch_signal_note_switch_performed()` was called *after* `sk_vm_context_switch()`
|
||||
returned -- which, for a real switch, is on *resume*, an arbitrary number of ticks later, not
|
||||
on the actual switch-out. Moved to fire immediately before the switch call, so
|
||||
`switch_count_cumulative`/`switch_ticks_since` (the new DoE columns this whole follow-on was
|
||||
for) attribute to the correct event.
|
||||
|
||||
**Verification, per this project's discipline:** all 3 architectures rebuilt clean after the
|
||||
`sk_vm_at_outermost_interpret()` gate alone, boot-tested with the `HB-ON` probe still active --
|
||||
log growth confirmed steady (no freeze) on amd64 (9324 frozen before the fix; 9035-10618 lines,
|
||||
continuously growing, after), aarch64, and riscv64, all reaching natural completion with clean
|
||||
QMP-independent exits (`-no-reboot` + outer `timeout`) and zero leaked QEMU processes. DoE columns
|
||||
sampled directly: `switch_current_slot` matches the emitting VM's own registered slot in every
|
||||
row checked, that VM's own readiness column correctly reset to 0 on its own rows,
|
||||
`switch_count_cumulative` monotonically increasing across the run (spot-checked every 200th row).
|
||||
`[Hermes]`-tagged DoE rows confirmed present (she does become the outermost-attributed VM and
|
||||
genuinely executes FORTH, not just trampoline-bounces) with sane values. Rebuilt and re-verified
|
||||
clean on all 3 architectures again after the two follow-on bug fixes (`native_switch_back_to`,
|
||||
`note_switch_performed()` ordering) together. Finally, the temporary `HB-ON` boot-time probe
|
||||
reverted from `kernel_main.c`, and all 3 architectures rebuilt and re-verified clean at rest
|
||||
(idle `ok>`, no DoE spam, correct console banner) with the probe gone.
|
||||
|
||||
**Still open, not addressed this pass:** the trampoline's placeholder behavior itself (Stage 2's
|
||||
original "no production behavior defined yet -- immediately yield back, forever") is now safe to
|
||||
enter (no corruption) but still wasteful whenever a VM with no queued work crosses the readiness
|
||||
threshold -- it bounces in and immediately back out via the fixed, correct mechanism, at real but
|
||||
bounded cost (nothing like the old unbounded livelock). This is the same eligibility question
|
||||
raised earlier in this follow-on and left open pending a proper answer: a message-arrival-hook
|
||||
based "does this VM actually have pending work" signal was scoped in conversation (a per-VM
|
||||
counter/flag, written at the FORTH-side message-enqueue site, read the same ISR-safe way
|
||||
readiness already is) but not yet implemented -- superseded in priority by this stack-ownership
|
||||
finding, which had to be fixed first. Revisit once this correction lands.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user