Fix use-after-free in sk_repl_idle()'s idle-tick VM resolution (FABRIC-3.md §XIII)
Build / build-amd64-iso (push) Waiting to run
Build / build-aarch64-iso (push) Waiting to run
Build / build-riscv64-img (push) Waiting to run

Root-caused a heap-corruption bug that reliably failed WIREBIND identity
attach on the third attach/detach cycle in one boot. sk_console_readline()
and sk_console_getkey() captured `active_vm` once from their caller and
kept passing that same (possibly long-stale) pointer to sk_repl_idle() on
every idle tick serviced while blocked waiting for input. If the VM it
pointed at was killed (WIREBIND detach) mid-block, the existing bailout
only checked a generic "is anyone attached" boolean -- masked as soon as a
different identity attached next -- so blk_vm_flush_all() kept writing
into a freed VM struct sitting on kmalloc's own free list, corrupting the
free list's linked-list metadata itself.

Both idle branches now re-resolve the live active VM fresh from
g_repl_active_vm on every tick, matching the dispatch-side fix already
made for the sibling bug in §XII.3.

Verified: rebuilt amd64, reran the exact three-cycle repro that reliably
corrupted the heap before the fix -- free-list census stayed stable
through the same idle window that previously collapsed to zero. All three
architectures (amd64/aarch64/riscv64) boot clean to the zuse)ok> prompt.

kmalloc_debug_census()/kmalloc_debug_census_bytes() kept as permanent
diagnostic infrastructure; every other temporary probe added during the
investigation was reverted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EXieurDfDSsDFdnSyusuWo
This commit is contained in:
Robert Allan James
2026-09-10 20:30:41 -04:00
co-authored by Claude Sonnet 5
parent 70421bdd43
commit 9142dda2d6
38 changed files with 236983 additions and 3 deletions
+101
View File
@@ -2140,3 +2140,104 @@ produced one interesting, unconfirmed data point before the first panic: `-123 4
`DOUBLE-OVERFLOW` via `D.` on amd64, identically for `zuse`, `rajames`, and `00` -- worth checking
against the other two architectures once the campaign completes; not yet root-caused or reported
as a bug on its own.
## XIII. Heap corruption under repeated WIREBIND attach/detach cycling — root-caused and CLOSED
2026-09-10
Surfaced while running the std79 exerciser campaign: cycling identities through one boot
(attach → `USE` → exercise → detach → attach next) on amd64 reliably failed after the *third*
WIREBIND identity cycle in a boot -- `xhci: USB MSC block-subsystem attach failed`, and the
target identity never births (`USE: NN not found`).
**Ruled out along the way, with live evidence, not just inspection:**
- **Not the image files** -- a failing identity's own thumbdrive attaches and births cleanly
from a cold boot in isolation.
- **Not a timing race between WIREBIND bind and the hotplug/IRQ event.**
- **Not heap fragmentation in the ordinary sense, and not something a defragmenter would fix.**
`kmalloc.c`'s allocator was read in full: real block splitting (`allocate_from_block()`/
`can_split()`) and real bidirectional coalescing on every `kfree()` (`coalesce_neighbors()`),
both correct. A free-list census (`kmalloc_debug_census()`, kept as permanent debug
infrastructure -- see below) plus a second census summing total free/used *bytes*
(`kmalloc_debug_census_bytes()`, added specifically to discriminate this investigation's two
competing hypotheses) confirmed that genuine cross-VM fragmentation is real but benign at this
scale: `free_blocks` climbs by roughly 1,000-1,100 per birth/kill cycle (interleaved
`DictEntry` allocations from other live VMs prevent `coalesce_neighbors()` from merging a
killed VM's freed entries back together), while `total_free_bytes` stayed flat at ~497-509 MB
and `largest_free` never dropped below ~492 MB across two full cycles. A third, later boot with
three full cycles (00/01/02) reproduced the real failure signature under the same probes:
`total_free_bytes` collapsed from ~508 MB to **literally 0** between two consecutive idle
ticks, with `used_blocks` staying a sane ~7,300 throughout -- proving this was never
fragmentation, it was the free list's own linked-list metadata being overwritten.
- **Not a garbage/oversized BAM allocation from a bad device-geometry read** -- geometry probes
around `blkio_info()`/`blk_format_or_load_disk()` showed clean, identical values every
successful cycle, and the failing cycle never even reached that code.
- **Not `capsule_wirebind_try_attach()`'s own body, including cert verification (BINDSTEP)** --
directly tested by bypassing the BINDSTEP block entirely; the corruption still occurred at the
same point, one cycle later, ruling the whole cert-verification/X.509 path out.
- **Not the MSG-TICK pump's own `vm_interpret()` dispatch, `blk_migration_idle_check()`, or
`capsule_wirebind_overflow_idle_check()`** -- each individually bracketed with pre/post census
probes; all read clean at the exact tick the corruption appeared elsewhere.
**Root cause, confirmed via bisection down to the exact tick and line:** `sk_repl_idle()`'s
`blk_vm_flush_all(active_vm)` call was the exact point where `total_free_bytes` collapsed
(`TD:flush pre lf=492362056 fb=1140``TD:flush post lf=0 fb=0`, same tick, no intervening
code). `blk_vm_flush_all()` (`block_words.c`) reads and writes `vm->blk_vm_lbn[]`,
`vm->blk_vm_cbuf[]`, `vm->blk_vm_dirty[]`, and `vm->blk_vm_epoch` on whatever `VM *` it's given
-- and the `active_vm` it was given traced back to a genuine use-after-free in
`src/starkernel/repl.c`, sibling to the bug already fixed once in §XII.3 (2026-09-09/10) but not
fully closed by that fix:
- `sk_repl_run()`'s main loop resolves `active = g_repl_active_vm ? g_repl_active_vm : vm;` once
per line, then calls `sk_console_readline(input, sizeof(input), active, 1)` -- which can block
for an arbitrarily long real-world time waiting for the next character.
- **While blocked**, `sk_console_readline()`'s own idle branch called
`sk_repl_idle(active_vm)` every `SK_IDLE_BEAT_INTERVAL` using that *same parameter*, captured
once, for the entire duration of the block (`repl.c`, was line 978).
- The bailout meant to catch a mid-read logout (`if (reanchor_prompt && n == 0 &&
!sk_console_identity_present()) return -1;`) checks a **generic "is anyone attached at all"
boolean**, not "is the specific identity `active_vm` belonged to still attached" -- the exact
anti-pattern already flagged once before in this codebase (see
`feedback_boolean_guards_vs_identity_comparison.md`). So: identity A is the active VM when this
read begins and gets WIREBIND-detached (killed, freed) while the read sits idle;
`sk_console_identity_present()` reads **false** for one tick, but before this idle branch ever
observes that false and bails, identity B attaches (a normal, unrelated next cycle) and
presence flips back to **true** -- permanently masking that the *original* `active_vm` (A) is
long gone. Every subsequent idle tick inside this same still-blocked read keeps calling
`sk_repl_idle(active_vm)` -> `blk_vm_flush_all(active_vm)` on A's freed `VM` struct.
- A's freed struct is, at that point, sitting on kmalloc's own free list.
`blk_vm_check_epoch()`'s unconditional field writes (`vm->blk_vm_lbn[i] = 0;` etc., inside
`blk_vm_flush_all()`'s own epoch check) land inside that free block's memory -- and if any of
`VM`'s `blk_vm_*` fields happen to overlap where `kmalloc.c`'s own `heap_block_t` stores
`next`/`size`/`free`, the write corrupts the free list's own linked-list metadata. This exactly
explains the observed signature: a walk that used to see ~2,100 healthy free blocks suddenly
sees 0, because the `next` pointer chaining them together got overwritten with zeros.
Reproducibly on the *third* identity cycle specifically because that's the point at which the
"gone, then something else attaches" masking sequence first lines up with a still-blocked read
from an *earlier* cycle's identity.
**Fix (commit follows this entry):** re-resolve the live active VM fresh on every idle tick,
inside both `sk_console_readline()`'s and `sk_console_getkey()`'s idle branches, exactly
matching the pattern already used at the dispatch site (§XII.3's fix): `VM *live_active =
g_repl_active_vm ? g_repl_active_vm : (VM *)sk_get_mama_vm();` then `sk_repl_idle(live_active);`
-- never trust the `active_vm` parameter captured before the block began. `sk_get_mama_vm()`
(Hera) is the safe fallback, matching `sk_repl_run()`'s own `g_repl_active_vm ? g_repl_active_vm
: vm` shape. Both functions' `active_vm` parameters are now unused for this purpose (silenced
with `(void)active_vm;`, kept in the signature since `repl.h` declares it and other callers still
pass one) -- no fix on the *bailout* boolean at line ~1029 was needed or made, since the memory
corruption is now impossible regardless of whether that cosmetic prompt-reanchor logic still has
its own generic-boolean flaw.
**Verified:** rebuilt amd64, reran the exact three-cycle repro (00 attach/USE/detach, 01
attach/USE/detach, 02 attach/USE/detach) that reliably corrupted the heap before the fix, waited
through the same idle window that previously collapsed `total_free_bytes` to 0 -- census stayed
rock solid (`lf=497605128 fb=2401 tfb=508518296`, unchanged) across dozens of subsequent idle
ticks and `blk_vm_flush_all()` calls.
**Debug infrastructure kept, not reverted:** `kmalloc_debug_census()` and
`kmalloc_debug_census_bytes()` (`src/starkernel/memory/kmalloc.c` /
`include/starkernel/kmalloc.h`) -- both small, cheap, read-only free-list walks, kept as
permanent reusable diagnostic infrastructure per the existing `kmalloc_debug_census()`
precedent. Every other probe added during this investigation (`block_words.c`,
`block_subsystem.c`, `capsule_birth.c`, `vm_core.c`, `mama_forth_words.c`, and the rest of
`repl.c`'s temporary brackets) was reverted after use, per the established "write, run once,
capture, revert" discipline.