Fix aarch64 Stadium/COOL O(ncells) scan; rerun std79 DoE clean, 81/81 (FABRIC-3.md §XVI)

Root cause of the 90+ minute aarch64 VM-birth stall found in §XV: stadium_admit()'s
eviction-fallback scan iterated the entire stadium_ncells array filtered by owner,
not the calling VM's own resident cells as its own doc comment claimed. Combined
with stadium_grant_quota() always splitting from Hera's shrinking free list and
stadium_word_dispatch() calling stadium_admit() per distinct word a VM's capsule
executes, this compounded into a real O(n) blowup — catastrophic specifically on
aarch64 because its -m 4096 (vs 1024 on amd64/riscv64) inflates the kmalloc heap
kmalloc_init() bisects down to, which inflates stadium_ncells 4x (335,544 vs 83,886
cells, measured from boot logs).

Fixed by threading a real per-VM doubly-linked resident-cell list
(StadiumVMQuota.resident_head + stadium_resident_next[]/stadium_resident_prev[])
so the fallback scan is bounded by that VM's own resident count, not the global
cell array size.

Verified with a full rerun of the 3x9x3 std79 DoE campaign from scratch: one
continuous boot per architecture, all 9 identities simultaneously live throughout
(the 3-boot aarch64/riscv64 batching workaround is no longer needed). 81/81 trials
correct, 0 mismatches, DOE-RUN header sequence md5-identical across all three raw
logs. Identity 04's attach on aarch64, which stalled 90+ minutes before, now
completes in ~34s; full boot-to-DoE-complete in ~290s.

Corrects an earlier misreading (carried into §XV, std79-doe.fth's comments, and
the project memory note) that described the symptom as a runaway "335,000+ cycles"
dispatch counter — those were cell array indices, not an event count.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EXieurDfDSsDFdnSyusuWo

FABRIC-3.md

@@ -2500,19 +2500,20 @@ baseline: **zero mismatches**. `T14`-equivalent (`-123 456 M* SWAP D.`) correctl
 in every trial -- confirms §XII.4 bug 1's fix (`M*`, commit `9a09949`) holds under this DoE's own
 `VM-EXEC`/native dispatch, not just the original exerciser's raw-text-feed path.
 
-**aarch64: a second real bug found, worked around, not root-caused.** Attaching all 9 identities
-simultaneously (matching amd64's approach) hit a severe wall partway through: identities
-`bob`/`rajames`, `00`-`03` attached normally (each ~1-2 minutes), but the 6th simultaneously-live
-VM (`04`) never completed -- QEMU stayed at 100%+ CPU, genuinely computing (confirmed via
-`ps` CPU-time climbing steadily, not a deadlock), but produced next to no new console output for
-over 90 minutes before the attempt was abandoned. The `Stadium: dispatch cell=NNNNN
-behaviour=COOL` background patron-churn counter (visible in the log immediately before the
-stall) had already reached **335,000+** cycles at that point -- tens of thousands more than the
-same checkpoint shows for earlier, successful identities -- suggesting a genuine algorithmic
-blowup (likely superlinear in live-VM count) in the Stadium COOL dispatch path specifically on
-aarch64, not just "TCG is slower." amd64 attached all 8 non-`zuse` identities with no such cliff
-at any point. **Not root-caused here** -- flagged as a real defect (see the discussion below);
-this session's priority was completing the requested DoE, not chasing a new performance bug.
+**aarch64: a second real bug found, worked around, not root-caused at the time (see §XVI for the
+full root-cause and fix, added 2026-09-11/12 -- the paragraph below is kept as an accurate record
+of what was actually observed live, but its causal framing was wrong and is corrected in §XVI).**
+Attaching all 9 identities simultaneously (matching amd64's approach) hit a severe wall partway
+through: identities `bob`/`rajames`, `00`-`03` attached normally (each ~1-2 minutes), but the 6th
+simultaneously-live VM (`04`) never completed -- QEMU stayed at 100%+ CPU, genuinely computing
+(confirmed via `ps` CPU-time climbing steadily, not a deadlock), but produced next to no new
+console output for over 90 minutes before the attempt was abandoned. The `Stadium: dispatch
+cell=NNNNN behaviour=COOL` lines visible in the log immediately before the stall showed cell
+indices in the 335,000+ range -- at the time this was misread as a runaway *event counter*
+("335,000+ cycles"); it is not one. Only a dozen or so such lines exist in the entire log --
+`cell=NNNNN` is a cell *array index*, and 335,xxx sits almost exactly where Hera's own free list
+converges after repeated halvings against a ~335,544-cell Stadium (see §XVI for the actual
+mechanism). amd64 attached all 8 non-`zuse` identities with no such cliff at any point.
 
 **Design revised to work around it:** each architecture other than amd64 runs as **3 boots of 3
 simultaneously-live identities each** (`zuse`+`rajames`+`00`, `01`+`02`+`03`, `04`+`05`+`06`),
@@ -2534,11 +2535,85 @@ identities appearing exactly 3 times. Zero mismatches against baseline on either
 identity, every replicate, every architecture, byte-identical to the single established
 baseline. Raw logs and this verification are in `experiments/std79-doe/results-20260911/`.
 
-**Follow-on item, not investigated further here:** the aarch64 Stadium/COOL dispatch stall.
-Worth a dedicated investigation later -- candidates to check first: whether `Stadium:
-dispatch cell=NNNNN` count genuinely scales worse than linearly with live VM count (the visible
-symptom), and whether it's aarch64-specific in a real sense or just the first architecture where
-TCG overhead pushed an existing superlinear cost over a wall-clock threshold nobody had hit
-before (amd64's own per-tick cost may have the same underlying shape, just cheap enough per-tick
-under native-ish TCG codegen that it never became visible in any campaign run so far).
+**Follow-on item:** the aarch64 Stadium/COOL dispatch stall was investigated, root-caused, and
+fixed -- see §XVI, and `experiments/std79-doe/results-20260911-stadium-fix/` for a clean rerun
+of this entire campaign, one boot per architecture, all 9 identities simultaneously live
+throughout, with the 3-boot batching workaround above no longer needed at all.
+
+## XVI. aarch64 Stadium/COOL scaling stall — root-caused and fixed (2026-09-11/12)
+
+**Investigation.** §XV's aarch64 stall was re-examined by hand-tracing the KRELTSC timestamps
+around identity `04`'s birth: its *console* birth completed normally, but the subsequent gap
+before its *user/runcap* birth was ~221.6 billion KRELTSC ticks -- roughly 26-27x identity `03`'s
+entire birth cycle -- with **zero** log output in between, ruling out ordinary TCG slowness (a
+genuinely slow-but-progressing computation would still print something eventually) in favor of a
+real algorithmic cost blowup somewhere in the VM-birth path.
+
+**Measured, not guessed:** both architectures' boot logs print their actual Stadium sizing --
+`Stadium: 335544 cells (20971 KB), 202 VM slots` on aarch64 (`-m 4096`) vs. `Stadium: 83886
+cells (5242 KB), 50 VM slots` on amd64 (`-m 1024`, same session) -- **exactly a 4.00x ratio**,
+tracking the RAM ratio precisely. `kernel_main.c` always *requests* a 2 GiB kmalloc heap
+(`KARGS_DEFAULT_HEAP_SIZE`); `kmalloc_init()` bisects that request down until the PMM can serve
+it as one contiguous region, so aarch64's 4x RAM lets it land far closer to the full 2 GiB
+request than amd64/riscv64's 1 GiB total RAM can. Both `stadium_ncells` and
+`stadium_max_vm_count_val` are fixed percentages of that heap (`STADIUM_MEMORY_PERCENT`=1%,
+`STADIUM_VM_MEMORY_PERCENT`=50% of the remainder), so this RAM difference propagates directly
+into Stadium's internal array sizes.
+
+**Root cause (`src/starkernel/vm/stadium.c`):** `stadium_grant_quota()` (called once per VM
+birth, from `capsule_birth.c`) always splits *from Hera's own free list*, handing the new VM
+half and keeping the remainder -- so Hera's free-list head position converges toward `ncells`
+after repeated halvings (for `ncells`=335,544, roughly there after ~11 successive grants, which
+is almost exactly where §XV's observed `cell=335,380..335,513` eviction indices sit -- not a
+coincidence, a fingerprint of quota depletion). Once a newly-granted VM's *own* small quota runs
+dry (a handful of distinct dictionary words into its own capsule's execution -- `stadium_admit()`
+is called from `stadium_word_dispatch()`, `stadium_words.c`, once per first-execution of each
+distinct word, not just once at VM birth), `stadium_admit()` fell back to a scan that its own doc
+comment already claimed was "scoped to that SAME VM's own resident patrons only" -- but the
+actual implementation was `for (i = 0; i < stadium_ncells; i++) { if (owner[i] != slot) continue;
+... }`, an **O(stadium_ncells) full-array scan** (with a `stadium_density()` call per candidate),
+not O(that VM's own resident count) as documented. At 335,544 cells on aarch64 (vs. 83,886 on
+amd64), and hit repeatedly as more words execute against an ever-thinner per-VM quota, this
+compounds into the observed 90+ minute stall -- the doc comment's stated intent and the actual
+code had silently diverged.
+
+**Fix:** threaded a real doubly-linked per-quota resident-cell list (`resident_head` on
+`StadiumVMQuota`, plus global `stadium_resident_next[]`/`stadium_resident_prev[]` parallel
+arrays, sized and zero-initialized in `stadium_boot_init()` alongside the existing bitmap/owner
+arrays) so `stadium_admit()`'s eviction-fallback scan walks only the calling VM's own resident
+cells -- bounded by that VM's granted quota (a small, RAM-independent number in practice), not
+`stadium_ncells`. `resident_list_push()`/`resident_list_remove()` (`stadium.c`) maintain the list
+at every point a cell becomes resident (both `stadium_admit()` success paths) or stops being
+resident (`stadium_evict()`, called before the bitmap is cleared so the owning slot is still
+known). No other call site touches cell residency, so this is a closed, self-contained fix.
+
+**Verification: a full, clean, single-boot rerun of the entire §XV campaign on all three
+architectures**, all 9 identities simultaneously live for the whole run in every case (the
+3-boot batching workaround is no longer needed anywhere) --
+`experiments/std79-doe/results-20260911-stadium-fix/`:
+
+- amd64: boot + all 9 attaches + 27-trial run, one continuous QEMU session, **~165s total**.
+- aarch64: same, **~290s total** -- identity `04`'s attach specifically, which stalled 90+
+  minutes before the fix, now completes in **~34s**.
+- riscv64: same, **~161s total** (riscv64 was never affected, included for full re-verification
+  since the fix touches shared Stadium code all three architectures link against).
+
+All three logs' `DOE-RUN,run_id,id_idx,id_label,rep` header sequences are **md5-identical**
+(confirming the master Fisher-Yates shuffle is genuinely architecture-independent under the same
+seed), and the canonical 28-number per-trial result string appears **exactly 27 times in each
+log, byte-identical** -- **81/81 trials correct across all three architectures, 0 mismatches,
+0 workarounds needed.** This is the first genuinely defect-free run of this campaign.
+
+**Correction, superseding §XV's causal framing:** the "335,000+ cycles" language in §XV was a
+misreading (cell array indices, not an event counter -- only ~12 total `Stadium: dispatch` lines
+exist in the whole stalled log) carried into that section's live write-up and into
+`experiments/std79-doe/std79-doe.fth`'s own comment block and the
+`project_std79_doe_complete.md` memory note; all three have been corrected to point here.
+
+**Latent secondary issue, recorded not fixed (out of scope for this pass):** the free-list
+halving in `stadium_grant_quota()` has no floor -- after enough successive VM births Hera's own
+free list drops below 2 cells and every subsequent grant is silently refused (`half == 0` -> -1),
+which `capsule_birth.c` currently treats as non-fatal. Same root mechanism as the fix above, a
+different (silent-failure-to-birth, not stall) symptom, not yet a problem at any VM count actually
+exercised so far.