Fix O(N) idle-loop messaging pump; full 3x9x3 turn-attractor campaign clean on all 3 architectures (FABRIC-3.md §XXII)

sk_repl_idle()'s messaging pump (repl.c) walked the entire live-VM
registry every idle beat (~1Hz) and dispatched a full VM-EXEC
"MSG-TICK" -- a dictionary lookup plus a 32-slot arena scan -- into
every live VM, every tick, unconditionally, forever. Fine at Tripod's
original 3-VM scale; a full 9-identity turn-attractor campaign
exposed it as a genuine wall on riscv64 specifically (its TCG makes
each dispatch cost more): the same campaign that completed in 194s/
339s on amd64/aarch64 never finished on riscv64 at 9 VMs across three
attempts, while 8 VMs there was fine in 159s.

Ruled out capacity explanations before touching anything: bumping
riscv64's QEMU RAM 1024->4096 changed nothing (reverted), and a live
STADIUM-RES@/MSG-STATUS probe with all 9 VMs attached showed no
depletion. Host memory pressure was also ruled out directly (one
background task did get OOM-killed once during the investigation,
but the identical stall reproduced again with 9.2GB free). The real
mistake was three premature kills under 4 minutes with no way to
tell "slow" from "stuck" from outside the guest -- fixed by having
run_doe_batch.sh sample the qemu process's own /proc/<pid>/stat utime
every 60s; with that signal, riscv64 at 9 VMs was unambiguously alive
(climbing utime, no hang), just disproportionately slow going from 8
VMs (159s) to 9 (600s+ and climbing).

This was never really a riscv64-only bug: an O(N) per-second walk
over the full VM population doesn't scale to the hundreds of VMs this
fleet is headed toward, on any architecture -- riscv64 just made it
visible first, at N=9, because its per-dispatch cost is highest.

Fixed by round-robin batching: the pump now dispatches to at most
SK_MSG_PUMP_BATCH (4) live VMs per idle beat via a persistent cursor
that resumes where the previous beat left off, instead of all of them
every time. Bounds both the scan and dispatch cost to O(K) regardless
of total VM count; any single VM's queue now drains roughly every
ceil(N/K) beats instead of every beat, still bounded and still
matching the pump's own existing best-effort contract. No new C
primitives, no messaging/Stadium changes.

Verified: clean build on all 3 architectures, then the full 3x9x3
campaign re-run on all 3 (not just riscv64) per the standing rule
that a defect repair requires a clean re-run everywhere before
anything counts as closed:

  amd64   198s  0 faults  99/99 tokens x8  656/656 K-conserved
  aarch64 339s  0 faults  99/99 tokens x8  659/659 K-conserved
  riscv64 178s  0 faults  99/99 tokens x8  659/659 K-conserved

riscv64 went from "never completes" to faster than aarch64, same
campaign, same seed, same identity set. No regression on amd64/
aarch64.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EXieurDfDSsDFdnSyusuWo

FABRIC-3.md

@@ -2928,3 +2928,114 @@ rebuild, and the LaTeX report comparing it against the §XV/§XVIII serialized b
 next step, now that the mechanism itself has been shown correct at small scale rather than
 assumed correct from a clean exit code.
 
+## XXII. Full 9-identity campaign: a real riscv64-only stall, root-caused as an O(N) idle-pump
+cost with no cap -- fixed by round-robin batching, all three architectures now clean (2026-09-12)
+
+**amd64 and aarch64 passed cleanly first try** at full scale (9 identities, 27 reps): amd64 in
+194s, aarch64 in 339s, both zero faults, all 8 messaged identities at exactly 99/99 expected
+tokens, `fleet_conserved=1` across every telemetry row (654/659 respectively). riscv64 did not.
+
+**riscv64 stalled with zero visible output, repeatedly, and it took real discipline not to
+misdiagnose it.** The first full-scale attempt showed nothing -- no `DOE-HEADER` line, nothing --
+for 1673s before the orchestration script's own deadline killed it. The instinct was "hung";
+the evidence for that was thin. A 2-identity riscv64 smoke test completed cleanly in 70s (same
+mechanism, same code, working fine on this architecture at small scale), and bisecting identity
+count upward (5, 6, 7, 8 total VMs -- 135s, 145s, 159s, all clean) found nothing until *exactly*
+9 total VMs, matching `N-ID` exactly. That specific shape -- fine at 8, dead at 9, on one
+architecture only -- pulled hard toward "hard capacity ceiling." It wasn't.
+
+**Two capacity hypotheses were tested and killed, not just argued away.** Bumping riscv64's
+QEMU RAM 1024→4096 (`Makefile.starkernel`, reverted after the test) changed nothing -- same
+stall signature, ruling out guest memory budget. A live probe of `STADIUM-RES@`/`MSG-STATUS`
+with all 9 VMs attached (before invoking any campaign) read 21735, 0 pending messages, 1
+channel -- indistinguishable from what a 2-VM attach showed earlier, ruling out "9-VM attach
+overhead alone exhausts the reservoir." Host memory pressure was also ruled out directly: the
+very first attempt happened while other desktop applications were using ~7GB of an 11GB host
+(a background task was genuinely OOM-killed once during this investigation), but the identical
+stall reproduced again with 9.2GB free.
+
+**The actual mistake, caught before it compounded: three premature kills with no way to tell
+"slow" from "stuck" from outside the guest.** Every kill happened under 4 minutes, based on
+"no console output yet." That's a weak signal on this codebase specifically -- this session's
+own campaign output repeatedly showed prints concatenating without a leading CR
+(`-1073741824 DOE-RUN,17`, a `DOE-RUN,` line split across an interleaved CSV header), so absence
+of a expected line is not strong evidence nothing executed. The qemu process itself was checked
+each time and was always actively running (100-125% CPU), never a genuine deadlock signature.
+Fixed the instrumentation before drawing another conclusion: `run_doe_batch.sh`'s completion
+wait now samples the qemu process's own `/proc/<pid>/stat` utime every 60s and logs whether it's
+climbing or flat -- a real, external, architecture-independent progress signal instead of a
+guess from console silence. Raised the wait deadline 1500s→5400s at the same time, since the
+whole point of a "looks stuck" call is worthless if the process never gets the time to prove
+otherwise.
+
+**With that instrumentation, riscv64 at 9 VMs was unambiguously alive, just slow -- climbing
+utime, steady ~4000-4500 jiffies per 60s window, sustained across 8+ consecutive checks past
+600s** with no sign of degrading or spinning out. That ruled out "hung" for good, but the
+slowness itself was still a real problem worth root-causing rather than waiting out indefinitely
+-- going from 8 total VMs (159s) to 9 (600s+ and climbing) is not a smooth scaling curve; one
+more VM should cost roughly one more VM's worth of work, not 4-10x the wall time.
+
+**Root cause: `sk_repl_idle()`'s messaging pump (`src/starkernel/repl.c`) walked the entire live-
+VM registry every idle beat (~1 Hz, `SK_IDLE_BEAT_INTERVAL`) and dispatched a full `VM-EXEC
+"MSG-TICK"` -- a real dictionary lookup plus a 32-slot arena scan -- into *every* live VM, every
+single tick, unconditionally, forever.** This is O(N) per tick with no cap, and it runs
+regardless of whether the campaign is even active. It was fine at Tripod's original 3-VM scale,
+and still workable on amd64/aarch64 at 9 -- but not free, and the cost lands hardest on
+whichever architecture's TCG makes each of those N dispatches most expensive. At 9 VMs on
+riscv64 specifically, this loop was plausibly the dominant cost driving the wall-time cliff.
+The registry itself is also monotonically non-decreasing (`vm_registry_alloc()`'s own doc
+comment: "incremented on every ..., never decremented") -- not the direct cause here (dead
+entries are skipped cheaply, before any dispatch), but the same shape of unbounded growth that
+makes the pump's cost scale with *every VM ever born*, not just the ones still doing anything.
+
+**Why this was never really a riscv64 bug, and had to be fixed regardless of which architecture
+found it first:** Captain Bob's own framing mid-investigation -- "we really will be giving birth
+to potentially hundreds of VMs" -- reframed the whole question. An O(N) per-second walk over the
+full VM population, unconditionally, is not a riscv64 quirk; riscv64 is just where it became
+visible first, at N=9, because its TCG cost per dispatch is highest. At hundreds of VMs this is
+a wall on every architecture. Fixing the symptom for riscv64 alone (bigger timeout, more RAM,
+document a per-arch VM-count ceiling) would have left the real problem for the next architecture
+or the next order of magnitude of VM count to rediscover.
+
+**The fix: round-robin batching, no new primitives, no messaging/Stadium changes.** The pump now
+dispatches to at most `SK_MSG_PUMP_BATCH` (4) live VMs per idle beat instead of all of them, via
+a persistent cursor (`g_msg_pump_cursor`, a static index into the registry) that picks up where
+the previous beat left off and wraps around. This bounds both the registry scan and the dispatch
+cost to O(K) per tick regardless of total VM count -- the only cost is that any single VM's own
+message queue now drains roughly every `ceil(N/K)` beats instead of every beat, which is still
+bounded, still predictable, and matches the pump's own pre-existing "best-effort, resumes next
+beat" contract exactly. Zero new C primitives, zero cross-VM state queries, a pure loop-bound
+change confined to `repl.c`.
+
+**Verification: built clean on all three architectures (zero new warnings), then the full
+9-identity campaign was re-run on all three -- not just riscv64-- per the standing rule that a
+defect repair invalidates prior results and requires a clean re-run before anything counts as
+closed.** Results, this time with the fix in place:
+
+| arch    | wall time | faults | tokens (8 identities) | fleet_conserved |
+|---------|-----------|--------|------------------------|------------------|
+| amd64   | 198s      | 0      | 99/99 each             | 656/656 rows     |
+| aarch64 | 339s      | 0      | 99/99 each             | 659/659 rows     |
+| riscv64 | **178s**  | 0      | 99/99 each             | 659/659 rows     |
+
+riscv64 went from "never completes" (three kills, one past 1673s) to 178s -- faster than
+aarch64, in the same range as amd64 -- with the identical campaign, identical seed, identical
+identity set. amd64 and aarch64 show no regression from the fix (194s→198s, 339s→339s, both
+within run-to-run noise). All three: zero `UNKNOWN WORD`/fault lines, every messaged identity at
+exactly 99/99 expected tokens (no drops), fleet K conservation holding on every telemetry row.
+
+**Also added, at Captain Bob's request, for the next investigation like this one:**
+`run_doe_batch.sh` (orchestration script) now writes a companion `host-sample-<arch>-<ts>.csv`
+alongside every run's serial log -- wall-clock-timestamped samples, every 5s, of the qemu
+process's own `/proc/<pid>/stat` utime and host `/proc/meminfo` (used/available/swap). Guest-side
+`doe_log.c` cannot report either of these -- it runs *inside* the emulated architecture and has
+no visibility into real host CPU consumption or memory pressure, only its own emulated clock.
+This is standing instrumentation now, not a one-off diagnostic: every future campaign run gets a
+joinable host-load record for free, rather than needing hand-checked `/proc` probes and `free -h`
+calls reconstructed after the fact, the way this investigation had to.
+
+**Next step:** rebuild the analysis pipeline against these three full campaign CSVs (now genuinely
+messaging-coordinated, priority-driven turn order, not shuffle-position-driven) and produce the
+LaTeX report comparing this against the §XV/§XVIII serialized baseline -- the actual point of
+building the turn-attractor in the first place.
+