From 36f1d6ae9e08fcddff76a0d85865a9d35d79d4e4 Mon Sep 17 00:00:00 2001 From: Robert Allan James Date: Mon, 7 Sep 2026 11:39:18 -0400 Subject: [PATCH] =?UTF-8?q?FABRIC-3.md=20=C2=A7X:=20document=20the=20?= =?UTF-8?q?=C2=A7IX.5=20follow-on=20=E2=80=94=20xHCI=20root-cause/fix,=20l?= =?UTF-8?q?ive=20hotplug=20walkthrough,=20identity=20heap=20capacity=20fin?= =?UTF-8?q?dings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents three commits' worth of live-tested work in one continuous arc, picking up exactly where §IX.5 left off: - X.1: the "4th-device enumeration failure" was a QEMU test-harness port-topology artifact, not a driver bug (commit 30c26ad). - X.2: the real driver bug it uncovered once ports were fixed -- Configure Endpoint completions silently dropped under concurrent multi-device enumeration, root-caused via temporary reverted probes and fixed with the same single-in-flight discipline the rest of the driver already uses (commit e10fb76). - X.3: a live 9-device hotplug walkthrough (Zuse, bob/rajames, 00-06, one at a time via QMP) verifying the dynamic attach path and per-device identity data integrity, distinct from the boot-time scan path X.1/X.2 exercised. - X.4: exact, live-measured kernel-heap-arena numbers per identity VM (457,392 bytes, zero variance across 6 identities), a real leak found and fixed (orphaned console VM never torn down on user-VM birth failure, commit d8a195b), and the open architecture question this surfaces -- identity capacity is fragmentation-limited around 6 concurrent identities today, not yet decided how to address. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_014Ec88YKxxhZGG1RNnune78 --- FABRIC-3.md | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) diff --git a/FABRIC-3.md b/FABRIC-3.md index 1711a71d..0077fe9f 100644 --- a/FABRIC-3.md +++ b/FABRIC-3.md @@ -1495,3 +1495,154 @@ failure-path logging itself hasn't been exercised live yet, only compiled. Three failure that motivated §IX.5 (`xhci: not a Mass Storage/SCSI/BOT device` on a 4th concurrent attach) remains unexplained and unfixed — tracked as the next item, separate from the four closures above. + +## X. §IX.5 follow-on: xHCI concurrent enumeration, root-caused and CLOSED; identity VM heap +capacity measured, one real leak found and fixed, capacity question still OPEN — 2026-09-06/07 + +Picks up exactly where §IX.5 left off. Two separate root causes, both real, found by actually +attaching multiple devices live rather than reasoning about the driver in the abstract — same +lesson §VII and §IX already taught this project twice over. + +### X.1 — The "4th-device enumeration failure" was never a driver bug — CLOSED, commit `30c26ad` + +Root cause, confirmed live via QEMU's own `info usb` monitor command *before touching any +kernel code*: the default `qemu-xhci` controller (`p2=4,p3=4`) exposes only **4 real dual-role +ports**, not 8 as the parameter names suggest (p2/p3 count the *same* physical ports from the +USB2/USB3 register sets, not additive). Every device attach in this project used +`-device usb-storage,bus=xhci0.0,drive=...` with no explicit `port=`; past the 4th device, QEMU +silently auto-inserts an internal USB2 hub and everything past that is invisible to a driver +with no hub-descent logic. The driver's `"not a Mass Storage/SCSI/BOT device"` warning was +**correct** — the device really was a hub. The per-slot xHCI/BOT machinery from §VII was +inspected and found structurally sound for real top-level-port devices; no live corruption bug +was in it. + +**Fix, entirely in the QEMU test harness, not the kernel:** new `XHCI_PORTS` Make variable +(`Makefile.starkernel`, default 16, overridable) sizes `p2`/`p3` on all three arches with real +headroom above the current 9-device identity roster — sized deliberately, not exactly matched +to today's count, per the standing ruling against hardcoding a bound to today's scale (§VII.4). +`ZUSEDISK_QEMU_ARGS` now gives Zuse's drive an explicit `port=1`; `QEMU_EXTRA`'s own doc comment +shows the pattern for attaching more devices. + +**Found and fixed in passing:** `ZUSEDISK`'s Makefile default (`disk/zuse.img`) was stale — +deleted from git at `c3db963`, superseded by `disk/thumbdrives/zuse-thumb-ident.img`, but the +default was never updated, so a plain `make qemu` silently booted with no Zuse device attached +at all. Corrected to the real minted image. + +**Flagged, not touched (not asked):** `scripts/bleach_zuse_img.sh` and `disk/README.md` still +reference the deleted `disk/zuse.img` path. + +Three-arch `clean qemu` acceptance (single Zuse device, port 1) passed on amd64/aarch64/riscv64 +with this change. + +### X.2 — The real driver bug: Configure Endpoint completions dropped under concurrency — CLOSED, commit `e10fb76` + +With X.1's port fix in place, actually booting the real kernel with Zuse + all 8 identity +drives attached concurrently (9 devices, each its own real port) surfaced a second, genuine bug: +only **1 of 9** devices ever reached `blkio_usb: MSC device ready`. No error and no success for +the other 8 — silence, not a diagnosable failure. + +Root-caused live via temporary WARN-level diagnostic probes (written, captured, fully reverted +— the committed fix contains no probe code) at four points in `xhci.c`: the initial port scan, +the connect handler, the Command Completion Event handler, and the deferred per-slot dispatch +loop. The probes disproved the first hypothesis (a stalled `pending_events[]` connect queue — +all 9 devices actually completed Enable Slot + Address Device correctly) and found the real one: +`xhci_poll_events()`'s deferred dispatch loop submitted `XHCI_NEXT_ACTION_CONFIGURE_ENDPOINT` (a +Command Ring op) **unconditionally** for every slot with that action pending in a single pass — +unlike Enable Slot/Address Device/Disable Slot, which are correctly gated behind +`dev->connect_state == XHCI_CONN_IDLE` before ever submitting. With 2+ devices enumerating at +once, this let multiple Configure Endpoint commands sit outstanding on the Command Ring +simultaneously. Their completion is correlated purely via the single shared `dev->connect_state` +field, not the completion event's own Slot ID — so whichever slot's completion happened to land +while `connect_state` still read `AWAIT_CONFIGURE_ENDPOINT` got correctly chained into +`SET_CONFIG`; every other slot's completion arrived after `connect_state` had already moved on +and was silently swallowed by the handler's generic "unrelated command completion" catch-all — +no error path exists for that catch-all, which is exactly why it produced silence rather than a +diagnosable failure. + +**Fix:** the same single-in-flight discipline this driver already uses for every other Command +Ring op. If the Ring isn't free when a slot's Configure Endpoint action comes due, put the +action back on that slot instead of racing a second command onto a busy Ring; the next tick's +dispatch pass retries it. + +**Verified live:** the same 9-device concurrent boot now produces 9 `MSC device ready` lines and +zero xHCI errors (was 1 of 9). Three-arch single-device regression (the standard acceptance +case) still passed on all three arches. + +### X.3 — Live 9-device hotplug walkthrough, on request: dynamic attach path also verified, not just boot-time scan + +Separate from X.1/X.2's boot-time-attach testing, Captain Bob directed a live walkthrough: one +QEMU instance up with nothing attached, then Zuse, bob/rajames, and 00 through 06 hotplugged one +at a time via QMP `drive_add`+`device_add` on distinct `XHCI_PORTS`, watching the kernel's live +reaction after each. This exercises `xhci_poll_events()`'s Port Status Change *event* path +(true hotplug), not `xhci_scan_ports_for_already_connected()`'s boot-time initial-scan path +(X.1/X.2's scenario) — a genuinely different code path through the same driver. + +Every hotplug through 04 succeeded cleanly: `blkio_usb: MSC device ready`, correct distinct +per-device identity readback (a temporary probe read each drive's own `full_name`/`username` +straight off its own devblock via `blkio_read()`, confirming e.g. slot 3 really said `"00"`, not +Zuse's or bob's data — no cross-contamination), and a live `WIREBIND` birth per identity. Six +identities (including Zuse, who costs no extra VM) were live simultaneously with zero errors. +The 05 hotplug is where X.4 below picks up — its USB/xHCI attach itself succeeded fine (further +confirming X.2's fix generalizes to the hotplug path, not just boot-time scan); what failed next +was a different layer entirely. + +### X.4 — Identity VM kernel-heap capacity: exact numbers measured live, one real leak found and FIXED (commit `d8a195b`), the capacity/fragmentation question itself is OPEN + +X.3's 05 hotplug attached at the USB layer but then failed at `WIREBIND`'s VM-birth step with a +kernel allocator exhaustion error (`vm_create_word: malloc failed`, `vm_init: rolling window +malloc failed`, `PARITY:BIRTH_FAILED`) — graceful, no crash, boot continued to `(zuse) ok>`, but +a real scalability wall. Measured precisely (not estimated) via a temporary allocator-stats +probe in `capsule_run.c`'s `capsule_parity_log_birth`/`capsule_parity_log_birth_failed` (written, +captured, fully reverted — no probe code committed), replaying the same hotplug sequence: + +- Kernel heap arena (`alloc_kernel.c`, `SF_ARENA_SIZE`) is a **fixed 4 MiB (4,194,304 bytes)** — + does not grow. +- Boot baseline (Hera + Hermes + Artemis, the 3 system VMs): **781,904 bytes**. +- Zuse costs **nothing extra** — she authenticates into the existing Hera session rather than + birthing a new VM. +- Every other identity (`WIREBIND`) costs **exactly 457,392 bytes, with zero variance**, + confirmed identical across bob, 00, 01, 02, 03, 04. This is two VM births per identity: a + "default personality" console VM (228,576 bytes) then the real user session (228,816 bytes). +- **6 identities fit** (bob + 00 + 01 + 02 + 03 + 04): arena at 3,526,256 / 4,194,304 used, + 668,048 free. +- **The 7th identity (05) failed — not from clean capacity exhaustion.** 668,048 bytes were + free, more than the usual 457,392 needed, but the attempt died partway through (peak + 3,876,624, only 350,368 bytes into itself), with individual 128-136 byte word-registration + allocations failing. 662 `sf_free()` calls happened *during* that single failed attempt — + background heartbeat/physics activity from the 6 already-live VMs churning the same arena + concurrently. The allocator's free-list is documented as first-fit with **no splitting or + coalescing**, so fragmentation from that concurrent churn is the likely proximate cause, not + raw capacity. + +**The bug found and FIXED here:** the failure was *leaking* memory, permanently, on top of the +fragmentation problem. `capsule_wirebind_try_attach()` births its two VMs (console, then user) +in sequence; when the second failed, the console VM that had *already succeeded* was never torn +down — an orphan nothing ever points a real user at, since `WIREBIND` only ever hands the caller +the user VM's id. Confirmed live: after the failed attempt, `used_bytes` settled at 3,754,832 — +228,576 bytes *above* the pre-attempt baseline, i.e. one whole console VM's worth, permanently +gone. Every failed attach was making the arena strictly worse for the next attempt, not neutral. + +Fix: `capsule_vm_kill(username)` on the now-orphaned console VM before returning from the +failure path in `capsule_wirebind.c` — the same teardown `capsule_wirebind_eject()`/ +`capsule_wirebind_unclean_detach()` already use elsewhere in that file (`vm_cleanup()` + +`sf_free()`, confirmed live in §IX.2/§IX.3 to actually reclaim per-word dictionary allocations). +Verified live with the same probe, replaying the identical failure: `used_bytes` after the fix +returns to **exactly** the pre-attempt baseline (3,526,256, matching precisely) instead of +leaking 228,576 bytes. Three-arch acceptance passed with the fix included. + +**Still open — the actual architecture question, not yet decided (Captain Bob, 2026-09-07):** +"identity is everything in this OS." The leak fix makes retries neutral instead of compounding, +but the underlying ~6-identity practical ceiling (itself a fragmentation artifact, not a hard +capacity number) is unaddressed. Options on the table, none chosen yet: +- Grow the heap / make the arena size dynamic instead of a fixed 4 MiB compile-time constant. +- Shrink per-VM dictionaries — every `WIREBIND` user VM currently carries the full word set, + including things like `TTF-TEXT`/`SCROLL-BACK`/`KBD-SCAN` a non-console user session may not + need. +- Address the fragmentation contributor directly (splitting/coalescing in the free-list, or + serializing VM-birth allocation against concurrent background allocation) — this is *why* + identity 05 failed even with more free bytes than it needed. + +**Also flagged, not touched (not asked):** `Makefile.starkernel`'s +`printf '$(KERNEL_ARGS)\n' > starforth.cfg` breaks when `KERNEL_ARGS` starts with `--` (dash's +`printf` misreads it as an option). Worked around live during X.2's debug-level testing with a +harmless prefix token (`KERNEL_ARGS="x --log-level=debug"`); not fixed in the Makefile itself.