diff --git a/FABRIC-3.md b/FABRIC-3.md index daba9c8e..db6d02b6 100644 --- a/FABRIC-3.md +++ b/FABRIC-3.md @@ -5132,3 +5132,330 @@ trial cost -- still matches the project's "rule of 3's" DoE convention exactly ( reps as the ACL-RWT campaign). 6 cfgs x 30 reps = 180 main-block trials, roughly half the prior 360-trial estimate. +## XXXIV. Relaunch attempts surface a real `THRU` line-swallowing defect; a cross-session QEMU +collision; and a ratified pivot to a segmented per-ISA campaign design (2026-09-16/17) + +### XXXIV.1 -- Three silent relaunch failures, two false leads chased and ruled out, one real +bug found live + +Relaunching the 180-trial campaign (§XXXIII.7's own next step) via `SK_STARTUP_FORTH` +boot-injection (`make ... SK_CMD='...'`) failed silently three times in a row before the real +cause was found -- each attempt is recorded here because two of the three intermediate +theories were wrong and worth not re-chasing: + +1. **First attempt** -- `SK_CMD='S" multiuser-doe.4th" EXEC 12345 30 MU-EXEC-CAMPAIGN BYE'`. + Failed to *compile*, not a runtime bug: `Makefile.starkernel`'s own + `-DSK_STARTUP_FORTH='"$(SK_CMD)"'` wraps the value in C string-literal quotes, and the + embedded `S" ... "` FORTH syntax's own literal `"` characters terminate that C string early + -- `expected ')' before 'multiuser'`. Fixed by using `THRU` (block-numbered load, no quote + characters) instead of `S" name" EXEC` for boot-time injection specifically. +2. **Second attempt** -- `SK_CMD='5044 5055 THRU 12345 30 MU-EXEC-CAMPAIGN BYE'`. Compiled and + booted cleanly, but produced **zero** `MU-TRIAL` markers, no error, and no `BYE`-triggered + halt -- fell straight through to the normal interactive `ok>` prompt after a burst of + `vm_dictionary_track_entry: out of stable IDs` warnings far too small to represent 180 real + trials. **False lead #1, offered by Captain Bob and initially accepted:** host memory + pressure (CLion still open) corrupting the run. Ruled out empirically -- killed the instance, + confirmed `free -h` showed 9.1Gi available, reran the *identical* command, and it reproduced + the *identical* silent-failure symptom byte-for-byte. Memory was never the cause. +3. **Third attempt, after the memory theory was ruled out** -- **false lead #2**, this session's + own: hypothesized `SK_STARTUP_FORTH` runs before Zuse/WIREBIND authentication completes (true + -- confirmed by log ordering, the injection line prints well before `Zuse: identity confirmed + from attached thumbdrive`), and that some ACL-gated primitive was failing closed pre-auth. + Added a temporary `S" multiuser-doe.4th" EXEC` line to `capsules/init.4th` Block 2049 (loading + the capsule the same way `ACL.4th`/`lib.4th`/etc. already do, before Hera's own console exists + at all) to sidestep needing `THRU` in `SK_CMD` entirely -- rebuilt, reran with + `SK_CMD='12345 30 MU-EXEC-CAMPAIGN BYE'`. **Also silently produced nothing.** Captain Bob + corrected the approach directly: *"you should not have to edit anything, it ran before qemu + croaked"* -- i.e. the `THRU`-based approach was never actually wrong, and the pre-auth-timing + theory was a distraction. Reverted the `init.4th` edit immediately (`capsules/init.4th` is + back to its committed state, no trace of the probe left behind). + +**Root cause, found live, empirically, not guessed** -- rebuilt with `init.4th` reverted and the +original `THRU`-based `SK_CMD`, reproduced the silent failure a third time, then interacted with +the still-alive (silently-failed) instance directly over its QEMU serial chardev socket (`socat - +UNIX-CONNECT:build/amd64/kernel/qemu-serial-amd64-*.sock`) rather than rebuilding again: +- Typed `MU-EXEC-CAMPAIGN` bare (no args) at the live console -- it **was** defined and reachable + (hit the expected `DUP: Stack underflow`, correctly recovered per §XXXII.1's own fix: `VM fault + -- session recovered, resuming`). This proved `THRU` had genuinely loaded the capsule + correctly, on every one of the three attempts -- the capsule load was never the problem. +- Typed `12345 30 MU-EXEC-CAMPAIGN` as its **own separate line** immediately after -- it actually + ran this time: real per-tick `[HADES][DOE]` CSV rows started streaming for a live, newly-born + worker VM. + +**The actual defect: `THRU` does not resume interpretation of the remainder of its own input +line afterward.** In all three prior attempts, `5044 5055 THRU` and `12345 30 MU-EXEC-CAMPAIGN +BYE` were packed onto one line (either via `SK_STARTUP_FORTH`'s single `vm_interpret()` call, or +via one typed terminal line) -- and everything after `THRU` on that same line was silently +discarded once block-loading returned, never interpreted at all. No error, because nothing ever +tried to interpret it. This is a genuine, previously-undocumented gap distinct from anything +§XXXII's error-handling audit covered (that audit was scoped to `vm->error` propagation, not to +input-buffer resumption across a `LOAD`/`THRU` boundary) -- **confirmed behaviorally, not yet +traced to a specific line in `block_words.c`'s `block_word_thru()`/`block_word_load()`; the exact +C-level mechanism remains open for whoever picks this up next.** Practical consequence: the +`SK_STARTUP_FORTH` single-string boot-injection mechanism **cannot** drive a `THRU`-then-call +sequence at all until/unless this is fixed at the source -- the interactive two-line workaround +(typed live over the serial socket) sidesteps the bug, it does not fix it. Flagged here as a +real, reproducible defect worth its own fix later, out of scope for this section. + +### XXXIV.2 -- A cross-session QEMU collision killed the first real relaunch at 0 trials + +Once running for real (via the two-line workaround), the campaign was silently killed a fourth +time -- this one **not** self-inflicted. Another Claude Code session on the same machine +(peer session "Critical component") independently ran its own `make -f Makefile.starkernel +ARCH=amd64 clean qemu` at 20:57, in direct violation of this repo's own hard rule (only one QEMU +instance at a time, `accel=tcg` timing corrupted by concurrency). That session's own account, +given directly when pinged (`SendMessage`/cross-session): it found an "already-running +qemu-system-x86_64 process with ambiguous state," misjudged it as an orphan from its own earlier +work, and quit it via QMP before relaunching -- actually killing this session's live campaign +(seed 12345) at **zero completed trials**, log cut off mid-row with no clean shutdown banner. +That peer session's own working memory already carries a general warning against exactly this +class of mistake (never assume a live qemu process is an orphan without checking its cmdline/log +content first) -- flagged here as a second, independent confirmation that the warning is +correctly calibrated, not that it was followed carefully enough this time. + +The peer session then launched its **own** run of the identical experiment -- same seed 12345, +same `multiuser-doe.4th`, same persisted `disk/artemis.img` state -- at 21:11, confirmed +genuinely identical (not coincidentally similar) by matching `vm_id` hashes between the two logs' +worker births (e.g. `0a6783827b094e34-aed67f1ae0840697` appears in both). Rather than relaunch a +third time and risk a second collision, both sessions' users coordinated via cross-session +messaging and agreed: let the peer's run stand in, this session switches to monitoring its live +log instead of running its own. + +### XXXIV.3 -- Real per-cell cost measured live: the ratified 30-reps/cell design is a ~10.3-day +campaign, not hours + +After 8 of the peer session's 180 shuffled trials completed (`fail=0` on all), real per-cell cost +was measured directly from the live log using line-count as a work proxy -- cross-validated +against worker count before trusting it (`cfg=4`, 8-worker uniform, costs almost exactly 2x +`cfg=2`, 4-worker uniform, at the same workload -- scales sensibly, not noise): + +| cfg | concurrency | mode | measured cost (lines/trial) | +|---|---|---|---| +| 1 | 2 | heterogeneous | ~146K (lightest sampled) | +| 2 | 4 | uniform | ~256K | +| 3 | 4 | heterogeneous | ~148K (single noisy sample) | +| 4 | 8 | uniform | ~513K (2 consistent samples) | +| 5 | 8 | heterogeneous | ~755K (heaviest -- 2 consistent samples) | +| 0 | 2 | uniform | not sampled | + +One full pass of all 6 cells once each ≈ 1.96M lines ≈ 8.28 hours at the measured live rate +(~65.9 lines/sec). **The ratified §XXXIII.7 design (30 reps/cell, 180 trials total) extrapolates +to ≈248 hours (~10.3 days) end-to-end** -- dramatically longer than the ~20-minute +worst-single-cell estimate that originally justified 30 reps/cell in that section. This is a +load-bearing correction to §XXXIII.7's own sizing, not a new finding about the workloads +themselves (the lite-workload fix from that section is unaffected and still correct) -- the +30-rep *count*, not the per-trial cost, is what turned out to be far more expensive in aggregate +than anticipated. + +### XXXIV.4 -- Ratified pivot: a segmented per-ISA design, one seed per ISA, targeting ~24h/ISA + +Raised in conversation as "a Latin square modified for ISA, DoE per ISA" -- a way to further +segment the experiment rather than run one enormous single-ISA campaign. Follows this project's +own ACL-RWT precedent (a 3-seed x 3-ISA grid) but modified: **swap the seed dimension for the ISA +dimension rather than crossing both** -- one seed per ISA (not 3 seeds x 3 ISAs), full 6-cell +factorial preserved per ISA (no loss of granularity), reps/cell solved directly against a +concrete wall-clock target rather than picked arbitrarily. + +**Solved from the measured amd64 rate above:** a 24h/ISA budget ≈ 5.69M lines/ISA; one full pass +of the 6 cells ≈ 1.96M lines; 5.69M / 1.96M ≈ 2.9 passes. **N=3 reps/cell lands almost exactly on +target** (3 x 1.96M ≈ 5.89M lines ≈ ~24.8h/ISA) -- still matches this project's own "rule of +3's" DoE convention exactly, same as the ACL-RWT campaign's 3 seeds and this campaign's own +30-to-3 reduction lineage. + +**Ratified shape:** +- 6 cells x 3 reps = 18 trials/ISA +- ~24.8h/ISA (amd64-calibrated) +- x 3 ISAs, sequential (this repo's own one-QEMU-instance-at-a-time hard rule) ≈ ~74.5h + (~3.1 days) total -- versus ~10.3 days for the single-ISA/30-rep design, while adding real + cross-ISA coverage the campaign has never had. + +**Explicitly flagged as an assumption, not yet verified:** the ~24.8h/ISA figure is calibrated +only against amd64's own measured rate. aarch64/riscv64 per-trial cost for this specific +workload is unmeasured -- could run faster or slower under TCG, shifting the real per-ISA time in +either direction once actually run. **N=2 reps/cell** (~16.6h/ISA, ~49.7h/~2.1 days total) was +offered as a safety-margin alternative if the amd64-calibrated assumption turns out wrong on the +expensive side. **Nothing built yet -- design only, ratified in conversation, not yet +implemented as a driver capsule or launched on any architecture.** + +### XXXIV.5 -- The live amd64/30-rep campaign stopped by direct instruction; data and method +both preserved intact + +Given the §XXXIV.3 finding and the §XXXIV.4 pivot, Captain Bob directed stopping the live +peer-session campaign rather than letting its original ~10.3-day course run out, with an explicit +constraint: **do not destroy the campaign, preserve logs and data, do not lose the method.** + +Stopped via `SIGTERM` to the `qemu-system-x86_64` process itself (pid, not the wrapper `make`/ +`sh` script around it) -- this lets the Makefile's own post-exit cleanup sequence (kill the +`tail -f` helper, remove the QMP/serial sockets, run `scripts/extract_doe.sh`) execute exactly as +it would on a natural exit, rather than skipping straight past it with a harder kill on the +wrapper. Confirmed intact afterward, nothing lost: +- **Serial log**, full and unaffected: `logs/20260916-211104/amd64/qemu-amd64-20260916-211104.log` + (3.3M+ lines) +- **Extracted CSV**, produced by the normal shutdown path exactly as designed: + `experiments/bare_metal/runs/doe-amd64-20260916-211104.csv` (516MB, 3,301,732 data rows) +- **8 trials completed**, all `fail=0`, covering cells cfg=1/2/3/4/5 (cfg=0 never reached before + the stop) +- Sockets cleaned up normally; `multiuser-doe.4th` and every primitive built for it + (`WORKER-BIRTH`, `VM-ERROR?`, `MU-EXEC-CAMPAIGN`, etc. -- §XXXIII.4-6) untouched in the repo + +The peer session was notified of the stop and the reasoning directly (cross-session message), +including the exact preserved-file paths, so its own user isn't surprised by a QEMU process +disappearing without explanation. + +**Genuinely open, not decided here:** +- The per-ISA campaign itself is not yet built or launched on any architecture -- §XXXIV.4 is + design-only. +- `THRU`'s own line-swallowing defect (§XXXIV.1) is not root-caused at the C level, only worked + around interactively -- boot-time (`SK_STARTUP_FORTH`) injection of any `THRU`-then-call + sequence remains broken until it is. +- Whether/how the two partial amd64 runs' data (0 trials from the first collision, 8 trials from + the stopped second run) factor into or get discarded against the new per-ISA design is + undecided. + +## XXXV. Block-backed DoE persistence + shorter per-ISA campaign shape -- design only, no code +yet, ratified in conversation (2026-09-17) + +### XXXV.1 -- Why the current tooling cannot survive bare-metal boot + +Every existing DoE-extraction path (`scripts/extract_doe.sh`, `tail -f` on the QEMU serial log, +`socat`-based live interaction with the serial chardev socket) depends on a **host** sitting +outside the guest, scraping its serial stream in real time. That host does not exist on real +bare-metal hardware -- there is no QEMU wrapper, no serial chardev socket, no `tail -f`. Anything +the campaign needs to survive past its own run must be written to persistent storage **by the +guest itself**, using only mechanisms that work identically whether the "disk" is `virtio_blk.c` +(QEMU) or the real USB/NVMe path (`blkio_usb.c`, bare metal). `doe_log.c` today does neither -- +every row goes to `console_puts()` only (confirmed reading the file live this session), which is +serial-only by construction. + +### XXXV.2 -- Drive-size blocker found first, before any format design: `disk/artemis.img` is +30 MiB, three orders of magnitude too small + +Checked `disk/artemis.img` directly this session: **31,457,280 bytes (30 MiB), measured, not +assumed.** Cross-checked against the one real DoE volume this project has actually produced -- +§XXXIV.5's preserved artifact, `experiments/bare_metal/runs/doe-amd64-20260916-211104.csv`, is +**516 MB for 8 trials** (raw per-tick rows, `HB-ON` bracketing the whole campaign in +`multiuser-doe.4th` Block 5055). Raw per-tick mirroring to block storage is not viable at +**any** device size in this project's normal range -- the 18-trial/ISA design (§XXXIV.4) would +extrapolate to roughly **1.1 GB/ISA** of raw per-tick data if captured the same way, and even the +current 30 MiB image is short of one single trial's worth (516 MB / 8 ≈ 64.5 MB/trial) by more +than 2x. **This has to be fixed before any writer design matters**, and it is a two-part fix, not +one: + +1. **Grow `disk/artemis.img` itself.** Recommend resizing to **256 MiB** -- a real SSD/USB drive + on actual bare-metal hardware will dwarf this trivially, so there is no reason to size the dev + image tightly; 256 MiB gives comfortable headroom over the summary-only budget computed below + (§XXXV.4) even before any tick-sampling extension, without gambling on a number that turns out + short mid-campaign. **Not done yet -- flagging the concrete risk found while scoping this, + not fixing it, per this session's own instruction not to apply unrequested fixes:** growing the + raw file (e.g. `truncate -s 256M disk/artemis.img`, appending zero bytes at the end) changes + `total_devblocks`, and the entire top-of-device system-metadata fence is addressed + **backward from the end of the device** (`block_subsystem.c`: `devblock_idx = total_devblocks + - 1 - devblock_from_top`). A naive resize shifts every existing fence consumer -- Zuse's + genesis marker, the eligibility list, `artemis_sig_t`, the log-persistence ring -- to point at + different physical bytes than what's actually there, silently losing or corrupting the 8 + already-minted thumbdrive identities and everything else live on this image + (`project_thumbdrive_identities_minted_20260906.md`). Two real options, genuinely open, not + decided here: (a) write a proper migration -- read the fence's live content at the current + 30 MiB geometry, then rewrite it at the correct offsets for the new 256 MiB geometry, before + ever letting anything else touch the grown file; or (b) accept identity loss and re-mint fresh + on a newly-created blank 256 MiB image, since this is presently a dev/QEMU test artifact, not + a production one -- Captain Bob's call, not assumed here either way. +2. **Confirm `disk/artemis.img` is genuinely shared across all three ISAs' sequential QEMU + invocations** (same `$(ARTDISK)` path referenced identically for the amd64/aarch64/riscv64 + `qemu` targets in `Makefile.starkernel`, lines ~848/931/1021/1152) -- meaning three ISAs' + worth of campaign data will land on the **same physical device**, one after another, not three + separate images. The persistence format below must therefore carry an explicit ISA tag per + record (or per-ISA base-offset segmentation) so a later pass can tell the three campaigns' + data apart on one shared disk; this was not a concern for the single-ISA/30-rep design this + pivoted away from. + +### XXXV.3 -- Persistence format: reuse `log_region.c`'s own pattern, don't invent a new one + +`log_region.c`/`log_region.h` (FABRIC-3.md §XXVI follow-on, Step 4) already solved almost exactly +this problem for a smaller, different payload (per-VM log messages): a fixed-size binary slot per +record, a control header (magic/version/CRC, growable devblock count, head/tail/record_count), +addressed via `blk_meta_zone_read()`/`blk_meta_zone_write()` at a reserved `devblock_from_top` +base, living in the same top-of-device system-metadata fence Zuse's genesis marker and +`artemis_sig_t` already use. Proposed reuse, not a new mechanism: + +- **New, separate region** (own control header, own `devblock_from_top` base past the existing + ceiling -- `log_region.h` currently reserves up to 96; this region starts at 97+), not an + extension of `log_region.c` itself -- DoE data has a completely different shape (structured + numeric columns, not free-text messages) and a different, much larger growth ceiling than the + 32-devblock (128 KiB) cap `LOG_REGION_MAX_DEVBLOCKS` chose for log messages. Keeping it separate + also means a runaway DoE region can never crowd out real log persistence or Zuse identity data, + the same isolation discipline `log_region.h`'s own doc comment already argues for. +- **Binary slots, not ASCII CSV**, same reasoning `log_slot_t` already applies: the freestanding + `snprintf` on this target has known gaps (no `%f`, and `doe_log.c` itself already hand-rolls hex + formatting to work around a silent-substitution bug found live 2026-09-16), and binary rows are + meaningfully smaller than the current ASCII CSV row (today's row is one `%s`-joined line of ~29 + fields, several hundred bytes as text; the same fields packed as fixed-width binary are a small + fraction of that). CSV reconstruction happens **host-side, after the run**, as a small + extraction tool reading the region back off the device -- exactly mirroring what + `scripts/extract_doe.sh` already does for serial logs today, just pointed at a block region + instead of a log file. This is the one piece that's legitimately allowed to be host/QEMU-only, + because it runs after the campaign, not during it; on real bare-metal hardware the equivalent + step is "pull the drive, read it with a small standalone reader," not "tail a live serial + stream" -- which is the actual point of self-containment this whole redesign is for. +- **Self-driving from within the campaign capsule.** `multiuser-doe.4th`'s own + `MU-EMIT-TRIAL-MARKER` (Block 5051) already exists as a console-only per-trial summary line + (`run=`/`cfg=`/`rep=`/`nw=`/`mode=`/`fail=`). The natural, minimal extension is a new kernel + primitive (e.g. `DOE-PERSIST-TRIAL`, mirroring `LOG-APPEND`'s thin-wrapper-over-a-kernel-write + pattern in `capsules/artemis/init.4th` Block 4860) called right alongside it, writing one binary + slot per trial directly to the new region -- no host process needs to be watching for this data + to survive. + +### XXXV.4 -- What actually gets persisted: trial summaries, not a raw tick mirror + +Given the 30 MiB (soon 256 MiB) device is 2-3 orders of magnitude smaller than raw per-tick data +at any realistic campaign size, raw per-tick mirroring to block storage is ruled out entirely, not +just decimated: + +- **v1, primary target: one record per trial**, built from exactly the fields + `MU-EMIT-TRIAL-MARKER` already computes (run id, cfg, rep, nw, mode, fail count) plus a handful + of trial-end aggregate physics reads already available at that point (e.g. final + `fleet_k_q48`/`fleet_conserved`, cumulative `switch_count`). At 18 trials/ISA x 3 ISAs = 54 + records total, this is trivially small (well under 1 MiB even generously sized per record) -- + no tuning, no risk of overrunning even the current 30 MiB device, let alone 256 MiB. This alone + is enough to reconstruct the whole campaign's pass/fail and top-line physics-conservation + picture with zero dependency on any live serial capture. +- **v2, optional, explicitly deferred:** decimated sub-trial sampling (e.g. one row every N + heartbeat ticks instead of every tick) for anyone who wants intra-trial resolution later. Sized + against whatever's left of the device budget after v1 and whatever migration/resize path + (§XXXV.2) is chosen -- rough order of magnitude, at 256 MiB total and reserving generous + headroom for existing fence consumers, a decimation factor in the range of several dozen ticks + per sample would be needed to fit an 18-trial/ISA x 3-ISA campaign at anything like the + §XXXIV.5-measured per-trial tick volume; **not sized precisely here, left for whoever + implements v2, and not a blocker for v1 shipping first.** +- Both, if v2 is ever built, share the same fixed-slot/control-header shape as §XXXV.3, in + **separate** regions so a v2 overrun can never threaten v1's trial-summary data. + +### XXXV.5 -- Shorter-running design: prefer N=2 reps/cell over N=3 given the new IO overhead + +§XXXIV.4 already ratified a segmented per-ISA design (6 cells, one seed/ISA) and left N=2 vs N=3 +reps/cell as an explicitly open safety-margin choice (~16.6h/ISA at N=2 vs ~24.8h/ISA at N=3, +amd64-calibrated). Recommend **N=2 (12 trials/ISA, ~49.7h/~2.1 days total)** as the primary +target now, not N=3, for a reason specific to this section's own design: even v1's trial-summary +writes add real block-IO overhead (`blk_meta_zone_write()` read-modify-write per record) on top of +the pure-serial-log baseline §XXXIV.3's cost measurement was calibrated from, and aarch64/riscv64 +per-trial cost is still unmeasured extrapolation risk (flagged, unresolved, in §XXXIV.4 itself). +N=2 keeps the full 6-cell factorial (no loss of design granularity, only replication depth) while +leaving more margin against both unknowns landing on the expensive side at once. N=3 stays +available as a documented follow-up top-up pass later if N=2's data looks clean and time allows, +rather than committing to the larger N up front. + +### XXXV.6 -- Genuinely open, not decided here + +- **Which drive-resize path** (§XXXV.2: proper fence migration vs. accept identity loss and + re-mint fresh) -- Captain Bob's call before any code is written. +- **Exact v2 decimation factor**, if/when v2 is built at all (§XXXV.4). +- Whether `THRU`'s own line-swallowing defect (§XXXIV.1, still un-root-caused at the C level) gets + fixed before or after this work -- it blocks true single-command `SK_STARTUP_FORTH` autolaunch + of the campaign regardless of persistence format; the two-line interactive workaround still + applies either way. +- Disposal of the two stale partial amd64 runs' data (§XXXIV.5's own still-open item) is unrelated + to this pivot and remains undecided. +- **Nothing built yet** -- this section is design only, ratified in conversation, no capsule/C + code written or committed for either the drive resize, the persistence region, or the N=2 + campaign shape. +