FABRIC-3.md §XXXV: block-backed DoE persistence design + N=2 per-ISA shape (design only, no code yet)
Build / build-amd64-iso (push) Waiting to run
Build / build-aarch64-iso (push) Waiting to run
Build / build-riscv64-img (push) Waiting to run

Scopes the self-contained (no host serial capture) persistence needed
before the per-ISA campaign driver can run on real bare-metal hardware.
Found disk/artemis.img is only 30 MiB (measured) against a 516 MB/8-trial
raw per-tick baseline -- raw mirroring is ruled out at any realistic
device size. Proposes reusing log_region.c's binary-slot/control-header
pattern for a new trial-summary-only region, flags the fence-addressing
risk in growing the image, and recommends N=2 reps/cell over N=3 given
the added block-IO overhead. Nothing built -- ratified in conversation only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
2026-09-17 11:25:57 -04:00
co-authored by Claude Sonnet 5
parent 94215e8b47
commit a83cbe016f
+327
View File
@@ -5132,3 +5132,330 @@ trial cost -- still matches the project's "rule of 3's" DoE convention exactly (
reps as the ACL-RWT campaign). 6 cfgs x 30 reps = 180 main-block trials, roughly half the
prior 360-trial estimate.
## XXXIV. Relaunch attempts surface a real `THRU` line-swallowing defect; a cross-session QEMU
collision; and a ratified pivot to a segmented per-ISA campaign design (2026-09-16/17)
### XXXIV.1 -- Three silent relaunch failures, two false leads chased and ruled out, one real
bug found live
Relaunching the 180-trial campaign (§XXXIII.7's own next step) via `SK_STARTUP_FORTH`
boot-injection (`make ... SK_CMD='...'`) failed silently three times in a row before the real
cause was found -- each attempt is recorded here because two of the three intermediate
theories were wrong and worth not re-chasing:
1. **First attempt** -- `SK_CMD='S" multiuser-doe.4th" EXEC 12345 30 MU-EXEC-CAMPAIGN BYE'`.
Failed to *compile*, not a runtime bug: `Makefile.starkernel`'s own
`-DSK_STARTUP_FORTH='"$(SK_CMD)"'` wraps the value in C string-literal quotes, and the
embedded `S" ... "` FORTH syntax's own literal `"` characters terminate that C string early
-- `expected ')' before 'multiuser'`. Fixed by using `THRU` (block-numbered load, no quote
characters) instead of `S" name" EXEC` for boot-time injection specifically.
2. **Second attempt** -- `SK_CMD='5044 5055 THRU 12345 30 MU-EXEC-CAMPAIGN BYE'`. Compiled and
booted cleanly, but produced **zero** `MU-TRIAL` markers, no error, and no `BYE`-triggered
halt -- fell straight through to the normal interactive `ok>` prompt after a burst of
`vm_dictionary_track_entry: out of stable IDs` warnings far too small to represent 180 real
trials. **False lead #1, offered by Captain Bob and initially accepted:** host memory
pressure (CLion still open) corrupting the run. Ruled out empirically -- killed the instance,
confirmed `free -h` showed 9.1Gi available, reran the *identical* command, and it reproduced
the *identical* silent-failure symptom byte-for-byte. Memory was never the cause.
3. **Third attempt, after the memory theory was ruled out** -- **false lead #2**, this session's
own: hypothesized `SK_STARTUP_FORTH` runs before Zuse/WIREBIND authentication completes (true
-- confirmed by log ordering, the injection line prints well before `Zuse: identity confirmed
from attached thumbdrive`), and that some ACL-gated primitive was failing closed pre-auth.
Added a temporary `S" multiuser-doe.4th" EXEC` line to `capsules/init.4th` Block 2049 (loading
the capsule the same way `ACL.4th`/`lib.4th`/etc. already do, before Hera's own console exists
at all) to sidestep needing `THRU` in `SK_CMD` entirely -- rebuilt, reran with
`SK_CMD='12345 30 MU-EXEC-CAMPAIGN BYE'`. **Also silently produced nothing.** Captain Bob
corrected the approach directly: *"you should not have to edit anything, it ran before qemu
croaked"* -- i.e. the `THRU`-based approach was never actually wrong, and the pre-auth-timing
theory was a distraction. Reverted the `init.4th` edit immediately (`capsules/init.4th` is
back to its committed state, no trace of the probe left behind).
**Root cause, found live, empirically, not guessed** -- rebuilt with `init.4th` reverted and the
original `THRU`-based `SK_CMD`, reproduced the silent failure a third time, then interacted with
the still-alive (silently-failed) instance directly over its QEMU serial chardev socket (`socat -
UNIX-CONNECT:build/amd64/kernel/qemu-serial-amd64-*.sock`) rather than rebuilding again:
- Typed `MU-EXEC-CAMPAIGN` bare (no args) at the live console -- it **was** defined and reachable
(hit the expected `DUP: Stack underflow`, correctly recovered per §XXXII.1's own fix: `VM fault
-- session recovered, resuming`). This proved `THRU` had genuinely loaded the capsule
correctly, on every one of the three attempts -- the capsule load was never the problem.
- Typed `12345 30 MU-EXEC-CAMPAIGN` as its **own separate line** immediately after -- it actually
ran this time: real per-tick `[HADES][DOE]` CSV rows started streaming for a live, newly-born
worker VM.
**The actual defect: `THRU` does not resume interpretation of the remainder of its own input
line afterward.** In all three prior attempts, `5044 5055 THRU` and `12345 30 MU-EXEC-CAMPAIGN
BYE` were packed onto one line (either via `SK_STARTUP_FORTH`'s single `vm_interpret()` call, or
via one typed terminal line) -- and everything after `THRU` on that same line was silently
discarded once block-loading returned, never interpreted at all. No error, because nothing ever
tried to interpret it. This is a genuine, previously-undocumented gap distinct from anything
§XXXII's error-handling audit covered (that audit was scoped to `vm->error` propagation, not to
input-buffer resumption across a `LOAD`/`THRU` boundary) -- **confirmed behaviorally, not yet
traced to a specific line in `block_words.c`'s `block_word_thru()`/`block_word_load()`; the exact
C-level mechanism remains open for whoever picks this up next.** Practical consequence: the
`SK_STARTUP_FORTH` single-string boot-injection mechanism **cannot** drive a `THRU`-then-call
sequence at all until/unless this is fixed at the source -- the interactive two-line workaround
(typed live over the serial socket) sidesteps the bug, it does not fix it. Flagged here as a
real, reproducible defect worth its own fix later, out of scope for this section.
### XXXIV.2 -- A cross-session QEMU collision killed the first real relaunch at 0 trials
Once running for real (via the two-line workaround), the campaign was silently killed a fourth
time -- this one **not** self-inflicted. Another Claude Code session on the same machine
(peer session "Critical component") independently ran its own `make -f Makefile.starkernel
ARCH=amd64 clean qemu` at 20:57, in direct violation of this repo's own hard rule (only one QEMU
instance at a time, `accel=tcg` timing corrupted by concurrency). That session's own account,
given directly when pinged (`SendMessage`/cross-session): it found an "already-running
qemu-system-x86_64 process with ambiguous state," misjudged it as an orphan from its own earlier
work, and quit it via QMP before relaunching -- actually killing this session's live campaign
(seed 12345) at **zero completed trials**, log cut off mid-row with no clean shutdown banner.
That peer session's own working memory already carries a general warning against exactly this
class of mistake (never assume a live qemu process is an orphan without checking its cmdline/log
content first) -- flagged here as a second, independent confirmation that the warning is
correctly calibrated, not that it was followed carefully enough this time.
The peer session then launched its **own** run of the identical experiment -- same seed 12345,
same `multiuser-doe.4th`, same persisted `disk/artemis.img` state -- at 21:11, confirmed
genuinely identical (not coincidentally similar) by matching `vm_id` hashes between the two logs'
worker births (e.g. `0a6783827b094e34-aed67f1ae0840697` appears in both). Rather than relaunch a
third time and risk a second collision, both sessions' users coordinated via cross-session
messaging and agreed: let the peer's run stand in, this session switches to monitoring its live
log instead of running its own.
### XXXIV.3 -- Real per-cell cost measured live: the ratified 30-reps/cell design is a ~10.3-day
campaign, not hours
After 8 of the peer session's 180 shuffled trials completed (`fail=0` on all), real per-cell cost
was measured directly from the live log using line-count as a work proxy -- cross-validated
against worker count before trusting it (`cfg=4`, 8-worker uniform, costs almost exactly 2x
`cfg=2`, 4-worker uniform, at the same workload -- scales sensibly, not noise):
| cfg | concurrency | mode | measured cost (lines/trial) |
|---|---|---|---|
| 1 | 2 | heterogeneous | ~146K (lightest sampled) |
| 2 | 4 | uniform | ~256K |
| 3 | 4 | heterogeneous | ~148K (single noisy sample) |
| 4 | 8 | uniform | ~513K (2 consistent samples) |
| 5 | 8 | heterogeneous | ~755K (heaviest -- 2 consistent samples) |
| 0 | 2 | uniform | not sampled |
One full pass of all 6 cells once each ≈ 1.96M lines ≈ 8.28 hours at the measured live rate
(~65.9 lines/sec). **The ratified §XXXIII.7 design (30 reps/cell, 180 trials total) extrapolates
to ≈248 hours (~10.3 days) end-to-end** -- dramatically longer than the ~20-minute
worst-single-cell estimate that originally justified 30 reps/cell in that section. This is a
load-bearing correction to §XXXIII.7's own sizing, not a new finding about the workloads
themselves (the lite-workload fix from that section is unaffected and still correct) -- the
30-rep *count*, not the per-trial cost, is what turned out to be far more expensive in aggregate
than anticipated.
### XXXIV.4 -- Ratified pivot: a segmented per-ISA design, one seed per ISA, targeting ~24h/ISA
Raised in conversation as "a Latin square modified for ISA, DoE per ISA" -- a way to further
segment the experiment rather than run one enormous single-ISA campaign. Follows this project's
own ACL-RWT precedent (a 3-seed x 3-ISA grid) but modified: **swap the seed dimension for the ISA
dimension rather than crossing both** -- one seed per ISA (not 3 seeds x 3 ISAs), full 6-cell
factorial preserved per ISA (no loss of granularity), reps/cell solved directly against a
concrete wall-clock target rather than picked arbitrarily.
**Solved from the measured amd64 rate above:** a 24h/ISA budget ≈ 5.69M lines/ISA; one full pass
of the 6 cells ≈ 1.96M lines; 5.69M / 1.96M ≈ 2.9 passes. **N=3 reps/cell lands almost exactly on
target** (3 x 1.96M ≈ 5.89M lines ≈ ~24.8h/ISA) -- still matches this project's own "rule of
3's" DoE convention exactly, same as the ACL-RWT campaign's 3 seeds and this campaign's own
30-to-3 reduction lineage.
**Ratified shape:**
- 6 cells x 3 reps = 18 trials/ISA
- ~24.8h/ISA (amd64-calibrated)
- x 3 ISAs, sequential (this repo's own one-QEMU-instance-at-a-time hard rule) ≈ ~74.5h
(~3.1 days) total -- versus ~10.3 days for the single-ISA/30-rep design, while adding real
cross-ISA coverage the campaign has never had.
**Explicitly flagged as an assumption, not yet verified:** the ~24.8h/ISA figure is calibrated
only against amd64's own measured rate. aarch64/riscv64 per-trial cost for this specific
workload is unmeasured -- could run faster or slower under TCG, shifting the real per-ISA time in
either direction once actually run. **N=2 reps/cell** (~16.6h/ISA, ~49.7h/~2.1 days total) was
offered as a safety-margin alternative if the amd64-calibrated assumption turns out wrong on the
expensive side. **Nothing built yet -- design only, ratified in conversation, not yet
implemented as a driver capsule or launched on any architecture.**
### XXXIV.5 -- The live amd64/30-rep campaign stopped by direct instruction; data and method
both preserved intact
Given the §XXXIV.3 finding and the §XXXIV.4 pivot, Captain Bob directed stopping the live
peer-session campaign rather than letting its original ~10.3-day course run out, with an explicit
constraint: **do not destroy the campaign, preserve logs and data, do not lose the method.**
Stopped via `SIGTERM` to the `qemu-system-x86_64` process itself (pid, not the wrapper `make`/
`sh` script around it) -- this lets the Makefile's own post-exit cleanup sequence (kill the
`tail -f` helper, remove the QMP/serial sockets, run `scripts/extract_doe.sh`) execute exactly as
it would on a natural exit, rather than skipping straight past it with a harder kill on the
wrapper. Confirmed intact afterward, nothing lost:
- **Serial log**, full and unaffected: `logs/20260916-211104/amd64/qemu-amd64-20260916-211104.log`
(3.3M+ lines)
- **Extracted CSV**, produced by the normal shutdown path exactly as designed:
`experiments/bare_metal/runs/doe-amd64-20260916-211104.csv` (516MB, 3,301,732 data rows)
- **8 trials completed**, all `fail=0`, covering cells cfg=1/2/3/4/5 (cfg=0 never reached before
the stop)
- Sockets cleaned up normally; `multiuser-doe.4th` and every primitive built for it
(`WORKER-BIRTH`, `VM-ERROR?`, `MU-EXEC-CAMPAIGN`, etc. -- §XXXIII.4-6) untouched in the repo
The peer session was notified of the stop and the reasoning directly (cross-session message),
including the exact preserved-file paths, so its own user isn't surprised by a QEMU process
disappearing without explanation.
**Genuinely open, not decided here:**
- The per-ISA campaign itself is not yet built or launched on any architecture -- §XXXIV.4 is
design-only.
- `THRU`'s own line-swallowing defect (§XXXIV.1) is not root-caused at the C level, only worked
around interactively -- boot-time (`SK_STARTUP_FORTH`) injection of any `THRU`-then-call
sequence remains broken until it is.
- Whether/how the two partial amd64 runs' data (0 trials from the first collision, 8 trials from
the stopped second run) factor into or get discarded against the new per-ISA design is
undecided.
## XXXV. Block-backed DoE persistence + shorter per-ISA campaign shape -- design only, no code
yet, ratified in conversation (2026-09-17)
### XXXV.1 -- Why the current tooling cannot survive bare-metal boot
Every existing DoE-extraction path (`scripts/extract_doe.sh`, `tail -f` on the QEMU serial log,
`socat`-based live interaction with the serial chardev socket) depends on a **host** sitting
outside the guest, scraping its serial stream in real time. That host does not exist on real
bare-metal hardware -- there is no QEMU wrapper, no serial chardev socket, no `tail -f`. Anything
the campaign needs to survive past its own run must be written to persistent storage **by the
guest itself**, using only mechanisms that work identically whether the "disk" is `virtio_blk.c`
(QEMU) or the real USB/NVMe path (`blkio_usb.c`, bare metal). `doe_log.c` today does neither --
every row goes to `console_puts()` only (confirmed reading the file live this session), which is
serial-only by construction.
### XXXV.2 -- Drive-size blocker found first, before any format design: `disk/artemis.img` is
30 MiB, three orders of magnitude too small
Checked `disk/artemis.img` directly this session: **31,457,280 bytes (30 MiB), measured, not
assumed.** Cross-checked against the one real DoE volume this project has actually produced --
§XXXIV.5's preserved artifact, `experiments/bare_metal/runs/doe-amd64-20260916-211104.csv`, is
**516 MB for 8 trials** (raw per-tick rows, `HB-ON` bracketing the whole campaign in
`multiuser-doe.4th` Block 5055). Raw per-tick mirroring to block storage is not viable at
**any** device size in this project's normal range -- the 18-trial/ISA design (§XXXIV.4) would
extrapolate to roughly **1.1 GB/ISA** of raw per-tick data if captured the same way, and even the
current 30 MiB image is short of one single trial's worth (516 MB / 8 ≈ 64.5 MB/trial) by more
than 2x. **This has to be fixed before any writer design matters**, and it is a two-part fix, not
one:
1. **Grow `disk/artemis.img` itself.** Recommend resizing to **256 MiB** -- a real SSD/USB drive
on actual bare-metal hardware will dwarf this trivially, so there is no reason to size the dev
image tightly; 256 MiB gives comfortable headroom over the summary-only budget computed below
(§XXXV.4) even before any tick-sampling extension, without gambling on a number that turns out
short mid-campaign. **Not done yet -- flagging the concrete risk found while scoping this,
not fixing it, per this session's own instruction not to apply unrequested fixes:** growing the
raw file (e.g. `truncate -s 256M disk/artemis.img`, appending zero bytes at the end) changes
`total_devblocks`, and the entire top-of-device system-metadata fence is addressed
**backward from the end of the device** (`block_subsystem.c`: `devblock_idx = total_devblocks
- 1 - devblock_from_top`). A naive resize shifts every existing fence consumer -- Zuse's
genesis marker, the eligibility list, `artemis_sig_t`, the log-persistence ring -- to point at
different physical bytes than what's actually there, silently losing or corrupting the 8
already-minted thumbdrive identities and everything else live on this image
(`project_thumbdrive_identities_minted_20260906.md`). Two real options, genuinely open, not
decided here: (a) write a proper migration -- read the fence's live content at the current
30 MiB geometry, then rewrite it at the correct offsets for the new 256 MiB geometry, before
ever letting anything else touch the grown file; or (b) accept identity loss and re-mint fresh
on a newly-created blank 256 MiB image, since this is presently a dev/QEMU test artifact, not
a production one -- Captain Bob's call, not assumed here either way.
2. **Confirm `disk/artemis.img` is genuinely shared across all three ISAs' sequential QEMU
invocations** (same `$(ARTDISK)` path referenced identically for the amd64/aarch64/riscv64
`qemu` targets in `Makefile.starkernel`, lines ~848/931/1021/1152) -- meaning three ISAs'
worth of campaign data will land on the **same physical device**, one after another, not three
separate images. The persistence format below must therefore carry an explicit ISA tag per
record (or per-ISA base-offset segmentation) so a later pass can tell the three campaigns'
data apart on one shared disk; this was not a concern for the single-ISA/30-rep design this
pivoted away from.
### XXXV.3 -- Persistence format: reuse `log_region.c`'s own pattern, don't invent a new one
`log_region.c`/`log_region.h` (FABRIC-3.md §XXVI follow-on, Step 4) already solved almost exactly
this problem for a smaller, different payload (per-VM log messages): a fixed-size binary slot per
record, a control header (magic/version/CRC, growable devblock count, head/tail/record_count),
addressed via `blk_meta_zone_read()`/`blk_meta_zone_write()` at a reserved `devblock_from_top`
base, living in the same top-of-device system-metadata fence Zuse's genesis marker and
`artemis_sig_t` already use. Proposed reuse, not a new mechanism:
- **New, separate region** (own control header, own `devblock_from_top` base past the existing
ceiling -- `log_region.h` currently reserves up to 96; this region starts at 97+), not an
extension of `log_region.c` itself -- DoE data has a completely different shape (structured
numeric columns, not free-text messages) and a different, much larger growth ceiling than the
32-devblock (128 KiB) cap `LOG_REGION_MAX_DEVBLOCKS` chose for log messages. Keeping it separate
also means a runaway DoE region can never crowd out real log persistence or Zuse identity data,
the same isolation discipline `log_region.h`'s own doc comment already argues for.
- **Binary slots, not ASCII CSV**, same reasoning `log_slot_t` already applies: the freestanding
`snprintf` on this target has known gaps (no `%f`, and `doe_log.c` itself already hand-rolls hex
formatting to work around a silent-substitution bug found live 2026-09-16), and binary rows are
meaningfully smaller than the current ASCII CSV row (today's row is one `%s`-joined line of ~29
fields, several hundred bytes as text; the same fields packed as fixed-width binary are a small
fraction of that). CSV reconstruction happens **host-side, after the run**, as a small
extraction tool reading the region back off the device -- exactly mirroring what
`scripts/extract_doe.sh` already does for serial logs today, just pointed at a block region
instead of a log file. This is the one piece that's legitimately allowed to be host/QEMU-only,
because it runs after the campaign, not during it; on real bare-metal hardware the equivalent
step is "pull the drive, read it with a small standalone reader," not "tail a live serial
stream" -- which is the actual point of self-containment this whole redesign is for.
- **Self-driving from within the campaign capsule.** `multiuser-doe.4th`'s own
`MU-EMIT-TRIAL-MARKER` (Block 5051) already exists as a console-only per-trial summary line
(`run=`/`cfg=`/`rep=`/`nw=`/`mode=`/`fail=`). The natural, minimal extension is a new kernel
primitive (e.g. `DOE-PERSIST-TRIAL`, mirroring `LOG-APPEND`'s thin-wrapper-over-a-kernel-write
pattern in `capsules/artemis/init.4th` Block 4860) called right alongside it, writing one binary
slot per trial directly to the new region -- no host process needs to be watching for this data
to survive.
### XXXV.4 -- What actually gets persisted: trial summaries, not a raw tick mirror
Given the 30 MiB (soon 256 MiB) device is 2-3 orders of magnitude smaller than raw per-tick data
at any realistic campaign size, raw per-tick mirroring to block storage is ruled out entirely, not
just decimated:
- **v1, primary target: one record per trial**, built from exactly the fields
`MU-EMIT-TRIAL-MARKER` already computes (run id, cfg, rep, nw, mode, fail count) plus a handful
of trial-end aggregate physics reads already available at that point (e.g. final
`fleet_k_q48`/`fleet_conserved`, cumulative `switch_count`). At 18 trials/ISA x 3 ISAs = 54
records total, this is trivially small (well under 1 MiB even generously sized per record) --
no tuning, no risk of overrunning even the current 30 MiB device, let alone 256 MiB. This alone
is enough to reconstruct the whole campaign's pass/fail and top-line physics-conservation
picture with zero dependency on any live serial capture.
- **v2, optional, explicitly deferred:** decimated sub-trial sampling (e.g. one row every N
heartbeat ticks instead of every tick) for anyone who wants intra-trial resolution later. Sized
against whatever's left of the device budget after v1 and whatever migration/resize path
(§XXXV.2) is chosen -- rough order of magnitude, at 256 MiB total and reserving generous
headroom for existing fence consumers, a decimation factor in the range of several dozen ticks
per sample would be needed to fit an 18-trial/ISA x 3-ISA campaign at anything like the
§XXXIV.5-measured per-trial tick volume; **not sized precisely here, left for whoever
implements v2, and not a blocker for v1 shipping first.**
- Both, if v2 is ever built, share the same fixed-slot/control-header shape as §XXXV.3, in
**separate** regions so a v2 overrun can never threaten v1's trial-summary data.
### XXXV.5 -- Shorter-running design: prefer N=2 reps/cell over N=3 given the new IO overhead
§XXXIV.4 already ratified a segmented per-ISA design (6 cells, one seed/ISA) and left N=2 vs N=3
reps/cell as an explicitly open safety-margin choice (~16.6h/ISA at N=2 vs ~24.8h/ISA at N=3,
amd64-calibrated). Recommend **N=2 (12 trials/ISA, ~49.7h/~2.1 days total)** as the primary
target now, not N=3, for a reason specific to this section's own design: even v1's trial-summary
writes add real block-IO overhead (`blk_meta_zone_write()` read-modify-write per record) on top of
the pure-serial-log baseline §XXXIV.3's cost measurement was calibrated from, and aarch64/riscv64
per-trial cost is still unmeasured extrapolation risk (flagged, unresolved, in §XXXIV.4 itself).
N=2 keeps the full 6-cell factorial (no loss of design granularity, only replication depth) while
leaving more margin against both unknowns landing on the expensive side at once. N=3 stays
available as a documented follow-up top-up pass later if N=2's data looks clean and time allows,
rather than committing to the larger N up front.
### XXXV.6 -- Genuinely open, not decided here
- **Which drive-resize path** (§XXXV.2: proper fence migration vs. accept identity loss and
re-mint fresh) -- Captain Bob's call before any code is written.
- **Exact v2 decimation factor**, if/when v2 is built at all (§XXXV.4).
- Whether `THRU`'s own line-swallowing defect (§XXXIV.1, still un-root-caused at the C level) gets
fixed before or after this work -- it blocks true single-command `SK_STARTUP_FORTH` autolaunch
of the campaign regardless of persistence format; the two-line interactive workaround still
applies either way.
- Disposal of the two stale partial amd64 runs' data (§XXXIV.5's own still-open item) is unrelated
to this pivot and remains undecided.
- **Nothing built yet** -- this section is design only, ratified in conversation, no capsule/C
code written or committed for either the drive resize, the persistence region, or the N=2
campaign shape.