Fix two catastrophically heavy workloads found running the real campaign
Build / build-amd64-iso (push) Waiting to run
Build / build-aarch64-iso (push) Waiting to run
Build / build-riscv64-img (push) Waiting to run

The full 360-trial campaign was launched, then killed after 4h39m of
CPU time with zero trials completed -- still stuck on the very first
touch of the very first trial.

Root cause, quantified from the workload's own source, not estimated:
RUN-CHAOS5 (workload-5.4th) is 1000000 0 DO CHAOS-FIELD CHAOS-RIPPLE
LOOP, multiplying out to ~2.4 trillion word executions for one call --
~495 days at the observed rate. Never designed to be called to
completion as a single touch in a repeatedly-birthed worker.

A second problem found computing the fix rather than discovering it
mid-run again: workload-1.4th's own birth-time self-execution (20
SQUARE-WAVE + 1500 SQUARE-BURST + 100000x MICRO-BURST, all three run
automatically at capsule load) sums to ~875 million words -- ~4.3
hours just to birth one worker -- and worker index 1 always maps to
it in heterogeneous mode, landing in half the campaign's cells
regardless of concurrency level.

Fix: two new capsules, workload-1-lite.4th and workload-5-lite.4th,
carrying the same word bodies verbatim but a bounded self-execution
tail, comparable in scale to the campaign's other workloads. The
originals are untouched; only multiuser-doe.4th's own WL-CAPSULE/
WL-ENTRY index 1 and 5 mappings were repointed.

Verified live before relaunching a third time: the actual worst case
in isolation (5 0 998 MU-RUN-TRIAL, concurrency=8 heterogeneous,
includes both fixed workloads plus RUN-OMNI) completed in ~20 minutes
wall-clock, Hera stayed healthy throughout. Clean 3-arch qemu boot.

Reps reduced 60 -> 30/cell (Bob's call after seeing the real per-trial
cost) -- still matches the project's "rule of 3's" DoE convention
(same as ACL-RWT's own 30 reps). 6 cfgs x 30 reps = 180 trials.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BWpNjdwPtFLuVLaAq44L9K
This commit is contained in:
Robert Allan James
2026-09-16 15:47:30 -04:00
co-authored by Claude Sonnet 5
parent 29f91f9531
commit 94215e8b47
16 changed files with 2048842 additions and 18 deletions
+42
View File
@@ -5090,3 +5090,45 @@ workloads like `RUN-CHAOS5`'s 1,000,000 iterations) -- a genuinely long-running
TCG, deliberately not kicked off without explicit confirmation first. The WIREBIND-automation TCG, deliberately not kicked off without explicit confirmation first. The WIREBIND-automation
fixed arm also remains unbuilt. fixed arm also remains unbuilt.
### XXXIII.7 -- Two workloads too heavy to touch as-is, found live at real cost
**Approved and launched, then killed after finding a real problem the design review never
caught:** the full campaign ran for 4h39m of CPU time without completing even the first
touch of the first trial. Root cause quantified directly from the workload's own source
(`workload-5.4th`), not estimated: `RUN-CHAOS5` is `1000000 0 DO CHAOS-FIELD CHAOS-RIPPLE
LOOP`, and `CHAOS-FIELD` alone is `24 0 DO CHAOS-BLOCK LOOP` with `CHAOS-BLOCK` itself ~100K
word executions -- multiplying out to **~2.4 trillion word executions for one call**. At the
observed rate (~56,300 words/sec), that's roughly 495 days for a single touch. This workload
was never designed to be called to completion as one touch in a repeatedly-birthed-and-killed
worker; whatever its role in the original single-VM L8 DoE, it doesn't fit this campaign's
shape at all.
**A second, related problem found computing the fix, not by running it first:** `workload-
1.4th`'s own birth-time self-execution (`20 SQUARE-WAVE` + `1500 SQUARE-BURST` + `100000x
MICRO-BURST`, all three run automatically at capsule load) sums to roughly **875 million word
executions -- ~4.3 hours just to birth one worker**, and worker index 1 always maps to this
capsule in heterogeneous mode, so the cost lands in half the campaign's cells regardless of
concurrency level. Caught by computing the real per-workload word counts from every capsule's
own source before relaunching a third time, not by discovering it mid-run again.
**Fix, matching the pattern Captain Bob had already authorized ("workloads are not carved in
stone, create new extended workloads where needed"):** two new capsules,
`workload-5-lite.4th` and `workload-1-lite.4th`, carrying the *same* word bodies verbatim
(`CHAOS-BLOCK`/`CHAOS-FIELD`; `COMPUTE`/`BURST`/`MICRO-BURST`) but a bounded self-execution
tail (`CHAOS-FIELD` once; `1000x MICRO-BURST`) -- comparable in scale to the campaign's other
workloads (~2-30M words), not the original files' unbounded repeat counts. The originals
(`workload-1.4th`, `workload-5.4th`) are untouched, still real, still available for whatever
their original context needs -- only `multiuser-doe.4th`'s own `WL-CAPSULE`/`WL-ENTRY` index
1 and 5 mappings were repointed to the lite variants.
**Verified live before relaunching the full campaign a third time:** ran the actual worst
case in isolation -- `5 0 998 MU-RUN-TRIAL` (cfg=5: concurrency=8, heterogeneous, so it
includes both previously-catastrophic workloads plus `RUN-OMNI`'s own ~30M words) -- and it
completed in ~20 minutes wall-clock (`MU-TRIAL run=998 cfg=5 rep=0 nw=8 mode=1 fail=0`), Hera
stayed healthy throughout. Clean 3-arch qemu boot on the real committed capsule changes.
**Reps reduced from 60 to 30 per cell**, Captain Bob's own call after seeing the real per-
trial cost -- still matches the project's "rule of 3's" DoE convention exactly (the same 30
reps as the ACL-RWT campaign). 6 cfgs x 30 reps = 180 main-block trials, roughly half the
prior 360-trial estimate.
+21 -15
View File
@@ -1,5 +1,5 @@
# Capsule Block Manifest — Auto-generated # Capsule Block Manifest — Auto-generated
<!-- Generated by mkcapsule --manifest 2026-09-16T14:25:13Z --> <!-- Generated by mkcapsule --manifest 2026-09-16T19:45:24Z -->
<!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh. --> <!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh. -->
<!-- Hand-written justifications and immutability notes live --> <!-- Hand-written justifications and immutability notes live -->
<!-- in MANIFEST.md alongside this auto-generated index. --> <!-- in MANIFEST.md alongside this auto-generated index. -->
@@ -27,16 +27,18 @@
| `init-l8-volatile.4th` | 4810, 4811, 4812, 4813 | `0x98caabbbd92abac4` | yes | | `init-l8-volatile.4th` | 4810, 4811, 4812, 4813 | `0x98caabbbd92abac4` | yes |
| `init.4th` | 2049, 2050, 2057 | `0x1ef4939ed32ec1e6` | yes | | `init.4th` | 2049, 2050, 2057 | `0x1ef4939ed32ec1e6` | yes |
| `lib.4th` | 4050 | `0x4b216635c359ef73` | yes | | `lib.4th` | 4050 | `0x4b216635c359ef73` | yes |
| `multiuser-doe.4th` | 5044, 5045, 5046, 5047, 5048, 5049, 5050, 5051, 5052, 5053, 5054, 5055 | `0x1dc57cdf83afb6ce` | yes | | `multiuser-doe.4th` | 5044, 5045, 5046, 5047, 5048, 5049, 5050, 5051, 5052, 5053, 5054, 5055 | `0x22140588ee7c39e6` | yes |
| `process.4th` | 4300, 4301 | `0x781afc1dbd0294f7` | yes | | `process.4th` | 4300, 4301 | `0x781afc1dbd0294f7` | yes |
| `sdk.4th` | 5109, 5110, 5111, 5112, 5113, 5114, 5115 | `0x008fdbbb62c94a3a` | yes | | `sdk.4th` | 5109, 5110, 5111, 5112, 5113, 5114, 5115 | `0x008fdbbb62c94a3a` | yes |
| `turtle.4th` | 5100, 5101, 5102, 5103, 5104, 5105, 5106, 5107, 5108 | `0x4d470418ca543365` | yes | | `turtle.4th` | 5100, 5101, 5102, 5103, 5104, 5105, 5106, 5107, 5108 | `0x4d470418ca543365` | yes |
| `user-font-demo.4th` | 4200, 4201, 4202 | `0xce1fd7d1b581a56d` | yes | | `user-font-demo.4th` | 4200, 4201, 4202 | `0xce1fd7d1b581a56d` | yes |
| `workload-0.4th` | 2200, 2201 | `0x93f86f60aeba8feb` | yes | | `workload-0.4th` | 2200, 2201 | `0x93f86f60aeba8feb` | yes |
| `workload-1-lite.4th` | 5058, 5059 | `0x44a7a7e3176dcc8d` | yes |
| `workload-1.4th` | 4406, 4415, 4425, 4435 | `0x63e251adb0a03613` | yes | | `workload-1.4th` | 4406, 4415, 4425, 4435 | `0x63e251adb0a03613` | yes |
| `workload-2.4th` | 4506, 4515, 4525, 4535, 4545 | `0xf113b3d0bcccae47` | yes | | `workload-2.4th` | 4506, 4515, 4525, 4535, 4545 | `0xf113b3d0bcccae47` | yes |
| `workload-3.4th` | 4606, 4615, 4625, 4635, 4645, 4655, 4665 | `0x62b7a71576ad1041` | yes | | `workload-3.4th` | 4606, 4615, 4625, 4635, 4645, 4655, 4665 | `0x62b7a71576ad1041` | yes |
| `workload-4.4th` | 2130, 2131, 2132 | `0x099619264e650b1c` | yes | | `workload-4.4th` | 2130, 2131, 2132 | `0x099619264e650b1c` | yes |
| `workload-5-lite.4th` | 5056, 5057 | `0xea0303f38824f254` | yes |
| `workload-5.4th` | *(none — raw code capsule)* | `0x526964c7c9506a11` | yes | | `workload-5.4th` | *(none — raw code capsule)* | `0x526964c7c9506a11` | yes |
| `workload-6.4th` | 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095 | `0x06fc0ce1e369ef5a` | yes | | `workload-6.4th` | 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095 | `0x06fc0ce1e369ef5a` | yes |
| `workload-7.4th` | 2150 | `0x02862291387eb7ac` | yes | | `workload-7.4th` | 2150 | `0x02862291387eb7ac` | yes |
@@ -374,18 +376,22 @@
| 5041 | `common:messaging.4th` | `0x201cfd6d39fcb22d` | ok | | 5041 | `common:messaging.4th` | `0x201cfd6d39fcb22d` | ok |
| 5042 | `common:messaging.4th` | `0x201cfd6d39fcb22d` | ok | | 5042 | `common:messaging.4th` | `0x201cfd6d39fcb22d` | ok |
| 5043 | `workload-calib1.4th` | `0x3b9f2d17b554fabc` | ok | | 5043 | `workload-calib1.4th` | `0x3b9f2d17b554fabc` | ok |
| 5044 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5044 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5045 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5045 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5046 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5046 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5047 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5047 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5048 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5048 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5049 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5049 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5050 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5050 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5051 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5051 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5052 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5052 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5053 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5053 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5054 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5054 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5055 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok | | 5055 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
| 5056 | `workload-5-lite.4th` | `0xea0303f38824f254` | ok |
| 5057 | `workload-5-lite.4th` | `0xea0303f38824f254` | ok |
| 5058 | `workload-1-lite.4th` | `0x44a7a7e3176dcc8d` | ok |
| 5059 | `workload-1-lite.4th` | `0x44a7a7e3176dcc8d` | ok |
| 5100 | `turtle.4th` | `0x4d470418ca543365` | ok | | 5100 | `turtle.4th` | `0x4d470418ca543365` | ok |
| 5101 | `turtle.4th` | `0x4d470418ca543365` | ok | | 5101 | `turtle.4th` | `0x4d470418ca543365` | ok |
| 5102 | `turtle.4th` | `0x4d470418ca543365` | ok | | 5102 | `turtle.4th` | `0x4d470418ca543365` | ok |
@@ -409,4 +415,4 @@
None. None.
--- ---
*37 capsule(s) scanned. Re-run `mkcapsule --manifest <dir>` to refresh.* *39 capsule(s) scanned. Re-run `mkcapsule --manifest <dir>` to refresh.*
+3 -3
View File
@@ -17,11 +17,11 @@ Block 5045
: WL-CAPSULE ( idx -- c-addr u ) : WL-CAPSULE ( idx -- c-addr u )
CASE CASE
0 OF S" workload-0.4th" ENDOF 0 OF S" workload-0.4th" ENDOF
1 OF S" workload-1.4th" ENDOF 1 OF S" workload-1-lite.4th" ENDOF
2 OF S" workload-2.4th" ENDOF 2 OF S" workload-2.4th" ENDOF
3 OF S" workload-3.4th" ENDOF 3 OF S" workload-3.4th" ENDOF
4 OF S" workload-4.4th" ENDOF 4 OF S" workload-4.4th" ENDOF
5 OF S" workload-5.4th" ENDOF 5 OF S" workload-5-lite.4th" ENDOF
6 OF S" workload-6.4th" ENDOF 6 OF S" workload-6.4th" ENDOF
7 OF S" workload-7.4th" ENDOF 7 OF S" workload-7.4th" ENDOF
8 OF S" workload-8.4th" ENDOF 8 OF S" workload-8.4th" ENDOF
@@ -36,7 +36,7 @@ Block 5046
2 OF S" RUN-SAW" ENDOF 2 OF S" RUN-SAW" ENDOF
3 OF S" DUAL-SINE" ENDOF 3 OF S" DUAL-SINE" ENDOF
4 OF S" RUN-PI" ENDOF 4 OF S" RUN-PI" ENDOF
5 OF S" RUN-CHAOS5" ENDOF 5 OF S" CHAOS-FIELD" ENDOF
6 OF S" RUN-OMNI" ENDOF 6 OF S" RUN-OMNI" ENDOF
7 OF S" RUN-MMUL" ENDOF 7 OF S" RUN-MMUL" ENDOF
8 OF S" RUN-FIB" ENDOF 8 OF S" RUN-FIB" ENDOF
+18
View File
@@ -0,0 +1,18 @@
Block 5058
( workload-1-lite.4th -- bounded MICRO-BURST, SXXXIII. )
( workload-1.4th's own self-exec tail (SQUARE-WAVE + )
( SQUARE-BURST + 100000x MICRO-BURST) is ~875M words -- )
( found live, ~4.3hrs just to birth one worker. Same )
( BURST/COMPUTE/MICRO-BURST bodies, bounded self-exec. )
: COMPUTE ( n -- )
0 SWAP 0 DO I 7 * + LOOP DROP ;
: BURST ( intensity -- )
100 * COMPUTE ;
: MICRO-BURST ( -- )
10 30 RANDOM BURST
50 150 RANDOM WAIT ;
Block 5059
: RUN-MICRO-BURSTS ( -- )
1000 0 DO MICRO-BURST LOOP ;
RUN-MICRO-BURSTS
+24
View File
@@ -0,0 +1,24 @@
Block 5056
( workload-5-lite.4th -- bounded CHAOS-FIELD, SXXXIII. )
( workload-5.4th's RUN-CHAOS5 is ~2.4T words -- unusable )
( as a touch. Same CHAOS-BLOCK body, bounded self-exec. )
: CHAOS-BLOCK
256 0 DO
I 4 MOD 0= IF
64 0 DO I J + J * DROP LOOP
ELSE
64 0 DO I J * I + DROP LOOP
THEN
I 8 MOD 0= IF
128 0 DO I J + J * I + DROP LOOP
THEN
LOOP
;
Block 5057
: CHAOS-FIELD
24 0 DO
CHAOS-BLOCK
LOOP
;
CHAOS-FIELD
BIN
View File
Binary file not shown.
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff