Fix two catastrophically heavy workloads found running the real campaign
The full 360-trial campaign was launched, then killed after 4h39m of CPU time with zero trials completed -- still stuck on the very first touch of the very first trial. Root cause, quantified from the workload's own source, not estimated: RUN-CHAOS5 (workload-5.4th) is 1000000 0 DO CHAOS-FIELD CHAOS-RIPPLE LOOP, multiplying out to ~2.4 trillion word executions for one call -- ~495 days at the observed rate. Never designed to be called to completion as a single touch in a repeatedly-birthed worker. A second problem found computing the fix rather than discovering it mid-run again: workload-1.4th's own birth-time self-execution (20 SQUARE-WAVE + 1500 SQUARE-BURST + 100000x MICRO-BURST, all three run automatically at capsule load) sums to ~875 million words -- ~4.3 hours just to birth one worker -- and worker index 1 always maps to it in heterogeneous mode, landing in half the campaign's cells regardless of concurrency level. Fix: two new capsules, workload-1-lite.4th and workload-5-lite.4th, carrying the same word bodies verbatim but a bounded self-execution tail, comparable in scale to the campaign's other workloads. The originals are untouched; only multiuser-doe.4th's own WL-CAPSULE/ WL-ENTRY index 1 and 5 mappings were repointed. Verified live before relaunching a third time: the actual worst case in isolation (5 0 998 MU-RUN-TRIAL, concurrency=8 heterogeneous, includes both fixed workloads plus RUN-OMNI) completed in ~20 minutes wall-clock, Hera stayed healthy throughout. Clean 3-arch qemu boot. Reps reduced 60 -> 30/cell (Bob's call after seeing the real per-trial cost) -- still matches the project's "rule of 3's" DoE convention (same as ACL-RWT's own 30 reps). 6 cfgs x 30 reps = 180 trials. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BWpNjdwPtFLuVLaAq44L9K
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
29f91f9531
commit
94215e8b47
+42
@@ -5090,3 +5090,45 @@ workloads like `RUN-CHAOS5`'s 1,000,000 iterations) -- a genuinely long-running
|
|||||||
TCG, deliberately not kicked off without explicit confirmation first. The WIREBIND-automation
|
TCG, deliberately not kicked off without explicit confirmation first. The WIREBIND-automation
|
||||||
fixed arm also remains unbuilt.
|
fixed arm also remains unbuilt.
|
||||||
|
|
||||||
|
### XXXIII.7 -- Two workloads too heavy to touch as-is, found live at real cost
|
||||||
|
|
||||||
|
**Approved and launched, then killed after finding a real problem the design review never
|
||||||
|
caught:** the full campaign ran for 4h39m of CPU time without completing even the first
|
||||||
|
touch of the first trial. Root cause quantified directly from the workload's own source
|
||||||
|
(`workload-5.4th`), not estimated: `RUN-CHAOS5` is `1000000 0 DO CHAOS-FIELD CHAOS-RIPPLE
|
||||||
|
LOOP`, and `CHAOS-FIELD` alone is `24 0 DO CHAOS-BLOCK LOOP` with `CHAOS-BLOCK` itself ~100K
|
||||||
|
word executions -- multiplying out to **~2.4 trillion word executions for one call**. At the
|
||||||
|
observed rate (~56,300 words/sec), that's roughly 495 days for a single touch. This workload
|
||||||
|
was never designed to be called to completion as one touch in a repeatedly-birthed-and-killed
|
||||||
|
worker; whatever its role in the original single-VM L8 DoE, it doesn't fit this campaign's
|
||||||
|
shape at all.
|
||||||
|
|
||||||
|
**A second, related problem found computing the fix, not by running it first:** `workload-
|
||||||
|
1.4th`'s own birth-time self-execution (`20 SQUARE-WAVE` + `1500 SQUARE-BURST` + `100000x
|
||||||
|
MICRO-BURST`, all three run automatically at capsule load) sums to roughly **875 million word
|
||||||
|
executions -- ~4.3 hours just to birth one worker**, and worker index 1 always maps to this
|
||||||
|
capsule in heterogeneous mode, so the cost lands in half the campaign's cells regardless of
|
||||||
|
concurrency level. Caught by computing the real per-workload word counts from every capsule's
|
||||||
|
own source before relaunching a third time, not by discovering it mid-run again.
|
||||||
|
|
||||||
|
**Fix, matching the pattern Captain Bob had already authorized ("workloads are not carved in
|
||||||
|
stone, create new extended workloads where needed"):** two new capsules,
|
||||||
|
`workload-5-lite.4th` and `workload-1-lite.4th`, carrying the *same* word bodies verbatim
|
||||||
|
(`CHAOS-BLOCK`/`CHAOS-FIELD`; `COMPUTE`/`BURST`/`MICRO-BURST`) but a bounded self-execution
|
||||||
|
tail (`CHAOS-FIELD` once; `1000x MICRO-BURST`) -- comparable in scale to the campaign's other
|
||||||
|
workloads (~2-30M words), not the original files' unbounded repeat counts. The originals
|
||||||
|
(`workload-1.4th`, `workload-5.4th`) are untouched, still real, still available for whatever
|
||||||
|
their original context needs -- only `multiuser-doe.4th`'s own `WL-CAPSULE`/`WL-ENTRY` index
|
||||||
|
1 and 5 mappings were repointed to the lite variants.
|
||||||
|
|
||||||
|
**Verified live before relaunching the full campaign a third time:** ran the actual worst
|
||||||
|
case in isolation -- `5 0 998 MU-RUN-TRIAL` (cfg=5: concurrency=8, heterogeneous, so it
|
||||||
|
includes both previously-catastrophic workloads plus `RUN-OMNI`'s own ~30M words) -- and it
|
||||||
|
completed in ~20 minutes wall-clock (`MU-TRIAL run=998 cfg=5 rep=0 nw=8 mode=1 fail=0`), Hera
|
||||||
|
stayed healthy throughout. Clean 3-arch qemu boot on the real committed capsule changes.
|
||||||
|
|
||||||
|
**Reps reduced from 60 to 30 per cell**, Captain Bob's own call after seeing the real per-
|
||||||
|
trial cost -- still matches the project's "rule of 3's" DoE convention exactly (the same 30
|
||||||
|
reps as the ACL-RWT campaign). 6 cfgs x 30 reps = 180 main-block trials, roughly half the
|
||||||
|
prior 360-trial estimate.
|
||||||
|
|
||||||
|
|||||||
+21
-15
@@ -1,5 +1,5 @@
|
|||||||
# Capsule Block Manifest — Auto-generated
|
# Capsule Block Manifest — Auto-generated
|
||||||
<!-- Generated by mkcapsule --manifest 2026-09-16T14:25:13Z -->
|
<!-- Generated by mkcapsule --manifest 2026-09-16T19:45:24Z -->
|
||||||
<!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh. -->
|
<!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh. -->
|
||||||
<!-- Hand-written justifications and immutability notes live -->
|
<!-- Hand-written justifications and immutability notes live -->
|
||||||
<!-- in MANIFEST.md alongside this auto-generated index. -->
|
<!-- in MANIFEST.md alongside this auto-generated index. -->
|
||||||
@@ -27,16 +27,18 @@
|
|||||||
| `init-l8-volatile.4th` | 4810, 4811, 4812, 4813 | `0x98caabbbd92abac4` | yes |
|
| `init-l8-volatile.4th` | 4810, 4811, 4812, 4813 | `0x98caabbbd92abac4` | yes |
|
||||||
| `init.4th` | 2049, 2050, 2057 | `0x1ef4939ed32ec1e6` | yes |
|
| `init.4th` | 2049, 2050, 2057 | `0x1ef4939ed32ec1e6` | yes |
|
||||||
| `lib.4th` | 4050 | `0x4b216635c359ef73` | yes |
|
| `lib.4th` | 4050 | `0x4b216635c359ef73` | yes |
|
||||||
| `multiuser-doe.4th` | 5044, 5045, 5046, 5047, 5048, 5049, 5050, 5051, 5052, 5053, 5054, 5055 | `0x1dc57cdf83afb6ce` | yes |
|
| `multiuser-doe.4th` | 5044, 5045, 5046, 5047, 5048, 5049, 5050, 5051, 5052, 5053, 5054, 5055 | `0x22140588ee7c39e6` | yes |
|
||||||
| `process.4th` | 4300, 4301 | `0x781afc1dbd0294f7` | yes |
|
| `process.4th` | 4300, 4301 | `0x781afc1dbd0294f7` | yes |
|
||||||
| `sdk.4th` | 5109, 5110, 5111, 5112, 5113, 5114, 5115 | `0x008fdbbb62c94a3a` | yes |
|
| `sdk.4th` | 5109, 5110, 5111, 5112, 5113, 5114, 5115 | `0x008fdbbb62c94a3a` | yes |
|
||||||
| `turtle.4th` | 5100, 5101, 5102, 5103, 5104, 5105, 5106, 5107, 5108 | `0x4d470418ca543365` | yes |
|
| `turtle.4th` | 5100, 5101, 5102, 5103, 5104, 5105, 5106, 5107, 5108 | `0x4d470418ca543365` | yes |
|
||||||
| `user-font-demo.4th` | 4200, 4201, 4202 | `0xce1fd7d1b581a56d` | yes |
|
| `user-font-demo.4th` | 4200, 4201, 4202 | `0xce1fd7d1b581a56d` | yes |
|
||||||
| `workload-0.4th` | 2200, 2201 | `0x93f86f60aeba8feb` | yes |
|
| `workload-0.4th` | 2200, 2201 | `0x93f86f60aeba8feb` | yes |
|
||||||
|
| `workload-1-lite.4th` | 5058, 5059 | `0x44a7a7e3176dcc8d` | yes |
|
||||||
| `workload-1.4th` | 4406, 4415, 4425, 4435 | `0x63e251adb0a03613` | yes |
|
| `workload-1.4th` | 4406, 4415, 4425, 4435 | `0x63e251adb0a03613` | yes |
|
||||||
| `workload-2.4th` | 4506, 4515, 4525, 4535, 4545 | `0xf113b3d0bcccae47` | yes |
|
| `workload-2.4th` | 4506, 4515, 4525, 4535, 4545 | `0xf113b3d0bcccae47` | yes |
|
||||||
| `workload-3.4th` | 4606, 4615, 4625, 4635, 4645, 4655, 4665 | `0x62b7a71576ad1041` | yes |
|
| `workload-3.4th` | 4606, 4615, 4625, 4635, 4645, 4655, 4665 | `0x62b7a71576ad1041` | yes |
|
||||||
| `workload-4.4th` | 2130, 2131, 2132 | `0x099619264e650b1c` | yes |
|
| `workload-4.4th` | 2130, 2131, 2132 | `0x099619264e650b1c` | yes |
|
||||||
|
| `workload-5-lite.4th` | 5056, 5057 | `0xea0303f38824f254` | yes |
|
||||||
| `workload-5.4th` | *(none — raw code capsule)* | `0x526964c7c9506a11` | yes |
|
| `workload-5.4th` | *(none — raw code capsule)* | `0x526964c7c9506a11` | yes |
|
||||||
| `workload-6.4th` | 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095 | `0x06fc0ce1e369ef5a` | yes |
|
| `workload-6.4th` | 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095 | `0x06fc0ce1e369ef5a` | yes |
|
||||||
| `workload-7.4th` | 2150 | `0x02862291387eb7ac` | yes |
|
| `workload-7.4th` | 2150 | `0x02862291387eb7ac` | yes |
|
||||||
@@ -374,18 +376,22 @@
|
|||||||
| 5041 | `common:messaging.4th` | `0x201cfd6d39fcb22d` | ok |
|
| 5041 | `common:messaging.4th` | `0x201cfd6d39fcb22d` | ok |
|
||||||
| 5042 | `common:messaging.4th` | `0x201cfd6d39fcb22d` | ok |
|
| 5042 | `common:messaging.4th` | `0x201cfd6d39fcb22d` | ok |
|
||||||
| 5043 | `workload-calib1.4th` | `0x3b9f2d17b554fabc` | ok |
|
| 5043 | `workload-calib1.4th` | `0x3b9f2d17b554fabc` | ok |
|
||||||
| 5044 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5044 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5045 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5045 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5046 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5046 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5047 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5047 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5048 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5048 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5049 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5049 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5050 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5050 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5051 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5051 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5052 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5052 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5053 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5053 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5054 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5054 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
| 5055 | `multiuser-doe.4th` | `0x1dc57cdf83afb6ce` | ok |
|
| 5055 | `multiuser-doe.4th` | `0x22140588ee7c39e6` | ok |
|
||||||
|
| 5056 | `workload-5-lite.4th` | `0xea0303f38824f254` | ok |
|
||||||
|
| 5057 | `workload-5-lite.4th` | `0xea0303f38824f254` | ok |
|
||||||
|
| 5058 | `workload-1-lite.4th` | `0x44a7a7e3176dcc8d` | ok |
|
||||||
|
| 5059 | `workload-1-lite.4th` | `0x44a7a7e3176dcc8d` | ok |
|
||||||
| 5100 | `turtle.4th` | `0x4d470418ca543365` | ok |
|
| 5100 | `turtle.4th` | `0x4d470418ca543365` | ok |
|
||||||
| 5101 | `turtle.4th` | `0x4d470418ca543365` | ok |
|
| 5101 | `turtle.4th` | `0x4d470418ca543365` | ok |
|
||||||
| 5102 | `turtle.4th` | `0x4d470418ca543365` | ok |
|
| 5102 | `turtle.4th` | `0x4d470418ca543365` | ok |
|
||||||
@@ -409,4 +415,4 @@
|
|||||||
None.
|
None.
|
||||||
|
|
||||||
---
|
---
|
||||||
*37 capsule(s) scanned. Re-run `mkcapsule --manifest <dir>` to refresh.*
|
*39 capsule(s) scanned. Re-run `mkcapsule --manifest <dir>` to refresh.*
|
||||||
|
|||||||
@@ -17,11 +17,11 @@ Block 5045
|
|||||||
: WL-CAPSULE ( idx -- c-addr u )
|
: WL-CAPSULE ( idx -- c-addr u )
|
||||||
CASE
|
CASE
|
||||||
0 OF S" workload-0.4th" ENDOF
|
0 OF S" workload-0.4th" ENDOF
|
||||||
1 OF S" workload-1.4th" ENDOF
|
1 OF S" workload-1-lite.4th" ENDOF
|
||||||
2 OF S" workload-2.4th" ENDOF
|
2 OF S" workload-2.4th" ENDOF
|
||||||
3 OF S" workload-3.4th" ENDOF
|
3 OF S" workload-3.4th" ENDOF
|
||||||
4 OF S" workload-4.4th" ENDOF
|
4 OF S" workload-4.4th" ENDOF
|
||||||
5 OF S" workload-5.4th" ENDOF
|
5 OF S" workload-5-lite.4th" ENDOF
|
||||||
6 OF S" workload-6.4th" ENDOF
|
6 OF S" workload-6.4th" ENDOF
|
||||||
7 OF S" workload-7.4th" ENDOF
|
7 OF S" workload-7.4th" ENDOF
|
||||||
8 OF S" workload-8.4th" ENDOF
|
8 OF S" workload-8.4th" ENDOF
|
||||||
@@ -36,7 +36,7 @@ Block 5046
|
|||||||
2 OF S" RUN-SAW" ENDOF
|
2 OF S" RUN-SAW" ENDOF
|
||||||
3 OF S" DUAL-SINE" ENDOF
|
3 OF S" DUAL-SINE" ENDOF
|
||||||
4 OF S" RUN-PI" ENDOF
|
4 OF S" RUN-PI" ENDOF
|
||||||
5 OF S" RUN-CHAOS5" ENDOF
|
5 OF S" CHAOS-FIELD" ENDOF
|
||||||
6 OF S" RUN-OMNI" ENDOF
|
6 OF S" RUN-OMNI" ENDOF
|
||||||
7 OF S" RUN-MMUL" ENDOF
|
7 OF S" RUN-MMUL" ENDOF
|
||||||
8 OF S" RUN-FIB" ENDOF
|
8 OF S" RUN-FIB" ENDOF
|
||||||
|
|||||||
@@ -0,0 +1,18 @@
|
|||||||
|
Block 5058
|
||||||
|
( workload-1-lite.4th -- bounded MICRO-BURST, SXXXIII. )
|
||||||
|
( workload-1.4th's own self-exec tail (SQUARE-WAVE + )
|
||||||
|
( SQUARE-BURST + 100000x MICRO-BURST) is ~875M words -- )
|
||||||
|
( found live, ~4.3hrs just to birth one worker. Same )
|
||||||
|
( BURST/COMPUTE/MICRO-BURST bodies, bounded self-exec. )
|
||||||
|
: COMPUTE ( n -- )
|
||||||
|
0 SWAP 0 DO I 7 * + LOOP DROP ;
|
||||||
|
: BURST ( intensity -- )
|
||||||
|
100 * COMPUTE ;
|
||||||
|
: MICRO-BURST ( -- )
|
||||||
|
10 30 RANDOM BURST
|
||||||
|
50 150 RANDOM WAIT ;
|
||||||
|
|
||||||
|
Block 5059
|
||||||
|
: RUN-MICRO-BURSTS ( -- )
|
||||||
|
1000 0 DO MICRO-BURST LOOP ;
|
||||||
|
RUN-MICRO-BURSTS
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
Block 5056
|
||||||
|
( workload-5-lite.4th -- bounded CHAOS-FIELD, SXXXIII. )
|
||||||
|
( workload-5.4th's RUN-CHAOS5 is ~2.4T words -- unusable )
|
||||||
|
( as a touch. Same CHAOS-BLOCK body, bounded self-exec. )
|
||||||
|
: CHAOS-BLOCK
|
||||||
|
256 0 DO
|
||||||
|
I 4 MOD 0= IF
|
||||||
|
64 0 DO I J + J * DROP LOOP
|
||||||
|
ELSE
|
||||||
|
64 0 DO I J * I + DROP LOOP
|
||||||
|
THEN
|
||||||
|
I 8 MOD 0= IF
|
||||||
|
128 0 DO I J + J * I + DROP LOOP
|
||||||
|
THEN
|
||||||
|
LOOP
|
||||||
|
;
|
||||||
|
|
||||||
|
Block 5057
|
||||||
|
: CHAOS-FIELD
|
||||||
|
24 0 DO
|
||||||
|
CHAOS-BLOCK
|
||||||
|
LOOP
|
||||||
|
;
|
||||||
|
CHAOS-FIELD
|
||||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user