diff --git a/experiments/std79-doe/README.md b/experiments/std79-doe/README.md index 78417d32..72d0bd92 100644 --- a/experiments/std79-doe/README.md +++ b/experiments/std79-doe/README.md @@ -68,3 +68,15 @@ nothing is corrupted in memory — only the printed character stream interleaves tooling that wants an exact reconstruction of the trial-output stream from these three raw logs needs to account for that (strip `[HADES][DOE ] ...` fragments and rejoin) rather than assume one physical line is one logical print. + +`analysis-20260912/` builds exactly that reconstruction and turns it into a real 3×9 factorial +analysis: `correlate_doe.py` labels every heartbeat-tick row with the trial that was active when +it printed, `combine.py` merges all three architectures into `combined.csv` (768 rows, Q48.16 +fields decoded to floats), and `analysis.R` computes per-cell (architecture × identity) means/SD +and a two-way ANOVA for each of 12 telemetry metrics, rendering one boxplot SVG per metric. See +`analysis-20260912/README.md` for the pipeline and `analysis-20260912/ANALYSIS.md` for the +report itself — key findings: identity significantly affects word-heat/window-sizing metrics +(expected — different identities execute different word sets), architecture significantly +affects timing metrics (APIC ticks/tick, timing variance, fleet heat — expected, different QEMU +targets), and **zero architecture × identity interaction on any metric** — the two factors' +effects are clean and separable, not confounded. diff --git a/experiments/std79-doe/analysis-20260912/ANALYSIS.md b/experiments/std79-doe/analysis-20260912/ANALYSIS.md new file mode 100644 index 00000000..ee12f5dc --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/ANALYSIS.md @@ -0,0 +1,554 @@ +# std79-doe 3×9 Factorial Analysis — Heartbeat/Physics Telemetry + +Generated 2026-09-12 12:45. Source: `combined.csv` (768 rows), built by +`correlate_doe.py` from the three raw logs in +`../results-20260912-with-heartbeat-csv/`. See that file's own docstring for the +correlation methodology (the heartbeat tick's async CSV printer and the DoE trial +loop's own console output share one serial line with no locking, so rows can land +mid-token in the raw logs; the underlying FORTH execution and values are +unaffected, only reconstructing which trial owns which row needed care). + +**Design:** 3 architectures (amd64, aarch64, riscv64) × 9 identities (zuse, +rajames, 00-06) × 3 replicates, Fisher-Yates-shuffled run order, same seed +(12345) on every architecture (FABRIC-3.md §XV). Each of the 27 trials per +architecture runs the same 24-word FORTH-79 exerciser; every heartbeat tick during +the whole `EXEC-STD79-DOE` run (bracketed by `HB-ON`/`HB-OFF`) emits one telemetry +row (`doe_log.c`), correlated back to whichever trial was active when it printed. + +**Known limitation:** aarch64's raw log lost 1 of 27 run_id markers to interleaving +beyond recovery (trial 12, identity `rajames` rep 1) -- its rows are folded into +trial 11 (identity `rajames` rep 0) in this dataset, so aarch64's `rajames` cell +for rep-sensitive metrics is not perfectly separable for those two replicates +specifically. Every other cell on every architecture is unaffected. + +## Key findings (p < 0.05, two-way ANOVA) + +- **Architecture main effect:** APIC ticks per heartbeat tick, TIME-TRUST, Timing variance, Hera fleet heat, Hermes fleet heat +- **Identity main effect:** Hot word count, Mean word execution heat, Rolling window width, Actual analysis window size +- **Architecture × identity interaction:** none +- **No significant effect of either factor:** Estimated timer jitter (ns), Max VM call depth, Artemis fleet heat + +TIME-TRUST's architecture effect is a numerical artifact worth reading correctly, +not a substantive result: it is essentially constant *within* each architecture +(residual variance ~1e-31, i.e. floating-point noise) and differs *between* +architectures — so the F statistic is enormous simply because the within-group +denominator is near zero, not because TIME-TRUST is meaningfully more variable +across architectures than the other metrics here. It is architecture-determined +and identity-independent, which is itself the interesting part. + +## Hot word count (`hot_word_count`) + +![Hot word count](svg/hot_word_count.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 67.91 | 1.411 | 2.08 | +| amd64 | rajames | 25 | 64.28 | 7.414 | 11.53 | +| amd64 | 00 | 26 | 64.46 | 7.235 | 11.22 | +| amd64 | 01 | 26 | 62.62 | 6.929 | 11.07 | +| amd64 | 02 | 28 | 62.5 | 6.251 | 10.00 | +| amd64 | 03 | 30 | 62.5 | 5.71 | 9.14 | +| amd64 | 04 | 33 | 66.55 | 5.386 | 8.09 | +| amd64 | 05 | 33 | 64.45 | 3.874 | 6.01 | +| amd64 | 06 | 31 | 65.1 | 3.953 | 6.07 | +| aarch64 | zuse | 23 | 69.35 | 1.191 | 1.72 | +| aarch64 | rajames | 16 | 65.19 | 8.336 | 12.79 | +| aarch64 | 00 | 26 | 62.92 | 7.451 | 11.84 | +| aarch64 | 01 | 27 | 59.81 | 8.005 | 13.38 | +| aarch64 | 02 | 37 | 61.27 | 8.425 | 13.75 | +| aarch64 | 03 | 29 | 60.79 | 7.208 | 11.86 | +| aarch64 | 04 | 33 | 67.48 | 5.745 | 8.51 | +| aarch64 | 05 | 33 | 62.73 | 4.692 | 7.48 | +| aarch64 | 06 | 32 | 63.19 | 4.351 | 6.89 | +| riscv64 | zuse | 23 | 69.48 | 1.039 | 1.50 | +| riscv64 | rajames | 26 | 65.27 | 11.27 | 17.27 | +| riscv64 | 00 | 27 | 62.33 | 8.953 | 14.36 | +| riscv64 | 01 | 28 | 61.68 | 7.252 | 11.76 | +| riscv64 | 02 | 27 | 62.3 | 8.109 | 13.02 | +| riscv64 | 03 | 28 | 63 | 7.123 | 11.31 | +| riscv64 | 04 | 32 | 67.53 | 6.069 | 8.99 | +| riscv64 | 05 | 32 | 65.56 | 5.691 | 8.68 | +| riscv64 | 06 | 34 | 63.85 | 3.839 | 6.01 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 193 | 96.5 | 2.333 | 0.09771 | +| id_label | 8 | 3985 | 498.2 | 12.04 | 2.984e-16 | +| arch:id_label | 16 | 379.9 | 23.74 | 0.574 | 0.9044 | +| Residuals | 741 | 3.065e+04 | 41.36 | — | — | + +## Mean word execution heat (`avg_word_heat`) + +![Mean word execution heat](svg/avg_word_heat.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 0.003006 | 0.001235 | 41.07 | +| amd64 | rajames | 25 | 0.008839 | 0.008672 | 98.11 | +| amd64 | 00 | 26 | 0.008918 | 0.006694 | 75.06 | +| amd64 | 01 | 26 | 0.01001 | 0.007327 | 73.18 | +| amd64 | 02 | 28 | 0.01069 | 0.004371 | 40.89 | +| amd64 | 03 | 30 | 0.01032 | 0.005016 | 48.61 | +| amd64 | 04 | 33 | 0.004205 | 0.002958 | 70.35 | +| amd64 | 05 | 33 | 0.008833 | 0.001235 | 13.98 | +| amd64 | 06 | 31 | 0.008481 | 0.003743 | 44.13 | +| aarch64 | zuse | 23 | 0.002911 | 0.001418 | 48.72 | +| aarch64 | rajames | 16 | 0.007391 | 0.005726 | 77.48 | +| aarch64 | 00 | 26 | 0.009563 | 0.006967 | 72.85 | +| aarch64 | 01 | 27 | 0.01071 | 0.007785 | 72.67 | +| aarch64 | 02 | 37 | 0.01146 | 0.007358 | 64.20 | +| aarch64 | 03 | 29 | 0.01046 | 0.005237 | 50.05 | +| aarch64 | 04 | 33 | 0.004262 | 0.00298 | 69.93 | +| aarch64 | 05 | 33 | 0.009139 | 0.001443 | 15.79 | +| aarch64 | 06 | 32 | 0.008941 | 0.003745 | 41.89 | +| riscv64 | zuse | 23 | 0.00289 | 0.00163 | 56.41 | +| riscv64 | rajames | 26 | 0.0106 | 0.0133 | 125.42 | +| riscv64 | 00 | 27 | 0.01041 | 0.009052 | 86.95 | +| riscv64 | 01 | 28 | 0.01068 | 0.007445 | 69.74 | +| riscv64 | 02 | 27 | 0.01104 | 0.005331 | 48.29 | +| riscv64 | 03 | 28 | 0.01007 | 0.005873 | 58.30 | +| riscv64 | 04 | 32 | 0.004007 | 0.002863 | 71.45 | +| riscv64 | 05 | 32 | 0.009038 | 0.001403 | 15.52 | +| riscv64 | 06 | 34 | 0.009104 | 0.003387 | 37.20 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 3.325e-05 | 1.662e-05 | 0.5221 | 0.5935 | +| id_label | 8 | 0.005366 | 0.0006707 | 21.07 | 6.051e-29 | +| arch:id_label | 16 | 0.0001362 | 8.514e-06 | 0.2674 | 0.9983 | +| Residuals | 741 | 0.02359 | 3.184e-05 | — | — | + +## Rolling window width (`window_width`) + +![Rolling window width](svg/window_width.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 256 | 0 | 0.00 | +| amd64 | rajames | 25 | 256 | 0 | 0.00 | +| amd64 | 00 | 26 | 256 | 0 | 0.00 | +| amd64 | 01 | 26 | 256 | 0 | 0.00 | +| amd64 | 02 | 28 | 256 | 0 | 0.00 | +| amd64 | 03 | 30 | 256 | 0 | 0.00 | +| amd64 | 04 | 33 | 318.1 | 192.3 | 60.47 | +| amd64 | 05 | 33 | 256 | 0 | 0.00 | +| amd64 | 06 | 31 | 256 | 0 | 0.00 | +| aarch64 | zuse | 23 | 256 | 0 | 0.00 | +| aarch64 | rajames | 16 | 256 | 0 | 0.00 | +| aarch64 | 00 | 26 | 256 | 0 | 0.00 | +| aarch64 | 01 | 27 | 256 | 0 | 0.00 | +| aarch64 | 02 | 37 | 256 | 0 | 0.00 | +| aarch64 | 03 | 29 | 256 | 0 | 0.00 | +| aarch64 | 04 | 33 | 318.1 | 192.3 | 60.47 | +| aarch64 | 05 | 33 | 256 | 0 | 0.00 | +| aarch64 | 06 | 32 | 256 | 0 | 0.00 | +| riscv64 | zuse | 23 | 256 | 0 | 0.00 | +| riscv64 | rajames | 26 | 256 | 0 | 0.00 | +| riscv64 | 00 | 27 | 256 | 0 | 0.00 | +| riscv64 | 01 | 28 | 256 | 0 | 0.00 | +| riscv64 | 02 | 27 | 256 | 0 | 0.00 | +| riscv64 | 03 | 28 | 256 | 0 | 0.00 | +| riscv64 | 04 | 32 | 296 | 147 | 49.66 | +| riscv64 | 05 | 32 | 256 | 0 | 0.00 | +| riscv64 | 06 | 34 | 256 | 0 | 0.00 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 1575 | 787.7 | 0.1922 | 0.8252 | +| id_label | 8 | 2.57e+05 | 3.213e+04 | 7.839 | 3.816e-10 | +| arch:id_label | 16 | 9163 | 572.7 | 0.1397 | 1 | +| Residuals | 741 | 3.037e+06 | 4098 | — | — | + +## Actual analysis window size (`actual_window_size`) + +![Actual analysis window size](svg/actual_window_size.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 256 | 0 | 0.00 | +| amd64 | rajames | 25 | 256 | 0 | 0.00 | +| amd64 | 00 | 26 | 256 | 0 | 0.00 | +| amd64 | 01 | 26 | 256 | 0 | 0.00 | +| amd64 | 02 | 28 | 256 | 0 | 0.00 | +| amd64 | 03 | 30 | 256 | 0 | 0.00 | +| amd64 | 04 | 33 | 318.1 | 192.3 | 60.47 | +| amd64 | 05 | 33 | 256 | 0 | 0.00 | +| amd64 | 06 | 31 | 256 | 0 | 0.00 | +| aarch64 | zuse | 23 | 256 | 0 | 0.00 | +| aarch64 | rajames | 16 | 256 | 0 | 0.00 | +| aarch64 | 00 | 26 | 256 | 0 | 0.00 | +| aarch64 | 01 | 27 | 256 | 0 | 0.00 | +| aarch64 | 02 | 37 | 256 | 0 | 0.00 | +| aarch64 | 03 | 29 | 256 | 0 | 0.00 | +| aarch64 | 04 | 33 | 318.1 | 192.3 | 60.47 | +| aarch64 | 05 | 33 | 256 | 0 | 0.00 | +| aarch64 | 06 | 32 | 256 | 0 | 0.00 | +| riscv64 | zuse | 23 | 256 | 0 | 0.00 | +| riscv64 | rajames | 26 | 256 | 0 | 0.00 | +| riscv64 | 00 | 27 | 256 | 0 | 0.00 | +| riscv64 | 01 | 28 | 256 | 0 | 0.00 | +| riscv64 | 02 | 27 | 256 | 0 | 0.00 | +| riscv64 | 03 | 28 | 256 | 0 | 0.00 | +| riscv64 | 04 | 32 | 296 | 147 | 49.66 | +| riscv64 | 05 | 32 | 256 | 0 | 0.00 | +| riscv64 | 06 | 34 | 256 | 0 | 0.00 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 1575 | 787.7 | 0.1922 | 0.8252 | +| id_label | 8 | 2.57e+05 | 3.213e+04 | 7.839 | 3.816e-10 | +| arch:id_label | 16 | 9163 | 572.7 | 0.1397 | 1 | +| Residuals | 741 | 3.037e+06 | 4098 | — | — | + +## Estimated timer jitter (ns) (`jitter_ns`) + +![Estimated timer jitter (ns)](svg/jitter_ns.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 0 | 0 | — | +| amd64 | rajames | 25 | 7.379e+17 | 3.689e+18 | 500.00 | +| amd64 | 00 | 26 | 7.095e+17 | 3.618e+18 | 509.90 | +| amd64 | 01 | 26 | 9.462e+04 | 4.824e+05 | 509.90 | +| amd64 | 02 | 28 | 6.588e+17 | 3.486e+18 | 529.15 | +| amd64 | 03 | 30 | 1.23e+18 | 4.68e+18 | 380.56 | +| amd64 | 04 | 33 | 5.59e+17 | 3.211e+18 | 574.46 | +| amd64 | 05 | 33 | 5.59e+17 | 3.211e+18 | 574.46 | +| amd64 | 06 | 31 | 5.951e+17 | 3.313e+18 | 556.78 | +| aarch64 | zuse | 23 | 0 | 0 | — | +| aarch64 | rajames | 16 | 0 | 0 | — | +| aarch64 | 00 | 26 | 7.095e+17 | 3.618e+18 | 509.90 | +| aarch64 | 01 | 27 | 6.832e+17 | 3.55e+18 | 519.62 | +| aarch64 | 02 | 37 | 9.971e+17 | 4.229e+18 | 424.10 | +| aarch64 | 03 | 29 | 6.361e+17 | 3.425e+18 | 538.52 | +| aarch64 | 04 | 33 | 5.59e+17 | 3.211e+18 | 574.46 | +| aarch64 | 05 | 33 | 1.118e+18 | 4.47e+18 | 399.80 | +| aarch64 | 06 | 32 | 5.765e+17 | 3.261e+18 | 565.69 | +| riscv64 | zuse | 23 | 0 | 0 | — | +| riscv64 | rajames | 26 | 1.419e+18 | 5.013e+18 | 353.27 | +| riscv64 | 00 | 27 | 6.832e+17 | 3.55e+18 | 519.62 | +| riscv64 | 01 | 28 | 6.588e+17 | 3.486e+18 | 529.15 | +| riscv64 | 02 | 27 | 6.832e+17 | 3.55e+18 | 519.62 | +| riscv64 | 03 | 28 | 6.588e+17 | 3.486e+18 | 529.15 | +| riscv64 | 04 | 32 | 5.765e+17 | 3.261e+18 | 565.69 | +| riscv64 | 05 | 32 | 5.765e+17 | 3.261e+18 | 565.69 | +| riscv64 | 06 | 34 | 1.085e+18 | 4.406e+18 | 406.02 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 2.475e+36 | 1.237e+36 | 0.1045 | 0.9008 | +| id_label | 8 | 4.303e+37 | 5.379e+36 | 0.4543 | 0.888 | +| arch:id_label | 16 | 4.688e+37 | 2.93e+36 | 0.2475 | 0.9989 | +| Residuals | 741 | 8.772e+39 | 1.184e+37 | — | — | + +## APIC ticks per heartbeat tick (`apic_delta`) + +![APIC ticks per heartbeat tick](svg/apic_delta.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 20 | 2.9 | 1.119 | 38.59 | +| amd64 | rajames | 22 | 2.045 | 4.18 | 204.37 | +| amd64 | 00 | 23 | 2.304 | 3.363 | 145.96 | +| amd64 | 01 | 23 | 2.739 | 1.214 | 44.33 | +| amd64 | 02 | 25 | 3.12 | 2.505 | 80.30 | +| amd64 | 03 | 27 | 3.481 | 2.737 | 78.61 | +| amd64 | 04 | 30 | 3.1 | 1.605 | 51.77 | +| amd64 | 05 | 30 | 1.767 | 6.157 | 348.51 | +| amd64 | 06 | 28 | 2.357 | 5.87 | 249.04 | +| aarch64 | zuse | 20 | 4.45 | 2.585 | 58.09 | +| aarch64 | rajames | 14 | 4.786 | 2.517 | 52.59 | +| aarch64 | 00 | 23 | 3.348 | 7.158 | 213.81 | +| aarch64 | 01 | 24 | 2.75 | 7.651 | 278.23 | +| aarch64 | 02 | 34 | 2.618 | 9.711 | 370.98 | +| aarch64 | 03 | 26 | 3.538 | 3.301 | 93.30 | +| aarch64 | 04 | 30 | 4.3 | 2.409 | 56.02 | +| aarch64 | 05 | 30 | 3.4 | 5.083 | 149.49 | +| aarch64 | 06 | 29 | 2.414 | 10.24 | 424.30 | +| riscv64 | zuse | 20 | 1.35 | 0.6708 | 49.69 | +| riscv64 | rajames | 23 | 1.435 | 2.952 | 205.71 | +| riscv64 | 00 | 24 | 1.667 | 1.761 | 105.67 | +| riscv64 | 01 | 25 | 1.2 | 2.217 | 184.78 | +| riscv64 | 02 | 24 | 1.167 | 2.599 | 222.75 | +| riscv64 | 03 | 25 | 1.16 | 2.939 | 253.40 | +| riscv64 | 04 | 29 | 1.448 | 1.572 | 108.53 | +| riscv64 | 05 | 29 | 1.207 | 3.783 | 313.47 | +| riscv64 | 06 | 31 | 1.452 | 3.414 | 235.19 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 495.9 | 247.9 | 11.21 | 1.626e-05 | +| id_label | 8 | 71.4 | 8.924 | 0.4036 | 0.9187 | +| arch:id_label | 16 | 135.9 | 8.492 | 0.384 | 0.986 | +| Residuals | 661 | 1.462e+04 | 22.11 | — | — | + +## TIME-TRUST (`time_trust`) + +![TIME-TRUST](svg/time_trust.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 0.6132 | 0 | 0.00 | +| amd64 | rajames | 25 | 0.6132 | 0 | 0.00 | +| amd64 | 00 | 26 | 0.6132 | 0 | 0.00 | +| amd64 | 01 | 26 | 0.6132 | 0 | 0.00 | +| amd64 | 02 | 28 | 0.6132 | 0 | 0.00 | +| amd64 | 03 | 30 | 0.6132 | 0 | 0.00 | +| amd64 | 04 | 33 | 0.6132 | 0 | 0.00 | +| amd64 | 05 | 33 | 0.6132 | 0 | 0.00 | +| amd64 | 06 | 31 | 0.6132 | 0 | 0.00 | +| aarch64 | zuse | 23 | 1 | 0 | 0.00 | +| aarch64 | rajames | 16 | 1 | 0 | 0.00 | +| aarch64 | 00 | 26 | 1 | 0 | 0.00 | +| aarch64 | 01 | 27 | 1 | 0 | 0.00 | +| aarch64 | 02 | 37 | 1 | 0 | 0.00 | +| aarch64 | 03 | 29 | 1 | 0 | 0.00 | +| aarch64 | 04 | 33 | 1 | 0 | 0.00 | +| aarch64 | 05 | 33 | 1 | 0 | 0.00 | +| aarch64 | 06 | 32 | 1 | 0 | 0.00 | +| riscv64 | zuse | 23 | 1 | 0 | 0.00 | +| riscv64 | rajames | 26 | 1 | 0 | 0.00 | +| riscv64 | 00 | 27 | 1 | 0 | 0.00 | +| riscv64 | 01 | 28 | 1 | 0 | 0.00 | +| riscv64 | 02 | 27 | 1 | 0 | 0.00 | +| riscv64 | 03 | 28 | 1 | 0 | 0.00 | +| riscv64 | 04 | 32 | 1 | 0 | 0.00 | +| riscv64 | 05 | 32 | 1 | 0 | 0.00 | +| riscv64 | 06 | 34 | 1 | 0 | 0.00 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 25.48 | 12.74 | 5.379e+31 | 0 | +| id_label | 8 | 1.306e-30 | 1.632e-31 | 0.6892 | 0.7014 | +| arch:id_label | 16 | 2.579e-30 | 1.612e-31 | 0.6806 | 0.8149 | +| Residuals | 741 | 1.755e-28 | 2.369e-31 | — | — | + +## Timing variance (`variance`) + +![Timing variance](svg/variance.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 0.6307 | 0 | 0.00 | +| amd64 | rajames | 25 | 0.6307 | 0 | 0.00 | +| amd64 | 00 | 26 | 0.6307 | 0 | 0.00 | +| amd64 | 01 | 26 | 0.6307 | 0 | 0.00 | +| amd64 | 02 | 28 | 0.6307 | 0 | 0.00 | +| amd64 | 03 | 30 | 0.6307 | 0 | 0.00 | +| amd64 | 04 | 33 | 0.6307 | 0 | 0.00 | +| amd64 | 05 | 33 | 0.6307 | 0 | 0.00 | +| amd64 | 06 | 31 | 0.6307 | 0 | 0.00 | +| aarch64 | zuse | 23 | 0 | 0 | — | +| aarch64 | rajames | 16 | 0 | 0 | — | +| aarch64 | 00 | 26 | 0 | 0 | — | +| aarch64 | 01 | 27 | 0 | 0 | — | +| aarch64 | 02 | 37 | 0 | 0 | — | +| aarch64 | 03 | 29 | 0 | 0 | — | +| aarch64 | 04 | 33 | 0 | 0 | — | +| aarch64 | 05 | 33 | 0 | 0 | — | +| aarch64 | 06 | 32 | 0 | 0 | — | +| riscv64 | zuse | 23 | 0 | 0 | — | +| riscv64 | rajames | 26 | 0 | 0 | — | +| riscv64 | 00 | 27 | 0 | 0 | — | +| riscv64 | 01 | 28 | 0 | 0 | — | +| riscv64 | 02 | 27 | 0 | 0 | — | +| riscv64 | 03 | 28 | 0 | 0 | — | +| riscv64 | 04 | 32 | 0 | 0 | — | +| riscv64 | 05 | 32 | 0 | 0 | — | +| riscv64 | 06 | 34 | 0 | 0 | — | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 67.75 | 33.88 | 4.293e+32 | 0 | +| id_label | 8 | 6.232e-31 | 7.79e-32 | 0.9873 | 0.4444 | +| arch:id_label | 16 | 1.249e-30 | 7.806e-32 | 0.9893 | 0.4663 | +| Residuals | 741 | 5.847e-29 | 7.89e-32 | — | — | + +## Max VM call depth (`vm_call_depth_max`) + +![Max VM call depth](svg/vm_call_depth_max.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 1 | 0 | 0.00 | +| amd64 | rajames | 25 | 0.96 | 0.2 | 20.83 | +| amd64 | 00 | 26 | 0.9615 | 0.1961 | 20.40 | +| amd64 | 01 | 26 | 0.9615 | 0.1961 | 20.40 | +| amd64 | 02 | 28 | 0.9643 | 0.189 | 19.60 | +| amd64 | 03 | 30 | 0.9667 | 0.1826 | 18.89 | +| amd64 | 04 | 33 | 0.9697 | 0.1741 | 17.95 | +| amd64 | 05 | 33 | 0.9697 | 0.1741 | 17.95 | +| amd64 | 06 | 31 | 0.9677 | 0.1796 | 18.56 | +| aarch64 | zuse | 23 | 1 | 0 | 0.00 | +| aarch64 | rajames | 16 | 1 | 0 | 0.00 | +| aarch64 | 00 | 26 | 0.9615 | 0.1961 | 20.40 | +| aarch64 | 01 | 27 | 0.963 | 0.1925 | 19.99 | +| aarch64 | 02 | 37 | 0.9459 | 0.2292 | 24.23 | +| aarch64 | 03 | 29 | 0.9655 | 0.1857 | 19.23 | +| aarch64 | 04 | 33 | 0.9697 | 0.1741 | 17.95 | +| aarch64 | 05 | 33 | 0.9394 | 0.2423 | 25.79 | +| aarch64 | 06 | 32 | 0.9688 | 0.1768 | 18.25 | +| riscv64 | zuse | 23 | 1 | 0 | 0.00 | +| riscv64 | rajames | 26 | 0.9231 | 0.2717 | 29.44 | +| riscv64 | 00 | 27 | 0.9259 | 0.2669 | 28.82 | +| riscv64 | 01 | 28 | 0.9643 | 0.189 | 19.60 | +| riscv64 | 02 | 27 | 0.963 | 0.1925 | 19.99 | +| riscv64 | 03 | 28 | 0.9643 | 0.189 | 19.60 | +| riscv64 | 04 | 32 | 0.9688 | 0.1768 | 18.25 | +| riscv64 | 05 | 32 | 0.9688 | 0.1768 | 18.25 | +| riscv64 | 06 | 34 | 0.9706 | 0.1715 | 17.67 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 0.007273 | 0.003636 | 0.1044 | 0.9009 | +| id_label | 8 | 0.1239 | 0.01549 | 0.4447 | 0.8942 | +| arch:id_label | 16 | 0.1016 | 0.006352 | 0.1823 | 0.9999 | +| Residuals | 741 | 25.82 | 0.03484 | — | — | + +## Hera fleet heat (`hera_heat`) + +![Hera fleet heat](svg/hera_heat.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 0.6532 | 0 | 0.00 | +| amd64 | rajames | 25 | 0.6532 | 0 | 0.00 | +| amd64 | 00 | 26 | 0.6532 | 0 | 0.00 | +| amd64 | 01 | 26 | 0.6532 | 0 | 0.00 | +| amd64 | 02 | 28 | 0.6532 | 0 | 0.00 | +| amd64 | 03 | 30 | 0.6532 | 0 | 0.00 | +| amd64 | 04 | 33 | 0.6532 | 0 | 0.00 | +| amd64 | 05 | 33 | 0.6532 | 0 | 0.00 | +| amd64 | 06 | 31 | 0.6532 | 0 | 0.00 | +| aarch64 | zuse | 23 | 0.6767 | 0 | 0.00 | +| aarch64 | rajames | 16 | 0.6767 | 0 | 0.00 | +| aarch64 | 00 | 26 | 0.6767 | 0 | 0.00 | +| aarch64 | 01 | 27 | 0.6767 | 0 | 0.00 | +| aarch64 | 02 | 37 | 0.6767 | 0 | 0.00 | +| aarch64 | 03 | 29 | 0.6767 | 0 | 0.00 | +| aarch64 | 04 | 33 | 0.6767 | 0 | 0.00 | +| aarch64 | 05 | 33 | 0.6767 | 0 | 0.00 | +| aarch64 | 06 | 32 | 0.6767 | 0 | 0.00 | +| riscv64 | zuse | 23 | 0.6384 | 0 | 0.00 | +| riscv64 | rajames | 26 | 0.6384 | 0 | 0.00 | +| riscv64 | 00 | 27 | 0.6384 | 0 | 0.00 | +| riscv64 | 01 | 28 | 0.6384 | 0 | 0.00 | +| riscv64 | 02 | 27 | 0.6384 | 0 | 0.00 | +| riscv64 | 03 | 28 | 0.6384 | 0 | 0.00 | +| riscv64 | 04 | 32 | 0.6384 | 0 | 0.00 | +| riscv64 | 05 | 32 | 0.6384 | 0 | 0.00 | +| riscv64 | 06 | 34 | 0.6384 | 0 | 0.00 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 0.1909 | 0.09546 | 3.106e+30 | 0 | +| id_label | 8 | 1.995e-31 | 2.494e-32 | 0.8114 | 0.5927 | +| arch:id_label | 16 | 3.935e-31 | 2.459e-32 | 0.8003 | 0.6863 | +| Residuals | 741 | 2.277e-29 | 3.073e-32 | — | — | + +## Hermes fleet heat (`hermes_heat`) + +![Hermes fleet heat](svg/hermes_heat.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 0.1692 | 0 | 0.00 | +| amd64 | rajames | 25 | 0.1692 | 0 | 0.00 | +| amd64 | 00 | 26 | 0.1692 | 0 | 0.00 | +| amd64 | 01 | 26 | 0.1692 | 0 | 0.00 | +| amd64 | 02 | 28 | 0.1692 | 0 | 0.00 | +| amd64 | 03 | 30 | 0.1692 | 0 | 0.00 | +| amd64 | 04 | 33 | 0.1692 | 0 | 0.00 | +| amd64 | 05 | 33 | 0.1692 | 0 | 0.00 | +| amd64 | 06 | 31 | 0.1692 | 0 | 0.00 | +| aarch64 | zuse | 23 | 0.1581 | 0 | 0.00 | +| aarch64 | rajames | 16 | 0.1581 | 0 | 0.00 | +| aarch64 | 00 | 26 | 0.1581 | 0 | 0.00 | +| aarch64 | 01 | 27 | 0.1581 | 0 | 0.00 | +| aarch64 | 02 | 37 | 0.1581 | 0 | 0.00 | +| aarch64 | 03 | 29 | 0.1581 | 0 | 0.00 | +| aarch64 | 04 | 33 | 0.1581 | 0 | 0.00 | +| aarch64 | 05 | 33 | 0.1581 | 0 | 0.00 | +| aarch64 | 06 | 32 | 0.1581 | 0 | 0.00 | +| riscv64 | zuse | 23 | 0.1762 | 0 | 0.00 | +| riscv64 | rajames | 26 | 0.1762 | 0 | 0.00 | +| riscv64 | 00 | 27 | 0.1762 | 0 | 0.00 | +| riscv64 | 01 | 28 | 0.1762 | 0 | 0.00 | +| riscv64 | 02 | 27 | 0.1762 | 0 | 0.00 | +| riscv64 | 03 | 28 | 0.1762 | 0 | 0.00 | +| riscv64 | 04 | 32 | 0.1762 | 0 | 0.00 | +| riscv64 | 05 | 32 | 0.1762 | 0 | 0.00 | +| riscv64 | 06 | 34 | 0.1762 | 0 | 0.00 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 0.04251 | 0.02125 | 1.491e+31 | 0 | +| id_label | 8 | 9.968e-33 | 1.246e-33 | 0.8744 | 0.5376 | +| arch:id_label | 16 | 1.967e-32 | 1.229e-33 | 0.8627 | 0.6132 | +| Residuals | 741 | 1.056e-30 | 1.425e-33 | — | — | + +## Artemis fleet heat (`artemis_heat`) + +![Artemis fleet heat](svg/artemis_heat.svg) + +| Architecture | Identity | n | mean | sd | CV% | +|---|---|---:|---:|---:|---:| +| amd64 | zuse | 23 | 1.545e+17 | 5.118e+17 | 331.32 | +| amd64 | rajames | 25 | 1.421e+17 | 4.919e+17 | 346.11 | +| amd64 | 00 | 26 | 2.05e+17 | 5.788e+17 | 282.37 | +| amd64 | 01 | 26 | 4.1e+08 | 7.633e+08 | 186.19 | +| amd64 | 02 | 28 | 1.269e+17 | 4.659e+17 | 367.17 | +| amd64 | 03 | 30 | 5.922e+16 | 3.244e+17 | 547.72 | +| amd64 | 04 | 33 | 1.077e+17 | 4.305e+17 | 399.80 | +| amd64 | 05 | 33 | 1.615e+17 | 5.186e+17 | 321.13 | +| amd64 | 06 | 31 | 1.719e+17 | 5.339e+17 | 310.56 | +| aarch64 | zuse | 23 | 7.184e+16 | 3.445e+17 | 479.58 | +| aarch64 | rajames | 16 | 2.065e+17 | 5.643e+17 | 273.25 | +| aarch64 | 00 | 26 | 1.906e+17 | 5.383e+17 | 282.37 | +| aarch64 | 01 | 27 | 3.671e+08 | 6.999e+08 | 190.65 | +| aarch64 | 02 | 37 | 8.931e+16 | 3.788e+17 | 424.10 | +| aarch64 | 03 | 29 | 5.697e+16 | 3.068e+17 | 538.52 | +| aarch64 | 04 | 33 | 1.001e+17 | 4.003e+17 | 399.80 | +| aarch64 | 05 | 33 | 1.502e+17 | 4.823e+17 | 321.13 | +| aarch64 | 06 | 32 | 1.549e+17 | 4.893e+17 | 315.89 | +| riscv64 | zuse | 23 | 8.062e+16 | 3.866e+17 | 479.58 | +| riscv64 | rajames | 26 | 2.853e+08 | 6.822e+08 | 239.16 | +| riscv64 | 00 | 27 | 1.374e+17 | 4.949e+17 | 360.29 | +| riscv64 | 01 | 28 | 3.973e+08 | 7.748e+08 | 195.00 | +| riscv64 | 02 | 27 | 6.868e+16 | 3.568e+17 | 519.62 | +| riscv64 | 03 | 28 | 6.622e+16 | 3.504e+17 | 529.15 | +| riscv64 | 04 | 32 | 5.794e+16 | 3.278e+17 | 565.69 | +| riscv64 | 05 | 32 | 1.159e+17 | 4.56e+17 | 393.50 | +| riscv64 | 06 | 34 | 1.636e+17 | 5.338e+17 | 326.29 | + +Two-way ANOVA (`metric ~ arch * id_label`): + +| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) | +|---|---:|---:|---:|---:|---:| +| arch | 2 | 2.807e+35 | 1.404e+35 | 0.801 | 0.4492 | +| id_label | 8 | 1.988e+36 | 2.486e+35 | 1.419 | 0.1849 | +| arch:id_label | 16 | 4.978e+35 | 3.111e+34 | 0.1776 | 0.9999 | +| Residuals | 741 | 1.298e+38 | 1.752e+35 | — | — | + diff --git a/experiments/std79-doe/analysis-20260912/README.md b/experiments/std79-doe/analysis-20260912/README.md new file mode 100644 index 00000000..a4d344d5 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/README.md @@ -0,0 +1,55 @@ +# std79-doe 3×9 factorial analysis (heartbeat/physics telemetry) + +Read `ANALYSIS.md` for the report itself (key findings, per-metric cell means, +two-way ANOVA tables, boxplot SVGs). This file just documents the pipeline. + +## Pipeline + +1. **Source data:** `../results-20260912-with-heartbeat-csv/{amd64,aarch64,riscv64}-doe-raw.log` + — raw QEMU serial logs from the `HB-ON`/`HB-OFF`-instrumented std79 DoE campaign + (see `../README.md` and FABRIC-3.md §XV/§XVI/§XVII for the campaign itself). + +2. **`correlate_doe.py `** — reconstructs which DoE trial + (`run_id`, `id_idx`, `id_label`, `rep`) was active for every heartbeat-tick CSV + row, despite the async tick printer splicing rows mid-token into the trial + loop's own console output on the shared serial line (see the script's own + docstring for the two-pass reconstruction method, and its `RUN_ID_MAP` for + why identity/rep are looked up from a known-clean prior run rather than + re-parsed from each log — one aarch64 marker was unrecoverable, see below). + Run once per architecture: + + ``` + python3 correlate_doe.py ../results-20260912-with-heartbeat-csv/amd64-doe-raw.log amd64-correlated.csv + python3 correlate_doe.py ../results-20260912-with-heartbeat-csv/aarch64-doe-raw.log aarch64-correlated.csv + python3 correlate_doe.py ../results-20260912-with-heartbeat-csv/riscv64-doe-raw.log riscv64-correlated.csv + ``` + +3. **`combine.py [in2.csv ...] `** — merges the three + per-architecture correlated CSVs into `combined.csv` (committed, 768 rows), + decoding Q48.16 fixed-point fields to plain floats and `jitter_bits`' raw + IEEE754 bit pattern to a real `jitter_ns` value: + + ``` + python3 combine.py amd64-correlated.csv aarch64-correlated.csv riscv64-correlated.csv combined.csv + ``` + +4. **`analysis.R`** (run from this directory) — reads `combined.csv`, computes + per-cell (architecture × identity) summary statistics and a two-way ANOVA + for each of the 12 varying telemetry metrics, renders one boxplot SVG per + metric to `svg/`, and writes `ANALYSIS.md`: + + ``` + Rscript analysis.R + ``` + + Requires `dplyr`, `tidyr`, `ggplot2`, `svglite` (all present in this + environment already). + +## Known limitation + +aarch64's raw log lost 1 of 27 `DOE-RUN` markers to interleaving beyond +recovery — trial 12 (identity `rajames`, rep 1). Its rows fold into trial 11 +(identity `rajames`, rep 0) in `combined.csv`, so aarch64's `rajames` cell +isn't perfectly separable between those two replicates specifically. Every +other cell, on every architecture, is unaffected — see `ANALYSIS.md`'s own +note and `correlate_doe.py`'s docstring for the full explanation. diff --git a/experiments/std79-doe/analysis-20260912/analysis.R b/experiments/std79-doe/analysis-20260912/analysis.R new file mode 100644 index 00000000..f1fd28ab --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/analysis.R @@ -0,0 +1,243 @@ +#!/usr/bin/env Rscript +# analysis.R -- 3(architecture) x 9(identity) factorial analysis of the +# std79-doe heartbeat/physics telemetry captured via HB-ON/HB-OFF +# (see ../README.md's results-20260912-with-heartbeat-csv/ entry). +# +# Input: combined.csv (this directory) -- one row per heartbeat tick, +# tagged with which (arch, run_id, id_idx, id_label, rep) trial was +# active when that tick's CSV row was printed, built by +# correlate_doe.py from the three raw logs in +# ../results-20260912-with-heartbeat-csv/. Q48.16 fields are already +# decoded to plain floats (avg_word_heat, time_trust, variance, +# hera/hermes/artemis_heat) and jitter_bits to jitter_ns (IEEE754 +# bit-pattern reinterpretation), both done in Python before this script +# runs, since R has no native uint64/bit-punning story worth using here. +# +# Output: ANALYSIS.md (this directory) + svg/*.svg (referenced from it). + +suppressMessages({ + library(dplyr) + library(tidyr) + library(ggplot2) + library(svglite) +}) + +set.seed(1) +# Run from this directory (contains combined.csv); the caller cd's here. + +df <- read.csv("combined.csv", stringsAsFactors = FALSE) + +id_levels <- c("zuse", "rajames", "00", "01", "02", "03", "04", "05", "06") +arch_levels <- c("amd64", "aarch64", "riscv64") +df$id_label <- factor(df$id_label, levels = id_levels) +df$arch <- factor(df$arch, levels = arch_levels) + +# apic_ticks is a monotonic hardware counter, not directly comparable +# across trials/architectures as a raw value -- the per-tick DELTA +# (hardware ticks consumed per heartbeat tick) is the actual timing +# signal. Computed per (arch, run_id) so a trial boundary never +# contaminates the delta of the row after it. +df <- df %>% + arrange(arch, run_id, tick_number) %>% + group_by(arch, run_id) %>% + mutate(apic_delta = apic_ticks - lag(apic_ticks)) %>% + ungroup() + +metrics <- c( + "hot_word_count", "avg_word_heat", "window_width", "actual_window_size", + "jitter_ns", "apic_delta", "time_trust", "variance", "vm_call_depth_max", + "hera_heat", "hermes_heat", "artemis_heat" +) + +metric_labels <- c( + hot_word_count = "Hot word count", + avg_word_heat = "Mean word execution heat", + window_width = "Rolling window width", + actual_window_size = "Actual analysis window size", + jitter_ns = "Estimated timer jitter (ns)", + apic_delta = "APIC ticks per heartbeat tick", + time_trust = "TIME-TRUST", + variance = "Timing variance", + vm_call_depth_max = "Max VM call depth", + hera_heat = "Hera fleet heat", + hermes_heat = "Hermes fleet heat", + artemis_heat = "Artemis fleet heat" +) + +dir.create("svg", showWarnings = FALSE) + +# ---- Per-cell (arch x id_label) summary table for one metric ---- +cell_summary <- function(metric) { + df %>% + filter(!is.na(.data[[metric]])) %>% + group_by(arch, id_label) %>% + summarise( + n = n(), + mean = mean(.data[[metric]]), + sd = sd(.data[[metric]]), + cv_pct = ifelse(mean != 0, 100 * sd / abs(mean), NA_real_), + .groups = "drop" + ) +} + +# ---- Two-way ANOVA (metric ~ arch * id_label), safe on zero-variance ---- +anova_table <- function(metric) { + d <- df %>% filter(!is.na(.data[[metric]])) + if (length(unique(d[[metric]])) <= 1) { + return(NULL) + } + fit <- tryCatch( + aov(as.formula(paste0(metric, " ~ arch * id_label")), data = d), + error = function(e) NULL + ) + if (is.null(fit)) return(NULL) + s <- summary(fit)[[1]] + s +} + +# ---- Boxplot: metric by id_label, faceted by arch ---- +make_plot <- function(metric) { + d <- df %>% filter(!is.na(.data[[metric]])) + if (length(unique(d[[metric]])) <= 1) return(NULL) + p <- ggplot(d, aes(x = id_label, y = .data[[metric]], fill = id_label)) + + geom_boxplot(outlier.size = 0.6, alpha = 0.85) + + facet_wrap(~arch, ncol = 1) + + labs( + title = metric_labels[[metric]], + x = "Identity", y = metric_labels[[metric]] + ) + + theme_bw(base_size = 11) + + theme(legend.position = "none", axis.text.x = element_text(angle = 45, hjust = 1)) + fname <- file.path("svg", paste0(metric, ".svg")) + ggsave(fname, p, width = 7, height = 8, device = svglite) + fname +} + +# ---- Run everything, collect markdown fragments ---- +md <- c() +md <- c(md, "# std79-doe 3×9 Factorial Analysis — Heartbeat/Physics Telemetry") +md <- c(md, "") +md <- c(md, sprintf("Generated %s. Source: `combined.csv` (%d rows), built by", format(Sys.time(), "%Y-%m-%d %H:%M"), nrow(df))) +md <- c(md, "`correlate_doe.py` from the three raw logs in") +md <- c(md, "`../results-20260912-with-heartbeat-csv/`. See that file's own docstring for the") +md <- c(md, "correlation methodology (the heartbeat tick's async CSV printer and the DoE trial") +md <- c(md, "loop's own console output share one serial line with no locking, so rows can land") +md <- c(md, "mid-token in the raw logs; the underlying FORTH execution and values are") +md <- c(md, "unaffected, only reconstructing which trial owns which row needed care).") +md <- c(md, "") +md <- c(md, "**Design:** 3 architectures (amd64, aarch64, riscv64) × 9 identities (zuse,") +md <- c(md, "rajames, 00-06) × 3 replicates, Fisher-Yates-shuffled run order, same seed") +md <- c(md, "(12345) on every architecture (FABRIC-3.md §XV). Each of the 27 trials per") +md <- c(md, "architecture runs the same 24-word FORTH-79 exerciser; every heartbeat tick during") +md <- c(md, "the whole `EXEC-STD79-DOE` run (bracketed by `HB-ON`/`HB-OFF`) emits one telemetry") +md <- c(md, "row (`doe_log.c`), correlated back to whichever trial was active when it printed.") +md <- c(md, "") +md <- c(md, "**Known limitation:** aarch64's raw log lost 1 of 27 run_id markers to interleaving") +md <- c(md, "beyond recovery (trial 12, identity `rajames` rep 1) -- its rows are folded into") +md <- c(md, "trial 11 (identity `rajames` rep 0) in this dataset, so aarch64's `rajames` cell") +md <- c(md, "for rep-sensitive metrics is not perfectly separable for those two replicates") +md <- c(md, "specifically. Every other cell on every architecture is unaffected.") +md <- c(md, "") + +# ---- Pass 1: compute everything, so the "Key findings" summary (which +# needs every metric's ANOVA result) can be written before the detailed +# per-metric sections that follow it. ---- +results <- list() +for (metric in metrics) { + results[[metric]] <- list( + cs = cell_summary(metric), + at = anova_table(metric), + plot_file = make_plot(metric) + ) +} + +sig <- function(p) !is.na(p) && p < 0.05 + +arch_sig <- c(); id_sig <- c(); inter_sig <- c(); none_sig <- c() +for (metric in metrics) { + at <- results[[metric]]$at + if (is.null(at)) next + p_arch <- if ("arch" %in% trimws(rownames(at))) at[trimws(rownames(at)) == "arch", "Pr(>F)"] else NA + p_id <- if ("id_label" %in% trimws(rownames(at))) at[trimws(rownames(at)) == "id_label", "Pr(>F)"] else NA + p_int <- if ("arch:id_label" %in% trimws(rownames(at))) at[trimws(rownames(at)) == "arch:id_label", "Pr(>F)"] else NA + label <- metric_labels[[metric]] + any_sig <- FALSE + if (sig(p_arch)) { arch_sig <- c(arch_sig, label); any_sig <- TRUE } + if (sig(p_id)) { id_sig <- c(id_sig, label); any_sig <- TRUE } + if (sig(p_int)) { inter_sig <- c(inter_sig, label); any_sig <- TRUE } + if (!any_sig) none_sig <- c(none_sig, label) +} + +md <- c(md, "## Key findings (p < 0.05, two-way ANOVA)") +md <- c(md, "") +md <- c(md, sprintf("- **Architecture main effect:** %s", + if (length(arch_sig)) paste(arch_sig, collapse = ", ") else "none")) +md <- c(md, sprintf("- **Identity main effect:** %s", + if (length(id_sig)) paste(id_sig, collapse = ", ") else "none")) +md <- c(md, sprintf("- **Architecture × identity interaction:** %s", + if (length(inter_sig)) paste(inter_sig, collapse = ", ") else "none")) +md <- c(md, sprintf("- **No significant effect of either factor:** %s", + if (length(none_sig)) paste(none_sig, collapse = ", ") else "none")) +md <- c(md, "") +md <- c(md, "TIME-TRUST's architecture effect is a numerical artifact worth reading correctly,") +md <- c(md, "not a substantive result: it is essentially constant *within* each architecture") +md <- c(md, "(residual variance ~1e-31, i.e. floating-point noise) and differs *between*") +md <- c(md, "architectures — so the F statistic is enormous simply because the within-group") +md <- c(md, "denominator is near zero, not because TIME-TRUST is meaningfully more variable") +md <- c(md, "across architectures than the other metrics here. It is architecture-determined") +md <- c(md, "and identity-independent, which is itself the interesting part.") +md <- c(md, "") + +for (metric in metrics) { + cs <- results[[metric]]$cs + at <- results[[metric]]$at + plot_file <- results[[metric]]$plot_file + + md <- c(md, sprintf("## %s (`%s`)", metric_labels[[metric]], metric)) + md <- c(md, "") + + if (is.null(plot_file)) { + md <- c(md, "_Constant across every trial and architecture — no variance, excluded from ANOVA/plot._") + md <- c(md, "") + next + } + + md <- c(md, sprintf("![%s](%s)", metric_labels[[metric]], plot_file)) + md <- c(md, "") + md <- c(md, "| Architecture | Identity | n | mean | sd | CV% |") + md <- c(md, "|---|---|---:|---:|---:|---:|") + for (i in seq_len(nrow(cs))) { + r <- cs[i, ] + md <- c(md, sprintf("| %s | %s | %d | %.4g | %.4g | %s |", + r$arch, r$id_label, r$n, r$mean, r$sd, + ifelse(is.na(r$cv_pct), "—", sprintf("%.2f", r$cv_pct)))) + } + md <- c(md, "") + + if (!is.null(at)) { + md <- c(md, "Two-way ANOVA (`metric ~ arch * id_label`):") + md <- c(md, "") + md <- c(md, "| Term | Df | Sum Sq | Mean Sq | F value | Pr(>F) |") + md <- c(md, "|---|---:|---:|---:|---:|---:|") + terms <- rownames(at) + for (i in seq_len(nrow(at))) { + row <- at[i, ] + # %g-style: near-zero within-cell residual variance (e.g. a metric + # that's essentially constant within an architecture, only varying + # BETWEEN architectures -- TIME-TRUST does exactly this) can blow + # the F statistic up to absurd magnitudes; format generally rather + # than with a fixed decimal count so that renders sanely too. + fval <- if (!is.na(row[["F value"]])) formatC(row[["F value"]], format = "g", digits = 4) else "—" + pval <- if (!is.na(row[["Pr(>F)"]])) formatC(row[["Pr(>F)"]], format = "g", digits = 4) else "—" + md <- c(md, sprintf("| %s | %d | %.4g | %.4g | %s | %s |", + trimws(terms[i]), row[["Df"]], row[["Sum Sq"]], row[["Mean Sq"]], fval, pval)) + } + md <- c(md, "") + } else { + md <- c(md, "_ANOVA not computed (insufficient variance)._") + md <- c(md, "") + } +} + +writeLines(md, "ANALYSIS.md") +cat("Wrote ANALYSIS.md and svg/*.svg\n") diff --git a/experiments/std79-doe/analysis-20260912/combine.py b/experiments/std79-doe/analysis-20260912/combine.py new file mode 100644 index 00000000..9b987ce0 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/combine.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +""" +combine.py -- build combined.csv from the three per-architecture +correlated CSVs (correlate_doe.py's output), decoding Q48.16 fixed-point +fields to plain floats and jitter_bits' IEEE754 bit pattern to a real +jitter_ns value, since R has no convenient native uint64/bit-punning +story worth using here. + +Usage (from this directory, after running correlate_doe.py for each +architecture against ../results-20260912-with-heartbeat-csv/*-doe-raw.log +to produce amd64-correlated.csv / aarch64-correlated.csv / +riscv64-correlated.csv): + + python3 combine.py amd64-correlated.csv aarch64-correlated.csv riscv64-correlated.csv combined.csv +""" +import csv +import struct +import sys + +Q48 = 65536.0 + +def main(): + if len(sys.argv) < 3: + print("usage: combine.py [in2.csv ...] ", file=sys.stderr) + sys.exit(1) + *in_paths, out_path = sys.argv[1:] + + fields = ["arch", "run_id", "id_idx", "id_label", "rep", "tick_number", "elapsed_ns", + "tick_interval_ns", "hot_word_count", "avg_word_heat", "window_width", + "actual_window_size", "predicted_label_hits", "jitter_ns", "apic_ticks", + "time_trust", "variance", "vm_call_depth_max", "hera_heat", "hermes_heat", + "artemis_heat"] + + out_rows = [] + for path in in_paths: + arch = path.split("-correlated.csv")[0].split("/")[-1] + with open(path) as f: + for row in csv.DictReader(f): + jitter_bits = int(row["jitter_bits"]) + jitter_ns = struct.unpack(' + +Reconstructs which DoE trial (run_id, id_idx, id_label, rep) was active +for every heartbeat-tick CSV row in a std79-doe raw log, despite the +async HB-ON tick printer splicing CSV rows mid-token into the trial +loop's own console output on the shared serial line -- including mid a +DOE-RUN,... marker itself. + +Method: + 1. Strip ANSI codes and the per-line "[VM-TAG] " console prefix (an + artifact of the log capture layer), concatenate every stripped + line with no separator. Find every [HADES][DOE ] header/row + occurrence and physically remove exactly those spans, recording + each removed data row's insertion offset in the CLEANED stream's + own coordinates. + 2. Parse only "DOE-RUN, ," (the run_id digits) from the clean + stream -- reliable even when a CSV row spliced into the id_idx/ + id_label/rep text right after it (observed live: a row's last + digit field can land with zero separator against the immediately + following trial digit, e.g. "...11642" + "6" -> "116426", making + id_idx unrecoverable from THIS log alone for the affected trial). + id_idx/id_label/rep are looked up from RUN_ID_MAP below instead of + re-parsed -- deterministic given the campaign's fixed seed (12345), + independently verified identical across 5+ prior clean runs + (results-20260911-donor-floor-fix/amd64-doe-raw.log and others). + 3. Assign each CSV row to whichever run_id marker's start offset is + the largest one at or before that row's insertion offset. +""" +import re +import sys +import csv +import bisect + +# From results-20260911-donor-floor-fix/amd64-doe-raw.log (seed 12345), +# independently re-verified identical (md5) across every std79-doe run +# since e2abc56. run_id -> (id_idx, id_label, rep). +RUN_ID_MAP = { + 0: (6, "04", 2), 1: (3, "01", 1), 2: (0, "zuse", 1), 3: (1, "rajames", 0), + 4: (5, "03", 0), 5: (0, "zuse", 2), 6: (2, "00", 2), 7: (8, "06", 2), + 8: (0, "zuse", 0), 9: (6, "04", 1), 10: (6, "04", 0), 11: (4, "02", 2), + 12: (1, "rajames", 1), 13: (7, "05", 1), 14: (7, "05", 2), 15: (4, "02", 0), + 16: (2, "00", 0), 17: (8, "06", 0), 18: (7, "05", 0), 19: (2, "00", 1), + 20: (8, "06", 1), 21: (5, "03", 2), 22: (1, "rajames", 2), 23: (3, "01", 2), + 24: (5, "03", 1), 25: (3, "01", 0), 26: (4, "02", 1), +} + +def build_stream(raw_text): + parts = [] + for line in raw_text.split('\n'): + m = re.match(r'^\[[^\]]+\]\s?(.*)$', line) + parts.append(m.group(1) if m else line) + return ''.join(parts) + +ROW_RE = re.compile( + r'\[HADES\]\[DOE \] ' + r'(?:(?P
tick_number,elapsed_ns,[^\[]*?)(?=\[HADES\]\[DOE \]|DOE-RUN|$)' + r'|(?P[0-9]+(?:,[0-9]+)+))' +) +RUN_ID_RE = re.compile(r'DOE-RUN,\s*([0-9]+)\s*,') + +def main(): + if len(sys.argv) != 3: + print("usage: correlate_doe.py ", file=sys.stderr) + sys.exit(1) + log_path, out_path = sys.argv[1], sys.argv[2] + + with open(log_path, "r", errors="replace") as f: + raw = f.read() + raw = re.sub(r'\x1b\[[0-9;]*[a-zA-Z]', '', raw) + stream = build_stream(raw) + + cleaned_parts = [] + cleaned_offset = 0 + last_end = 0 + csv_rows = [] # (cleaned_insertion_offset, fields) + header_fields = None + for m in ROW_RE.finditer(stream): + before = stream[last_end:m.start()] + cleaned_parts.append(before) + cleaned_offset += len(before) + if m.group('header') is not None: + header_fields = [x.strip() for x in m.group('header').split(',')] + else: + csv_rows.append((cleaned_offset, m.group('row').split(','))) + last_end = m.end() + cleaned_parts.append(stream[last_end:]) + cleaned_stream = ''.join(cleaned_parts) + + if header_fields is None: + print("FATAL: never found the DOE CSV header row", file=sys.stderr) + sys.exit(1) + + markers = [(m.start(), int(m.group(1))) for m in RUN_ID_RE.finditer(cleaned_stream)] + # Sanity: run_ids should appear in increasing offset order with values + # 0..26 each exactly once. Warn, don't crash, on anything unexpected -- + # a genuinely dropped/garbled run_id digit is possible in principle. + seen = [rid for _, rid in markers] + if sorted(set(seen)) != list(range(27)) or len(seen) != 27: + print(f"WARNING: expected exactly run_ids 0..26 once each, got {sorted(seen)}", file=sys.stderr) + + marker_offsets = [off for off, _ in markers] + + out_rows = [] + unlabeled = 0 + for offset, fields in csv_rows: + i = bisect.bisect_right(marker_offsets, offset) - 1 + if i >= 0: + run_id = markers[i][1] + id_idx, id_label, rep = RUN_ID_MAP.get(run_id, (None, None, None)) + elif 0 not in seen: + # No marker precedes this row because run_id 0's own marker + # (always the very first trial, loop index 0) got garbled + # beyond recovery in this particular log -- safe to assume + # every row before the first RECOVERED marker still belongs + # to trial 0, not truly unlabeled. + run_id = 0 + id_idx, id_label, rep = RUN_ID_MAP[0] + else: + run_id, id_idx, id_label, rep = None, None, None, None + unlabeled += 1 + out_rows.append(((run_id, id_idx, id_label, rep), fields)) + + with open(out_path, "w", newline="") as f: + w = csv.writer(f) + w.writerow(["run_id", "id_idx", "id_label", "rep"] + header_fields) + for (run_id, id_idx, id_label, rep), fields in out_rows: + w.writerow([run_id, id_idx, id_label, rep] + fields) + + print(f"trial markers found: {len(markers)}") + print(f"data rows: {len(out_rows)}, unlabeled (before first marker): {unlabeled}") + labeled_run_ids = sorted(set(t[0] for (t, _) in out_rows if t[0] is not None)) + print(f"distinct run_ids covered: {len(labeled_run_ids)} -> {labeled_run_ids}") + +if __name__ == "__main__": + main() diff --git a/experiments/std79-doe/analysis-20260912/svg/actual_window_size.svg b/experiments/std79-doe/analysis-20260912/svg/actual_window_size.svg new file mode 100644 index 00000000..4c4aa20f --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/actual_window_size.svg @@ -0,0 +1,313 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +400 +600 +800 +1000 + + + + +400 +600 +800 +1000 + + + + +400 +600 +800 +1000 + + + + +Identity +Actual analysis window size +Actual analysis window size + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/apic_delta.svg b/experiments/std79-doe/analysis-20260912/svg/apic_delta.svg new file mode 100644 index 00000000..045ad80c --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/apic_delta.svg @@ -0,0 +1,371 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +-50 +-25 +0 +25 + + + + +-50 +-25 +0 +25 + + + + +-50 +-25 +0 +25 + + + + +Identity +APIC ticks per heartbeat tick +APIC ticks per heartbeat tick + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/artemis_heat.svg b/experiments/std79-doe/analysis-20260912/svg/artemis_heat.svg new file mode 100644 index 00000000..bea33418 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/artemis_heat.svg @@ -0,0 +1,465 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +0.0e+00 +5.0e+17 +1.0e+18 +1.5e+18 + + + + +0.0e+00 +5.0e+17 +1.0e+18 +1.5e+18 + + + + +0.0e+00 +5.0e+17 +1.0e+18 +1.5e+18 + + + + +Identity +Artemis fleet heat +Artemis fleet heat + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/avg_word_heat.svg b/experiments/std79-doe/analysis-20260912/svg/avg_word_heat.svg new file mode 100644 index 00000000..6a690813 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/avg_word_heat.svg @@ -0,0 +1,308 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +0.00 +0.02 +0.04 +0.06 + + + + +0.00 +0.02 +0.04 +0.06 + + + + +0.00 +0.02 +0.04 +0.06 + + + + +Identity +Mean word execution heat +Mean word execution heat + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/hera_heat.svg b/experiments/std79-doe/analysis-20260912/svg/hera_heat.svg new file mode 100644 index 00000000..5e5dcee3 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/hera_heat.svg @@ -0,0 +1,302 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +0.64 +0.65 +0.66 +0.67 + + + + +0.64 +0.65 +0.66 +0.67 + + + + +0.64 +0.65 +0.66 +0.67 + + + + +Identity +Hera fleet heat +Hera fleet heat + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/hermes_heat.svg b/experiments/std79-doe/analysis-20260912/svg/hermes_heat.svg new file mode 100644 index 00000000..6a3a3336 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/hermes_heat.svg @@ -0,0 +1,302 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +0.160 +0.165 +0.170 +0.175 + + + + +0.160 +0.165 +0.170 +0.175 + + + + +0.160 +0.165 +0.170 +0.175 + + + + +Identity +Hermes fleet heat +Hermes fleet heat + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/hot_word_count.svg b/experiments/std79-doe/analysis-20260912/svg/hot_word_count.svg new file mode 100644 index 00000000..d09fad93 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/hot_word_count.svg @@ -0,0 +1,364 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +30 +40 +50 +60 +70 + + + + + +30 +40 +50 +60 +70 + + + + + +30 +40 +50 +60 +70 + + + + + +Identity +Hot word count +Hot word count + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/jitter_ns.svg b/experiments/std79-doe/analysis-20260912/svg/jitter_ns.svg new file mode 100644 index 00000000..37d42129 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/jitter_ns.svg @@ -0,0 +1,356 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +0.0e+00 +5.0e+18 +1.0e+19 +1.5e+19 + + + + +0.0e+00 +5.0e+18 +1.0e+19 +1.5e+19 + + + + +0.0e+00 +5.0e+18 +1.0e+19 +1.5e+19 + + + + +Identity +Estimated timer jitter (ns) +Estimated timer jitter (ns) + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/time_trust.svg b/experiments/std79-doe/analysis-20260912/svg/time_trust.svg new file mode 100644 index 00000000..0380bbcd --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/time_trust.svg @@ -0,0 +1,311 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +0.6 +0.7 +0.8 +0.9 +1.0 + + + + + +0.6 +0.7 +0.8 +0.9 +1.0 + + + + + +0.6 +0.7 +0.8 +0.9 +1.0 + + + + + +Identity +TIME-TRUST +TIME-TRUST + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/variance.svg b/experiments/std79-doe/analysis-20260912/svg/variance.svg new file mode 100644 index 00000000..3a5a7ff3 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/variance.svg @@ -0,0 +1,299 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +0.0 +0.2 +0.4 +0.6 + + + + +0.0 +0.2 +0.4 +0.6 + + + + +0.0 +0.2 +0.4 +0.6 + + + + +Identity +Timing variance +Timing variance + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/vm_call_depth_max.svg b/experiments/std79-doe/analysis-20260912/svg/vm_call_depth_max.svg new file mode 100644 index 00000000..668a6cf1 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/vm_call_depth_max.svg @@ -0,0 +1,338 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +0.00 +0.25 +0.50 +0.75 +1.00 + + + + + +0.00 +0.25 +0.50 +0.75 +1.00 + + + + + +0.00 +0.25 +0.50 +0.75 +1.00 + + + + + +Identity +Max VM call depth +Max VM call depth + + + diff --git a/experiments/std79-doe/analysis-20260912/svg/window_width.svg b/experiments/std79-doe/analysis-20260912/svg/window_width.svg new file mode 100644 index 00000000..bd72aec8 --- /dev/null +++ b/experiments/std79-doe/analysis-20260912/svg/window_width.svg @@ -0,0 +1,313 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +riscv64 + + + + + + + + + + +aarch64 + + + + + + + + + + +amd64 + + + + + + + + + + + +zuse +rajames +00 +01 +02 +03 +04 +05 +06 +400 +600 +800 +1000 + + + + +400 +600 +800 +1000 + + + + +400 +600 +800 +1000 + + + + +Identity +Rolling window width +Rolling window width + + +