63 lines
3.0 KiB
TeX
63 lines
3.0 KiB
TeX
%% SCRAP: experiments/02-experiments/README
|
|
%% SOURCE: docs/working/experiments/02-experiments/README.md
|
|
%% STATUS: CURRENT
|
|
%% FITS: experiments/ch-framework
|
|
%% EDITORIAL: lifted — prose rewritten to press voice
|
|
|
|
\section{Experimental Framework Overview}
|
|
|
|
StarForth's experimental programme applies rigorous design-of-experiments methodology
|
|
to the physics-driven adaptive runtime, yielding a body of evidence that grounds
|
|
every performance claim in reproducible, statistically validated data. Four
|
|
experimental series structure the work.
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Factorial DoE} — Full $2^k$ factorial designs sweep all binary
|
|
combinations of feedback-loop configurations, revealing main effects and
|
|
interaction terms that single-factor studies cannot detect.
|
|
\item \textbf{Heartbeat DoE} — Focused experiments on the heartbeat subsystem
|
|
measure temporal stability, load-response coupling, and convergence
|
|
dynamics of the adaptive tick mechanism.
|
|
\item \textbf{James Law} — Window-scaling experiments validate the conservation
|
|
invariant $\Lambda \times (\mathrm{DoF}+1) = W$ across multiple window
|
|
sizes, degrees of freedom, and workload shapes.
|
|
\item \textbf{Physics Optimization} — Targeted parameter sweeps quantify the
|
|
marginal contribution of individual physics-engine parameters, establishing
|
|
a tuning baseline for production builds.
|
|
\end{itemize}
|
|
|
|
\subsection{Experiment Lifecycle}
|
|
|
|
Every experiment follows a five-phase protocol:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Design} — state the hypothesis, identify factors, and specify
|
|
response variables before collecting any data.
|
|
\item \textbf{Protocol} — write a formal execution guide documenting methodology
|
|
and success criteria.
|
|
\item \textbf{Execute} — run the experiment and collect data without mid-course
|
|
adjustments.
|
|
\item \textbf{Analyse} — apply statistical methods (ANOVA, Levene's test,
|
|
credible-interval estimation) to the collected measurements.
|
|
\item \textbf{Document} — write a summary and promote publication-ready results
|
|
to the research archive.
|
|
\end{enumerate}
|
|
|
|
\subsection{Key Result: Deterministic Behaviour}
|
|
|
|
Across 90 experimental runs in the physics-engine validation campaign, StarForth
|
|
achieves \textbf{0\% algorithmic variance}, confirming formally proven deterministic
|
|
behaviour of the physics-driven adaptive runtime. The $2^7$ factorial campaign
|
|
extends this over 38{,}400 runs with consistent convergence to the lowest-CV
|
|
steady state.
|
|
|
|
\subsection{Statistical Methodology}
|
|
|
|
Each campaign employs factorial designs for multi-factor analysis, heartbeat-driven
|
|
data collection at millisecond resolution, and statistical validation via ANOVA and
|
|
Levene's test. Experimental protocols are reproducible: run scripts, R analysis
|
|
code, and raw CSV data are archived alongside results.
|
|
|
|
%% TODO(bob): confirm canonical published path for the DoE metrics schema and
|
|
%% publication-results documents referenced in the source
|