132 lines
5.0 KiB
TeX
132 lines
5.0 KiB
TeX
%% SCRAP: experiments/02-experiments/factorial-doe/guide
|
|
%% SOURCE: docs/working/experiments/02-experiments/factorial-doe/guide.md
|
|
%% STATUS: CURRENT
|
|
%% FITS: experiments/ch-factorial, cookbook/ch-doe
|
|
%% EDITORIAL: lifted — prose rewritten to press voice
|
|
|
|
\section{Complete $2^6$ Factorial Design of Experiments}
|
|
\label{sec:factorial-doe-guide}
|
|
|
|
\subsection{Design Rationale}
|
|
|
|
Incremental tuning — enabling one feedback loop, measuring, then enabling a
|
|
second — cannot reveal interaction effects. Two loops that each contribute
|
|
a 3\% improvement in isolation may deliver 10\% when combined, or may cancel
|
|
each other. The $2^6$ factorial design addresses this by testing \emph{all}
|
|
64 binary combinations of the six feedback loops in a single randomised
|
|
experiment, collecting a unified dataset for post-hoc analysis.
|
|
|
|
\subsection{The Six Feedback Loops}
|
|
|
|
\begin{center}
|
|
\begin{tabular}{clp{7cm}}
|
|
\toprule
|
|
Loop & Makefile flag & Purpose \\
|
|
\midrule
|
|
L1 & \texttt{ENABLE\_LOOP\_1\_HEAT\_TRACKING} & Count word execution frequency \\
|
|
L2 & \texttt{ENABLE\_LOOP\_2\_ROLLING\_WINDOW} & Capture execution history (circular buffer) \\
|
|
L3 & \texttt{ENABLE\_LOOP\_3\_LINEAR\_DECAY} & Age words by exponential decay over time \\
|
|
L4 & \texttt{ENABLE\_LOOP\_4\_PIPELINING\_METRICS} & Track word-to-word transitions \\
|
|
L5 & \texttt{ENABLE\_LOOP\_5\_WINDOW\_INFERENCE} & Infer optimal window width via Levene's test \\
|
|
L6 & \texttt{ENABLE\_LOOP\_6\_DECAY\_INFERENCE} & Infer decay slope via linear regression \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{center}
|
|
|
|
\subsection{Configuration Naming}
|
|
|
|
Each configuration is named by a six-digit binary string \texttt{L1\_L2\_L3\_L4\_L5\_L6}
|
|
where each digit is 0 (off) or 1 (on). The baseline \texttt{000000} represents
|
|
pure FORTH-79 with no adaptive physics; \texttt{111111} enables all loops.
|
|
|
|
\subsection{Execution}
|
|
|
|
Three run levels are available:
|
|
|
|
\begin{lstlisting}[language=bash]
|
|
# Minimal validation (~30-45 min, 64 builds x 1 run)
|
|
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_RUN
|
|
|
|
# Standard production (~2-4 hours, 64 x 30 = 1,920 runs)
|
|
./scripts/run_factorial_doe.sh --runs-per-config 30 FULL_FACTORIAL
|
|
|
|
# High-precision overnight (~6-12 hours, 64 x 100 = 6,400 runs)
|
|
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_DOE
|
|
\end{lstlisting}
|
|
|
|
%% TODO(bob): confirm canonical path for run_factorial_doe.sh in published repo
|
|
|
|
Each configuration requires a clean \texttt{make} rebuild to prevent state
|
|
contamination. All runs are randomised into a single execution matrix to
|
|
eliminate thermal ramp, temporal, and ordering biases.
|
|
|
|
\subsection{Output Structure}
|
|
|
|
The experiment produces a CSV file with one row per run and a set of run logs:
|
|
|
|
\begin{lstlisting}[language=bash]
|
|
<experiment_label>/
|
|
experiment_results.csv # All measurements (1,920+ rows)
|
|
experiment_summary.txt # Timing, metadata, statistics
|
|
configuration_manifest.txt # All 64 configs documented
|
|
test_matrix.txt # Randomised execution order
|
|
run_logs/ # Per-run logs
|
|
\end{lstlisting}
|
|
|
|
Each CSV row contains a timestamp, configuration identifier, run number,
|
|
35+ performance metrics from the test harness, and the six loop-enable flags.
|
|
This schema allows every metric to be correlated with every loop combination.
|
|
|
|
\subsection{Analysis}
|
|
|
|
\paragraph{Main effects.}
|
|
The average change in a response metric when loop $k$ transitions from off to on,
|
|
marginalised over all other loops:
|
|
|
|
\begin{equation}
|
|
\hat{\beta}_k = \bar{y}_{(L_k=1)} - \bar{y}_{(L_k=0)}
|
|
\end{equation}
|
|
|
|
\paragraph{Two-way interactions.}
|
|
The interaction effect for loops $a$ and $b$:
|
|
|
|
\begin{equation}
|
|
\hat{\beta}_{ab} = \bar{y}_{(1,1)} - \bar{y}_{(1,0)} - \bar{y}_{(0,1)} + \bar{y}_{(0,0)}
|
|
\end{equation}
|
|
|
|
A positive $\hat{\beta}_{ab}$ indicates synergy; a negative value indicates
|
|
suppression.
|
|
|
|
\begin{lstlisting}[language=Python]
|
|
import pandas as pd
|
|
df = pd.read_csv("experiment_results.csv")
|
|
|
|
factor_cols = ['enable_loop_1_heat_tracking',
|
|
'enable_loop_2_rolling_window',
|
|
'enable_loop_3_linear_decay',
|
|
'enable_loop_4_pipelining_metrics',
|
|
'enable_loop_5_window_inference',
|
|
'enable_loop_6_decay_inference']
|
|
metric = 'vm_workload_duration_ns_q48'
|
|
|
|
main_effects = {
|
|
col: df[df[col]==1][metric].mean() - df[df[col]==0][metric].mean()
|
|
for col in factor_cols
|
|
}
|
|
\end{lstlisting}
|
|
|
|
%% TODO(bob): verify column names against actual CSV header from a completed run
|
|
|
|
\subsection{Key Design Principles}
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Rebuild every configuration.} Ensures isolated, uncontaminated
|
|
measurements for each loop combination.
|
|
\item \textbf{Randomise all runs.} The 1{,}920 runs are interleaved in a
|
|
single randomised matrix, not grouped by configuration.
|
|
\item \textbf{Collect once, analyse separately.} Data collection and statistical
|
|
analysis are strictly separated to prevent confirmation bias.
|
|
\item \textbf{Full factorial, not fractional.} All $2^6 = 64$ combinations are
|
|
tested; fractional designs would alias higher-order interactions.
|
|
\end{itemize}
|