43 lines
2.0 KiB
TeX
43 lines
2.0 KiB
TeX
%% SCRAP: archive/quality/regression/detection-framework
|
|
%% SOURCE: docs/working/archive/quality/regression/detection-framework.md
|
|
%% STATUS: HISTORICAL
|
|
%% FITS: none
|
|
%% EDITORIAL: lifted — prose rewritten to press voice
|
|
|
|
\section*{Optimization Regression Detection Framework (November 2025)}
|
|
|
|
The StarForth optimization program defined a multi-metric regression detection
|
|
framework to distinguish genuine optimization---improvement across all relevant
|
|
metrics---from simple speedup that trades one metric for another.
|
|
|
|
\subsection*{Regression Taxonomy}
|
|
|
|
Six regression types were identified:
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Type~1: Metric Inversion.} Primary metric improves while a
|
|
secondary metric (memory, variance, stability) regresses. Detection
|
|
threshold: $>10\%$ secondary degradation.
|
|
\item \textbf{Type~2: Variance Introduction.} Mean improves but standard
|
|
deviation increases. Detection threshold: $>25\%$ variance increase.
|
|
\item \textbf{Type~3: Tail Latency.} Median improves but P99 worsens.
|
|
Detection threshold: $>15\%$ P99 degradation.
|
|
\item \textbf{Type~4: Cascading Failure.} Small-workload gains reverse at
|
|
production scale (e.g., dictionary growth, memory fragmentation over
|
|
1M+ instructions).
|
|
\item \textbf{Type~5: Workload Sensitivity.} Optimization overfits the
|
|
936-test training workload and degrades on other patterns.
|
|
\item \textbf{Type~6: Parameter Interaction.} Two independently-winning
|
|
parameters produce a regression when combined.
|
|
\end{itemize}
|
|
|
|
\subsection*{Decision Algorithm}
|
|
|
|
Each optimization candidate was evaluated as \textsc{accept} (no critical
|
|
regressions), \textsc{investigate} ($>2$ warnings), or \textsc{reject} (any
|
|
critical regression). The five-question validation test---is the optimization
|
|
optimal, stable, general, non-interacting, and maintainable?---served as the
|
|
human-readable gate before locking any parameter for subsequent experiments.
|
|
|
|
This framework governed the OPP~1 through OPP~5 experimental series.
|