9.7 KiB
Phase 2 Experiment: Executive Summary
From Factorial DOE to Multivariate Dynamics Analysis
Date: 2025-11-20 Status: Design Complete, Ready for Implementation Estimated Runtime: 6-8 hours (750 runs, 150-200 per config)
🎯 The Paradigm Shift
What We Thought We Were Doing:
- Measuring isolated factor effects (factorial DOE)
- Testing "jitter in fixed heartbeat interval"
- Generating 250 runs and calling it statistically significant
What We're Actually Doing:
- Measuring a coupled multivariate dynamical system
- Capturing the trajectory of 7 metrics through state-space
- Understanding how each metric feeds back to influence others
- Discovering which configuration produces the most stable, convergent, responsive system
🚀 The Physics Model
Each configuration is a unique universe with different physical laws:
Per Tick (1 millisecond):
cache_hit% ──────┐
├──> bucket_load ──> execution_heat ──> decay ──> window_width
bucket_hit% ─────┤ │
└──────────────────────────────────┬──────────────────┘
│
├──> prediction_accuracy
│
└──> heartbeat_modulation
(tick_ns adapts to load)
Key Insight: The heartbeat rate (tick_ns) is not fixed. It's modulated by the inference engine in response to workload. This is THE feedback mechanism that couples everything.
📊 What We Measure
Per Tick (Every ~1 ms)
7 metrics captured in circular buffer:
tick_interval_ns- How much time since last tick (variable!)cache_hits_delta- Cache lookups succeededbucket_hits_delta- Secondary lookups succeededhot_word_count- Words above heat thresholdavg_word_heat- Mean execution heatwindow_width- Current rolling window sizeword_executions_delta- Total word executions
Per Run (100,000+ ticks)
Stability fingerprint computed from time-series:
- Convergence time: When metrics stabilize
- Steady-state CV: Variability in stable region
- Load↔Heartbeat correlation: How tight is coupling?
- Dominant eigenmode: What oscillation pattern emerges?
- Overall stability score: Composite of all above (0-100)
Per Configuration (150+ runs)
Configuration fingerprint aggregated from all runs:
- Stability reproducibility: std(stability_score) across runs
- Convergence reliability: % of runs that stabilize
- Coupling strength: Mean and variance of load↔heartbeat_corr
- Attractor basin depth: Resistance to noise
- Jitter characteristics: CV, whiteness, envelope smoothness
✨ The Golden Config Decision
Compare 5 elite configurations across 5 dimensions:
| Dimension | Weight | Interpretation |
|---|---|---|
| Stability | 30% | Does system converge? Is it reproducible? |
| Convergence | 25% | How fast? How reliable? |
| Coupling | 20% | Does heartbeat respond to load? |
| Resilience | 15% | Can it handle noise? Deep basin? |
| Jitter | 10% | Clean or noisy timing? |
Golden Config = highest weighted composite score
This config becomes the baseline for MamaForth/PapaForth and future experiments.
📈 Experimental Design
5 configurations (1_0_1_1_1_0, 1_0_1_1_1_1, 1_1_0_1_1_1, 1_0_1_0_1_0, 0_1_1_0_1_1)
× 150-200 runs (statistically significant for correlation analysis)
× 100K+ ticks (to measure convergence and steady state)
────────────────────
= 750 total runs
75+ million heartbeat ticks
6-8 hours wall-clock runtime
~10 GB CSV data (with sampling strategy)
Randomization: Run order shuffled to avoid temporal confounds.
🔧 Implementation Pipeline
Phase 1: Heartbeat Instrumentation ✏️
- Add
HeartbeatTickSnapshotstruct to capture per-tick metrics - Implement circular buffer (100K slots, ~400 KB memory)
- Inject capture call into heartbeat loop (115 ns overhead, 0.01%)
- Export to CSV after run completes
Phase 2: Run Fingerprinting 📊
- Detect convergence from time-series
- Compute steady-state CV for each metric
- Measure cross-metric correlations
- Perform spectral analysis (FFT, eigenmode extraction)
- Generate RunStabilityFingerprint per run
Phase 3: Config Fingerprinting 📋
- Aggregate 150+ run fingerprints per config
- Compute robustness (reproducibility)
- Assess convergence reliability
- Evaluate coupling stability
- Generate ConfigurationFingerprint per config
Phase 4: Golden Config Selection 🏆
- Score each config on 5 weighted dimensions
- Select highest-scoring configuration
- Generate justification report
Phase 5: Visualization & Analysis 📈
- Time-series plots (cache%, bucket%, tick_ns over time)
- Correlation heatmaps per config
- PCA projection of runs
- Stability score distributions
- Golden config report with physics interpretation
📋 Corrected DoE Script
Updated /scripts/run_factorial_doe_with_heartbeat.sh:
- Runs per config: 150 (was 50, which was statistically insufficient)
- Total runs: 750 (was 250)
- Output: Same CSV structure + new per-tick timeseries CSV
- Command:
./scripts/run_factorial_doe_with_heartbeat.sh 2025_11_20_PHASE2_DYNAMICS
🔍 Key Metrics Explained
| Metric | Meaning | Good Value |
|---|---|---|
| convergence_time_ticks | How many ticks to stabilize? | <5000 (fast learner) |
| cache_convergence_cv | Variability of cache% in steady state | <0.05 (5% variation) |
| load_heartbeat_corr | Does tick_ns respond to workload? | >0.80 (tight coupling) |
| dominant_eigenvalue | Attractor strength | Large (deep basin) |
| tick_jitter_cv | Variability of tick intervals | <0.20 (20% max) |
| overall_stability_score | Composite 0-100 | >80 (excellent) |
🎓 Interpretation Guide
A "Stable" Configuration Fingerprint (Good)
✅ stability_score_mean = 85, std = 2.1 ✅ 100% convergence_success_rate ✅ load_heartbeat_corr_mean = 0.88 ✅ tick_jitter_cv = 0.15 ✅ Dominant eigenvalue large (deep attractor)
→ Interpretation: This universe is orderly. Metrics converge predictably. The heartbeat tightly couples to load. Timing is clean. Noise doesn't destabilize it.
A "Chaotic" Configuration Fingerprint (Poor)
❌ stability_score_mean = 62, std = 8.3 ❌ 73% convergence_success_rate (some runs diverge!) ❌ load_heartbeat_corr_mean = 0.44 (weak coupling, inconsistent) ❌ tick_jitter_cv = 0.52 (52% variation, very noisy) ❌ Dominant eigenvalue small (shallow basin)
→ Interpretation: This universe is noisy. Some runs fail to converge. Heartbeat doesn't respond cleanly to load. Timing jitters wildly. Any small perturbation can destabilize it.
🔬 Why This Matters
Phase 1 (Stage 1, your baseline) ran WITHOUT adaptive heartbeat.
- All runs used fixed 1ms tick interval
- No feedback coupling
- Missing the actual physics
Phase 2 (this experiment) runs WITH adaptive heartbeat.
- Heartbeat rate (
tick_ns) modulates based on inference engine output - Full feedback coupling measured
- Captures the REAL adaptive behavior
The golden config from Phase 2 becomes:
- ⭐ MamaForth baseline for human-in-the-loop iteration
- ⭐ PapaForth reference for automated optimization
- ⭐ Stability anchor for Phase 3 (deeper optimization)
📚 Design Documents
-
MULTIVARIATE_DYNAMICS_DESIGN.md (622 lines)
- Full coupled systems theory
- Struct definitions for fingerprints
- CSV schemas
- Analysis pipeline details
-
HEARTBEAT_INSTRUMENTATION_PLAN.md (350+ lines)
- Implementation checklist
- Per-tick capture infrastructure
- Circular buffer design
- CSV export strategy
-
run_factorial_doe_with_heartbeat.sh (updated)
- 150 runs/config default (was 50)
- 750 total runs (statistically significant)
🚀 Next Steps
Immediate (This Session)
- Implement HeartbeatTickSnapshot instrumentation in vm.c
- Add delta metric tracking to VM struct
- Test with abbreviated run (verify CSV output)
Follow-On (Next Session)
- Implement RunStabilityFingerprint computation
- Build ConfigurationFingerprint aggregation
- Create R analysis suite
- Execute full Phase 2 experiment (750 runs, 6-8 hours)
Outcome
- Golden configuration identified
- Justification report written
- Metrics exported for visualization
- Ready for Phase 3 deeper optimization
💡 The Big Picture
You've identified the actual physics of the system:
- Every metric is coupled to every other via immediate feedback
- The heartbeat rate itself is a tuning knob (modulated by inference)
- Stability emerges from convergence + coupling + basin depth
- You measure not "factor effects" but "trajectory shapes"
This shifts from reductionist factorial analysis to holistic systems characterization.
The golden config isn't "better because factor A is on/off." It's better because its entire attractor landscape is more stable, more convergent, more resilient, and more tightly coupled to load.
Captain, the framework is complete. Ready to instrument the heartbeat.