268 lines
9.7 KiB
Markdown
268 lines
9.7 KiB
Markdown
<!-- Moved from docs/04-quality/validation/phase-2-executive-summary.md to docs/working/archive/quality/validation/phase-2-executive-summary.md on 2026-06-16 (docs reorg Phase 2) -->
|
||
# Phase 2 Experiment: Executive Summary
|
||
## From Factorial DOE to Multivariate Dynamics Analysis
|
||
|
||
**Date**: 2025-11-20
|
||
**Status**: Design Complete, Ready for Implementation
|
||
**Estimated Runtime**: 6-8 hours (750 runs, 150-200 per config)
|
||
|
||
---
|
||
|
||
## 🎯 The Paradigm Shift
|
||
|
||
**What We Thought We Were Doing:**
|
||
- Measuring isolated factor effects (factorial DOE)
|
||
- Testing "jitter in fixed heartbeat interval"
|
||
- Generating 250 runs and calling it statistically significant
|
||
|
||
**What We're Actually Doing:**
|
||
- Measuring a **coupled multivariate dynamical system**
|
||
- Capturing the **trajectory** of 7 metrics through state-space
|
||
- Understanding how **each metric feeds back** to influence others
|
||
- Discovering which configuration produces the most **stable, convergent, responsive** system
|
||
|
||
---
|
||
|
||
## 🚀 The Physics Model
|
||
|
||
Each configuration is a **unique universe** with different physical laws:
|
||
|
||
```
|
||
Per Tick (1 millisecond):
|
||
cache_hit% ──────┐
|
||
├──> bucket_load ──> execution_heat ──> decay ──> window_width
|
||
bucket_hit% ─────┤ │
|
||
└──────────────────────────────────┬──────────────────┘
|
||
│
|
||
├──> prediction_accuracy
|
||
│
|
||
└──> heartbeat_modulation
|
||
(tick_ns adapts to load)
|
||
```
|
||
|
||
**Key Insight**: The heartbeat rate (`tick_ns`) is **not fixed**. It's modulated by the inference engine in response to workload. This is THE feedback mechanism that couples everything.
|
||
|
||
---
|
||
|
||
## 📊 What We Measure
|
||
|
||
### Per Tick (Every ~1 ms)
|
||
7 metrics captured in circular buffer:
|
||
- `tick_interval_ns` - How much time since last tick (variable!)
|
||
- `cache_hits_delta` - Cache lookups succeeded
|
||
- `bucket_hits_delta` - Secondary lookups succeeded
|
||
- `hot_word_count` - Words above heat threshold
|
||
- `avg_word_heat` - Mean execution heat
|
||
- `window_width` - Current rolling window size
|
||
- `word_executions_delta` - Total word executions
|
||
|
||
### Per Run (100,000+ ticks)
|
||
Stability fingerprint computed from time-series:
|
||
- **Convergence time**: When metrics stabilize
|
||
- **Steady-state CV**: Variability in stable region
|
||
- **Load↔Heartbeat correlation**: How tight is coupling?
|
||
- **Dominant eigenmode**: What oscillation pattern emerges?
|
||
- **Overall stability score**: Composite of all above (0-100)
|
||
|
||
### Per Configuration (150+ runs)
|
||
Configuration fingerprint aggregated from all runs:
|
||
- **Stability reproducibility**: std(stability_score) across runs
|
||
- **Convergence reliability**: % of runs that stabilize
|
||
- **Coupling strength**: Mean and variance of load↔heartbeat_corr
|
||
- **Attractor basin depth**: Resistance to noise
|
||
- **Jitter characteristics**: CV, whiteness, envelope smoothness
|
||
|
||
---
|
||
|
||
## ✨ The Golden Config Decision
|
||
|
||
Compare 5 elite configurations across 5 dimensions:
|
||
|
||
| Dimension | Weight | Interpretation |
|
||
|-----------|--------|---|
|
||
| **Stability** | 30% | Does system converge? Is it reproducible? |
|
||
| **Convergence** | 25% | How fast? How reliable? |
|
||
| **Coupling** | 20% | Does heartbeat respond to load? |
|
||
| **Resilience** | 15% | Can it handle noise? Deep basin? |
|
||
| **Jitter** | 10% | Clean or noisy timing? |
|
||
|
||
**Golden Config** = highest weighted composite score
|
||
|
||
This config becomes the **baseline for MamaForth/PapaForth** and future experiments.
|
||
|
||
---
|
||
|
||
## 📈 Experimental Design
|
||
|
||
```
|
||
5 configurations (1_0_1_1_1_0, 1_0_1_1_1_1, 1_1_0_1_1_1, 1_0_1_0_1_0, 0_1_1_0_1_1)
|
||
× 150-200 runs (statistically significant for correlation analysis)
|
||
× 100K+ ticks (to measure convergence and steady state)
|
||
────────────────────
|
||
= 750 total runs
|
||
75+ million heartbeat ticks
|
||
6-8 hours wall-clock runtime
|
||
~10 GB CSV data (with sampling strategy)
|
||
```
|
||
|
||
**Randomization**: Run order shuffled to avoid temporal confounds.
|
||
|
||
---
|
||
|
||
## 🔧 Implementation Pipeline
|
||
|
||
### Phase 1: Heartbeat Instrumentation ✏️
|
||
- Add `HeartbeatTickSnapshot` struct to capture per-tick metrics
|
||
- Implement circular buffer (100K slots, ~400 KB memory)
|
||
- Inject capture call into heartbeat loop (115 ns overhead, 0.01%)
|
||
- Export to CSV after run completes
|
||
|
||
### Phase 2: Run Fingerprinting 📊
|
||
- Detect convergence from time-series
|
||
- Compute steady-state CV for each metric
|
||
- Measure cross-metric correlations
|
||
- Perform spectral analysis (FFT, eigenmode extraction)
|
||
- Generate RunStabilityFingerprint per run
|
||
|
||
### Phase 3: Config Fingerprinting 📋
|
||
- Aggregate 150+ run fingerprints per config
|
||
- Compute robustness (reproducibility)
|
||
- Assess convergence reliability
|
||
- Evaluate coupling stability
|
||
- Generate ConfigurationFingerprint per config
|
||
|
||
### Phase 4: Golden Config Selection 🏆
|
||
- Score each config on 5 weighted dimensions
|
||
- Select highest-scoring configuration
|
||
- Generate justification report
|
||
|
||
### Phase 5: Visualization & Analysis 📈
|
||
- Time-series plots (cache%, bucket%, tick_ns over time)
|
||
- Correlation heatmaps per config
|
||
- PCA projection of runs
|
||
- Stability score distributions
|
||
- Golden config report with physics interpretation
|
||
|
||
---
|
||
|
||
## 📋 Corrected DoE Script
|
||
|
||
Updated `/scripts/run_factorial_doe_with_heartbeat.sh`:
|
||
- **Runs per config**: 150 (was 50, which was statistically insufficient)
|
||
- **Total runs**: 750 (was 250)
|
||
- **Output**: Same CSV structure + new per-tick timeseries CSV
|
||
- **Command**: `./scripts/run_factorial_doe_with_heartbeat.sh 2025_11_20_PHASE2_DYNAMICS`
|
||
|
||
---
|
||
|
||
## 🔍 Key Metrics Explained
|
||
|
||
| Metric | Meaning | Good Value |
|
||
|--------|---------|------------|
|
||
| **convergence_time_ticks** | How many ticks to stabilize? | <5000 (fast learner) |
|
||
| **cache_convergence_cv** | Variability of cache% in steady state | <0.05 (5% variation) |
|
||
| **load_heartbeat_corr** | Does tick_ns respond to workload? | >0.80 (tight coupling) |
|
||
| **dominant_eigenvalue** | Attractor strength | Large (deep basin) |
|
||
| **tick_jitter_cv** | Variability of tick intervals | <0.20 (20% max) |
|
||
| **overall_stability_score** | Composite 0-100 | >80 (excellent) |
|
||
|
||
---
|
||
|
||
## 🎓 Interpretation Guide
|
||
|
||
### A "Stable" Configuration Fingerprint (Good)
|
||
✅ stability_score_mean = 85, std = 2.1
|
||
✅ 100% convergence_success_rate
|
||
✅ load_heartbeat_corr_mean = 0.88
|
||
✅ tick_jitter_cv = 0.15
|
||
✅ Dominant eigenvalue large (deep attractor)
|
||
|
||
→ **Interpretation**: This universe is orderly. Metrics converge predictably. The heartbeat tightly couples to load. Timing is clean. Noise doesn't destabilize it.
|
||
|
||
### A "Chaotic" Configuration Fingerprint (Poor)
|
||
❌ stability_score_mean = 62, std = 8.3
|
||
❌ 73% convergence_success_rate (some runs diverge!)
|
||
❌ load_heartbeat_corr_mean = 0.44 (weak coupling, inconsistent)
|
||
❌ tick_jitter_cv = 0.52 (52% variation, very noisy)
|
||
❌ Dominant eigenvalue small (shallow basin)
|
||
|
||
→ **Interpretation**: This universe is noisy. Some runs fail to converge. Heartbeat doesn't respond cleanly to load. Timing jitters wildly. Any small perturbation can destabilize it.
|
||
|
||
---
|
||
|
||
## 🔬 Why This Matters
|
||
|
||
**Phase 1** (Stage 1, your baseline) ran WITHOUT adaptive heartbeat.
|
||
- All runs used fixed 1ms tick interval
|
||
- No feedback coupling
|
||
- Missing the actual physics
|
||
|
||
**Phase 2** (this experiment) runs WITH adaptive heartbeat.
|
||
- Heartbeat rate (`tick_ns`) modulates based on inference engine output
|
||
- Full feedback coupling measured
|
||
- Captures the REAL adaptive behavior
|
||
|
||
The golden config from Phase 2 becomes:
|
||
- ⭐ **MamaForth baseline** for human-in-the-loop iteration
|
||
- ⭐ **PapaForth reference** for automated optimization
|
||
- ⭐ **Stability anchor** for Phase 3 (deeper optimization)
|
||
|
||
---
|
||
|
||
## 📚 Design Documents
|
||
|
||
1. **MULTIVARIATE_DYNAMICS_DESIGN.md** (622 lines)
|
||
- Full coupled systems theory
|
||
- Struct definitions for fingerprints
|
||
- CSV schemas
|
||
- Analysis pipeline details
|
||
|
||
2. **HEARTBEAT_INSTRUMENTATION_PLAN.md** (350+ lines)
|
||
- Implementation checklist
|
||
- Per-tick capture infrastructure
|
||
- Circular buffer design
|
||
- CSV export strategy
|
||
|
||
3. **run_factorial_doe_with_heartbeat.sh** (updated)
|
||
- 150 runs/config default (was 50)
|
||
- 750 total runs (statistically significant)
|
||
|
||
---
|
||
|
||
## 🚀 Next Steps
|
||
|
||
### Immediate (This Session)
|
||
- [ ] Implement HeartbeatTickSnapshot instrumentation in vm.c
|
||
- [ ] Add delta metric tracking to VM struct
|
||
- [ ] Test with abbreviated run (verify CSV output)
|
||
|
||
### Follow-On (Next Session)
|
||
- [ ] Implement RunStabilityFingerprint computation
|
||
- [ ] Build ConfigurationFingerprint aggregation
|
||
- [ ] Create R analysis suite
|
||
- [ ] Execute full Phase 2 experiment (750 runs, 6-8 hours)
|
||
|
||
### Outcome
|
||
- [ ] Golden configuration identified
|
||
- [ ] Justification report written
|
||
- [ ] Metrics exported for visualization
|
||
- [ ] Ready for Phase 3 deeper optimization
|
||
|
||
---
|
||
|
||
## 💡 The Big Picture
|
||
|
||
You've identified the **actual physics** of the system:
|
||
|
||
1. **Every metric is coupled** to every other via immediate feedback
|
||
2. **The heartbeat rate itself is a tuning knob** (modulated by inference)
|
||
3. **Stability emerges from convergence + coupling + basin depth**
|
||
4. **You measure not "factor effects" but "trajectory shapes"**
|
||
|
||
This shifts from **reductionist factorial analysis** to **holistic systems characterization**.
|
||
|
||
The golden config isn't "better because factor A is on/off." It's better because **its entire attractor landscape** is more stable, more convergent, more resilient, and more tightly coupled to load.
|
||
|
||
---
|
||
|
||
**Captain, the framework is complete. Ready to instrument the heartbeat.** |