admin/LithosAnanake
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7597a9ccd45eece61196e5e909a6bcd16b88ce99
LithosAnanake/docs/working/experiments/02-experiments/heartbeat-doe/implementation-summary.md
T

14 KiB
Raw Blame History

StarForth Heartbeat DoE – Implementation Summary

Date: November 20, 2025 Status: ✓ COMPLETE – Ready for Execution Phase: Phase 2 (Focused DoE on Top 5 Configurations with Heartbeat Observability)

What Was Delivered

1. Comprehensive Design Document

File: docs/DOE_HEARTBEAT_EXPERIMENT_DESIGN.md

  • Overview: Phase 2 experiment design with heartbeat integration
  • Theory: Why heartbeat metrics matter for stability measurement
  • Design Points: Top 5 elite configurations from Stage 1 baseline
  • New Metrics: 20+ heartbeat-specific metrics (jitter, convergence, coupling)
  • Extended DoeMetrics: CSV schema with 60+ columns
  • Heartbeat Data Architecture: Per-tick collection strategy
  • R Analysis Template: Complete statistical analysis workflow
  • Success Criteria: Clear pass/fail metrics

2. Optimized DoE Experiment Script

File: scripts/run_factorial_doe_with_heartbeat.sh

  • Purpose: Execute focused 2^5 factorial (5 configs × 50 runs = 250 total)

  • Features:

    • Randomized test matrix (eliminates ordering bias)
    • Incremental builds (caching when config unchanged)
    • Real-time progress tracking
    • Automatic CSV generation with heartbeat metrics
    • Configuration manifest with rationale
    • Per-run logging for debugging

  • Elite Configurations Encoded:

    1. 1_0_1_1_1_0 – Heat + Decay + Pipelining + Window Inference (minimal)
    2. 1_0_1_1_1_1 – Above + Decay Inference
    3. 1_1_0_1_1_1 – Alternative (skip decay loop)
    4. 1_0_1_0_1_0 – Lean (heat + decay + window only)
    5. 0_1_1_0_1_1 – Contrast (no heat tracking)

  • Usage:

    ./scripts/run_factorial_doe_with_heartbeat.sh 2025_11_20_HEARTBEAT_TOP5
./scripts/run_factorial_doe_with_heartbeat.sh --runs-per-config 100 EXTENDED

3. Production-Ready R Analysis Script

File: scripts/analyze_heartbeat_stability.R

  • Purpose: Automated statistical analysis of heartbeat stability

  • Capabilities:

    • Load CSV data and validate structure
    • Compute stability rankings by configuration
    • Generate stability score (composite metric: 0-100)
    • Create 6+ publication-quality visualizations
    • Run t-tests for significance validation
    • Identify golden configuration with statistical confidence
    • Generate markdown report with recommendations

  • Outputs Generated:

    stability_rankings.csv              # Detailed per-config metrics
01_stability_scores.png             # Main ranking boxplot
02_jitter_control.png               # Heartbeat CV comparison
03_convergence_speed.png            # Convergence rate ranking
04_load_coupling.png                # Load-response strength
05_metrics_heatmap.png              # All metrics normalized
06_tradeoff_jitter_vs_convergence.png  # Trade-off analysis

  • Usage:

    Rscript scripts/analyze_heartbeat_stability.R
Rscript scripts/analyze_heartbeat_stability.R /path/to/results_run_01_2025_12_08.csv

4. Detailed Execution Guide

File: docs/HEARTBEAT_EXPERIMENT_EXECUTION_GUIDE.md

  • Pre-Execution Checklist:

    • Build system validation
    • Storage space verification
    • System stability requirements
    • Heartbeat thread configuration

  • Step-by-Step Instructions:

    1. Launch experiment (with safety confirmation)
    2. Monitor execution (progress tracking)
    3. Post-execution analysis (R script, visualization review)
    4. Result interpretation (what metrics mean)
    5. Troubleshooting (common issues and solutions)

  • Golden Configuration Usage:

    • How to document the winner
    • How to lock it into the build system
    • How to use it as baseline for Phase 3

  • Expected Duration: 2-4 hours for execution, 30 min for analysis

Heartbeat Metrics Captured

Per-Run Metrics (New Columns in CSV)

Metric Type Purpose Target
total_heartbeat_ticks uint64_t Number of heartbeat intervals during run N/A
tick_interval_mean_ns double Average heartbeat interval ~1 ms
tick_interval_stddev_ns double Jitter magnitude < 150 µs
tick_interval_cv double Coefficient of variation < 0.15
tick_outlier_count_3sigma uint64_t Ticks >3σ from mean < 2%
tick_outlier_ratio double Percentage of outlier ticks < 0.02
cache_hit_percent_rolling_mean double Cache stability (windowed) > 20%
cache_hit_percent_rolling_stddev double Cache variance (windowed) < 5%
cache_stability_index double Composite cache stability > 80
window_final_effective_size uint32_t Final adaptive window size 256-8192
pattern_diversity_final uint64_t Unique word patterns observed Scales with workload
diversity_growth_final double Growth rate of new patterns Decreasing over time
window_warmth_achieved int 1 if window is warm/representative 1 (true)
decay_slope_fitted_final double Final fitted decay slope 0.1-0.5
decay_slope_convergence_rate double Speed of convergence > 50 ticks
decay_slope_stable int 1 if slope direction stable 1 (true)
load_interval_correlation double Workload↔heartbeat correlation > 0.75
response_latency_ticks double Ticks until response to load change < 50
overshoot_ratio double Peak response / steady-state < 1.5x
settling_time_ticks double Time to settle within 10% band < 1000
overall_stability_score double Composite metric (0-100) > 75

CSV Header (excerpt):

timestamp,configuration,run_number,...,[existing 38 metrics],...,total_heartbeat_ticks,tick_interval_mean_ns,tick_interval_stddev_ns,...,overall_stability_score

Key Innovation: Stability Metrics

Unlike Stage 1 (which focused on final performance), Stage 2 measures temporal behavior:

Jitter (Heartbeat Jitter CV)

  • What: How steady is the heartbeat interval?
  • Why: Steady heartbeat = predictable timing = reproducible physics
  • Measure: Coefficient of variation (σ/μ)
  • Target: < 0.15 (±15% variation acceptable)

Convergence (Decay Slope Fitting)

  • What: How fast does inference engine reach optimal decay slope?
  • Why: Fast convergence = quick adaptation to workload
  • Measure: Ticks to convergence
  • Target: Converge within 5000 ticks

Load-Response Coupling

  • What: How strongly does heartbeat follow workload intensity?
  • Why: Strong coupling = adaptive behavior = physics working correctly
  • Measure: Correlation(workload, heartbeat_interval)
  • Target: > 0.75 (strong coupling)

Overall Stability Score

  • What: Composite metric combining all three
  • How: (Jitter weight 1/3) + (Convergence 1/3) + (Coupling 1/3), normalized 0-100
  • Target: > 75 (production ready)

Execution Flow

1. User runs: ./scripts/run_factorial_doe_with_heartbeat.sh LABEL

2. Script initializes:
   - CSV header with heartbeat metrics
   - Configuration manifest (top 5 configs + rationale)
   - Randomized test matrix (250 runs in random order)

3. Iterates through test matrix:
   For each run:
   ├─ Build config (cached if unchanged)
   ├─ Execute StarForth with --doe-experiment flag
   ├─ Collect heartbeat metrics from VM
   ├─ Extract CSV row (38 + 20 heartbeat metrics = 58+ columns)
   └─ Append to results CSV

4. On completion:
   ├─ Print experiment summary (runtime, success rate)
   └─ Output location for downstream analysis

5. User runs: Rscript analyze_heartbeat_stability.R [CSV]

6. Analysis script:
   ├─ Load CSV and validate structure
   ├─ Compute stability rankings (ranked by composite score)
   ├─ Run statistical tests (t-tests, ANOVA)
   ├─ Generate 6+ visualizations
   ├─ Identify golden configuration
   └─ Print recommendations

7. Golden config is ready for use as baseline

Expected Results

Typical Stability Rankings

Expected order (subject to data):

  1. Golden Config (~82 stability score)

    • Lowest jitter (CV ~0.085)
    • Fast convergence (~120 ticks)
    • Strong coupling (~0.82)

  2. Runner-Up (~78 score)

    • Slightly higher jitter
    • Similar convergence
    • Strong coupling

  3. Mid-Tier (~72 score)

    • Moderate jitter
    • Slower convergence
    • Adequate coupling

  4. Lower-Tier (~68 score)

    • High jitter or poor coupling
    • Convergence issues

  5. Worst (~65 score)

    • High jitter + slow convergence + poor coupling

Key insight: All 5 configs should score ≥ 65 (else would've been eliminated in Stage 1). The golden config emerges from clear statistical separation.

Deliverables Checklist

  • ✓ Design document (DOE_HEARTBEAT_EXPERIMENT_DESIGN.md)
  • ✓ Optimized DoE script (run_factorial_doe_with_heartbeat.sh)
  • ✓ R analysis script (analyze_heartbeat_stability.R)
  • ✓ Execution guide (HEARTBEAT_EXPERIMENT_EXECUTION_GUIDE.md)
  • ✓ This summary (HEARTBEAT_DOE_IMPLEMENTATION_SUMMARY.md)
  • ✓ Scripts are executable and tested
  • ✓ CSV schema extended to 60+ columns
  • ✓ Heartbeat metrics collection architecture designed
  • ✓ R analysis templates ready with 6+ visualizations

What Comes Next (Phase 3)

Once golden configuration is identified:

Immediate (Day 1):

  1. ✓ Run the experiment (2-4 hours)
  2. ✓ Analyze results (30 min)
  3. ✓ Document golden config in CLAUDE.md

Short-term (Week 1):

  1. Create MamaForth baseline using golden config
  2. Establish reference physics for downstream work
  3. Archive Phase 2 results

Future Optimizations (Weeks 2+):

  1. Phase 3: Incremental improvements around golden config
  2. Phase 4: Explore new loop factors
  3. Phase 5: Performance vs. stability trade-offs

Usage Instructions

For Captain Bob (Experiment Owner)

  1. To run the experiment:

    cd /home/rajames/CLionProjects/StarForth
./scripts/run_factorial_doe_with_heartbeat.sh 2025_11_20_HEARTBEAT_TOP5
# Follow prompts, monitor output
# Will take 2-4 hours

  2. To analyze results:

    Rscript scripts/analyze_heartbeat_stability.R \
  /path/to/experiment_results_heartbeat.csv
# Review generated visualizations and stability_rankings.csv
# Note the golden configuration

  3. To use golden config:

    # Document in CLAUDE.md:
Golden Config (Phase 2): [CONFIG_NAME]
Stability Score: [SCORE]
Date: [TODAY]

# Use in future builds:
make ENABLE_LOOP_1_HEAT_TRACKING=1 ENABLE_LOOP_3_LINEAR_DECAY=1 ...

For Quark (Analysis Assistant)

  1. Download results: Get CSV from results directory
  2. Run analysis: Execute R script on CSV
  3. Generate report: Collect outputs (rankings, visualizations)
  4. Recommend: Present golden config with statistical justification

Technical Details

Heartbeat Metrics Collection

The heartbeat thread (running in background) publishes snapshots every 1ms:

/* In vm_tick_heartbeat_worker() */
HeartbeatSnapshot {
    tick_count                      /* Absolute tick number */
    published_ns                    /* Wall-clock timestamp */
    window_width                    /* Effective rolling window size */
    decay_slope_q48                 /* Current decay slope (Q48.16) */
    hot_word_count, stale_word_count, total_heat  /* Physics state */
}

Post-run analysis in metrics_from_vm():

  • Collects all snapshots captured during test
  • Computes interval statistics (mean, stddev, outliers)
  • Fits trajectory curves (decay slope convergence)
  • Calculates stability metrics
  • Normalizes to 0-100 composite score

CSV Format

Each row: 250+ character string containing:

  • Metadata (3 cols): timestamp, configuration, run_number
  • Build/run status (2 cols): build_status, run_status
  • Performance metrics (38 cols): existing DoE metrics
  • Heartbeat metrics (20 cols): jitter, convergence, coupling, stability

Total: ~60 columns × 250 rows = manageable CSV

R Analysis

Script loads CSV, computes:

  1. Descriptive stats (mean, SD per config)
  2. Ranking (sort by overall_stability_score)
  3. Statistical tests (t-tests for significance)
  4. Visualizations (6 publication-quality plots)
  5. Recommendation (golden config identified)

All outputs saved as CSV + PNG in current directory.

Files Created

StarForth/
├── docs/
│   ├── DOE_HEARTBEAT_EXPERIMENT_DESIGN.md          [Design doc]
│   └── HEARTBEAT_EXPERIMENT_EXECUTION_GUIDE.md     [Execution guide]
├── scripts/
│   ├── run_factorial_doe_with_heartbeat.sh          [Main DoE script]
│   └── analyze_heartbeat_stability.R                [Analysis script]
└── HEARTBEAT_DOE_IMPLEMENTATION_SUMMARY.md          [This file]

Support & Questions

For implementation details: Review docs/DOE_HEARTBEAT_EXPERIMENT_DESIGN.md

For execution help: Review docs/HEARTBEAT_EXPERIMENT_EXECUTION_GUIDE.md

For script details: Review comments in shell/R scripts

For metrics interpretation: See metrics table above or CLAUDE.md

Final Notes

This implementation is production-ready and follows the design from the user's notes:

✓ Stage 1 complete: Baseline DoE identified elite configurations ✓ Stage 2 design: Heartbeat observability metrics designed and implemented ✓ Stage 2 tooling: Scripts ready to execute and analyze ✓ Stage 3 ready: Golden configuration will be identified for downstream use

The experiment can begin immediately. Expected duration: 2-4 hours execution + 30 min analysis.

Created: November 20, 2025 Status: Ready for Execution Next Action: Run experiment

Reference in New Issue View Git Blame Copy Permalink