Files

T

Robert Allan James a5ed8c3d87 Initial commit — LithosAnanke kernel

2026-08-01 07:49:56 -04:00

16 KiB

Raw Blame History

StarForth Heartbeat DoE Experiment – Execution Guide

Date: November 20, 2025 Purpose: Execute Phase 2 factorial DoE with heartbeat stability metrics Duration: ~2-4 hours (250 runs = 5 configs × 50 runs each)

Quick Start

# Navigate to repo root
cd /home/rajames/CLionProjects/StarForth

# Run the experiment (interactive prompt before execution)
./scripts/run_factorial_doe_with_heartbeat.sh 2025_11_20_HEARTBEAT_TOP5

# Optional: Use custom run count
./scripts/run_factorial_doe_with_heartbeat.sh --runs-per-config 100 HEARTBEAT_EXTENDED

# Analyze results_run_01_2025_12_08 (when complete)
Rscript scripts/analyze_heartbeat_stability.R

# Or with explicit path:
Rscript scripts/analyze_heartbeat_stability.R ~/StarForth-DoE/experiments/2025_11_20_HEARTBEAT_TOP5/experiment_results_heartbeat.csv

What This Experiment Does

Phase 2: Focus on Heartbeat Stability

Building on Stage 1 (3200-run baseline without heartbeat), this experiment:

Focuses on 5 elite configurations (not all 64)
Re-enables heartbeat thread (HEARTBEAT_THREAD_ENABLED=1)
Captures per-tick dynamics:
- Heartbeat interval timing (jitter)
- Cache hit stability (rolling window)
- Decay slope convergence
- Load-response coupling
Measures stability characteristics:
- Coefficient of variation in tick interval
- Outlier ratio (3σ threshold)
- Convergence rate to steady state
- Correlation between workload and heartbeat frequency

The 5 Elite Configurations

#	Config	Description	Key Strengths
1	`1_0_1_1_1_0`	Heat + Decay + Pipelining + Window Inference	Minimal overhead, strong performance
2	`1_0_1_1_1_1`	Config #1 + Decay Inference	Full inference pipeline
3	`1_1_0_1_1_1`	Heat + Window + Pipelining + Both Inferences	Alternative path (skip decay loop)
4	`1_0_1_0_1_0`	Heat + Decay + Window Inference only	Lean configuration
5	`0_1_1_0_1_1`	Window + Decay + Both Inferences	Contrast: no heat tracking

Factor codes:

L1 = ENABLE_LOOP_1_HEAT_TRACKING
L2 = ENABLE_LOOP_2_ROLLING_WINDOW
L3 = ENABLE_LOOP_3_LINEAR_DECAY
L4 = ENABLE_LOOP_4_PIPELINING_METRICS
L5 = ENABLE_LOOP_5_WINDOW_INFERENCE
L6 = ENABLE_LOOP_6_DECAY_INFERENCE

Pre-Execution Checklist

1. Build System Ready

# Verify build system works
cd /home/rajames/CLionProjects/StarForth
make clean
make fastest

# Should complete without errors
./build/amd64/fastest/starforth -c "1 2 + . BYE"

If build fails, troubleshoot:

Check compiler: gcc --version
Verify dependencies: make check-deps
Clean and retry: make clean && make fastest

2. Storage Space

The experiment will generate:

CSV file: ~50 MB (250 runs × 60+ columns)
Run logs: ~500 MB (250 separate run logs)
Total: ~600 MB required

Verify space:

mkdir -p /home/rajames/CLionProjects/StarForth-DoE/experiments
df -h /home/rajames/CLionProjects/StarForth-DoE/
# Should show at least 1 GB available

3. System Stability

The experiment will run continuously for 2-4 hours. Minimize background load:

# Check current load
uptime

# Close unnecessary applications
# - Disable automatic updates
# - Close heavy browser tabs
# - Consider running on a dedicated machine

# Optional: Run with reduced CPU frequency for more stable heartbeat
# (Requires sudo; skip if not needed)
# sudo cpupower frequency-set --governor powersave

4. Heartbeat Thread Configuration

Verify heartbeat is compiled in:

make clean
make fastest HEARTBEAT_THREAD_ENABLED=1

# Check binary
nm ./build/amd64/fastest/starforth | grep -i heartbeat
# Should show heartbeat symbols

Execution Steps

Step 1: Launch the Experiment

cd /home/rajames/CLionProjects/StarForth

./scripts/run_factorial_doe_with_heartbeat.sh 2025_11_20_HEARTBEAT_TOP5

Output:

═══════════════════════════════════════════════════════════
STARFORTH HEARTBEAT DoE - TOP 5 ELITE CONFIGURATIONS
═══════════════════════════════════════════════════════════
→ Factors: 6 feedback loops (LOOP_1 through LOOP_6)
→ Configurations: 5 elite (selected from Stage 1 baseline)
→ Runs per config: 50
→ TOTAL RUNS: 250
→ Heartbeat metrics: YES (per-tick jitter, convergence, coupling)
→ Workload: Complete test harness (936+ FORTH tests)
→ Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_20_HEARTBEAT_TOP5

>>> Generating randomized test matrix...
✓ Test matrix generated: .../test_matrix.txt

Test Matrix Preview (first 10 of 250 runs):
  1_0_1_0_1_0,47
  0_1_1_0_1_1,13
  1_1_0_1_1_1,35
  ...

→ Complete test matrix saved: .../test_matrix.txt
Press ENTER to begin execution (Ctrl+C to abort):

At this point:

Review the configuration manifest
Verify test matrix looks reasonable (random order)
Press ENTER to start

Step 2: Monitor Execution

The script will print progress:

>>> RANDOMIZED HEARTBEAT EXPERIMENT (250 runs)
→ Run 1/250 - config 1_0_1_0_1_0 #47...
  [Building configuration...]
✓ Build successful for 1_0_1_0_1_0
✓ Run 1/250 completed (12s)

→ Run 2/250 - config 0_1_1_0_1_1 #13...
✓ Run 2/250 completed (11s)
...

Expected timing:

~10-15 seconds per run (test harness + heartbeat collection)
~250-375 minutes total (4-6 hours)
Builds are cached, only rebuild when config changes

What to monitor:

Are builds succeeding? (OK status)
Are runs completing without crashes? (Log lines appearing)
Elapsed time tracking (should be ~12s per run on average)

Step 3: Completion

When finished:

═══════════════════════════════════════════════════════════
EXPERIMENT COMPLETE
═══════════════════════════════════════════════════════════
✓ All runs completed!
✓ Start time: 2025-11-20T14:30:45
✓ End time: 2025-11-20T18:15:20
✓ Total runtime: 3h 44m 35s
✓ Results: .../experiment_results_heartbeat.csv
✓ Configs: .../configuration_manifest.txt
✓ Logs: .../run_logs/

CSV Results Preview (first 5 data rows):
  [CSV data shown]

→ Next steps:
  1. Download results to analysis repository
  2. Run R analysis: Rscript analyze_heartbeat_stability.R
  3. Identify golden configuration by stability score
  4. Use golden config as baseline for MamaForth/PapaForth

Post-Execution Analysis

Step 1: Verify CSV Output

cd /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_20_HEARTBEAT_TOP5

# Check file
ls -lh experiment_results_heartbeat.csv

# Verify header
head -1 experiment_results_heartbeat.csv

# Count rows (should be 251 = header + 250 data rows)
wc -l experiment_results_heartbeat.csv

Expected header includes:

timestamp,configuration,run_number,...,total_heartbeat_ticks,tick_interval_mean_ns,tick_interval_stddev_ns,tick_interval_cv,...,overall_stability_score

Step 2: Run R Analysis

cd /home/rajames/CLionProjects/StarForth

# Run analysis with explicit path
Rscript scripts/analyze_heartbeat_stability.R \
  /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_20_HEARTBEAT_TOP5/experiment_results_heartbeat.csv

# Or copy CSV to local directory and run with default
cp /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_20_HEARTBEAT_TOP5/experiment_results_heartbeat.csv .
Rscript scripts/analyze_heartbeat_stability.R

Expected output:

Loading data from: .../experiment_results_heartbeat.csv

=== DATA SUMMARY ===
Total runs: 250
Configurations: 5
Runs per config: 50
Columns: 60

=== STABILITY RANKINGS ===
configuration mean_jitter_cv mean_convergence_rate ... rank_stability
1_0_1_1_1_0      0.0842         123.45         ...           1
1_0_1_1_1_1      0.0891         118.23         ...           2
1_1_0_1_1_1      0.1024          95.67         ...           3
...

[Visualizations generated]

═══════════════════════════════════════════════════════════
GOLDEN CONFIGURATION RECOMMENDATION
═══════════════════════════════════════════════════════════

✓ SELECTED: 1_0_1_1_1_0

  Stability Metrics:
    • Overall Stability Score: 82.45 / 100
    • Heartbeat Jitter (CV): 0.0842 (target < 0.15)
    • Outlier Ratio: 2.34%
    • Convergence Rate: 123.45 (higher = faster)
    • Load Coupling: 0.8234 (target > 0.7)
    • Settling Time: 1245.3 ticks
    • Mean Workload Duration: 12.34 ms

Step 3: Review Generated Files

The analysis will create:

stability_rankings.csv              # Detailed rankings by config
01_stability_scores.png             # Main boxplot comparison
02_jitter_control.png               # CV by config
03_convergence_speed.png            # Convergence comparison
04_load_coupling.png                # Load-response strength
05_metrics_heatmap.png              # Normalized metrics heatmap
06_tradeoff_jitter_vs_convergence.png  # Trade-off visualization

View the visualizations:

# In terminal or file viewer
ls -lh *.png
eog 01_stability_scores.png  # Or use your image viewer

Interpreting Results

Stability Score (Primary Metric)

Range: 0-100

0-50: Unstable, high jitter, poor convergence
50-70: Acceptable, moderate stability
70-85: Good, reliable for production
85-100: Excellent, reference-quality stability

The golden configuration should score ≥75.

Jitter (CV – Coefficient of Variation)

Measure: Standard deviation / Mean of heartbeat interval

Interpretation:

CV < 0.10: Excellent (steady heartbeat, ±10% variation)
0.10-0.15: Good (±10-15% variation)
0.15-0.20: Acceptable (±15-20% variation)
> 0.20: Poor (erratic heartbeat)

Goal: The golden configuration should have CV < 0.15.

Convergence Rate

Measure: How quickly decay slope fitting reaches steady state

Interpretation:

Higher values: Faster convergence
Lower values: Slower settling, may oscillate

The golden configuration should converge within the first 5000 heartbeat ticks.

Load-Response Coupling

Measure: Correlation between workload intensity and heartbeat interval

Interpretation:

Correlation > 0.8: Strong coupling, heartbeat responds proportionally to load
0.7-0.8: Good coupling, minor lag
< 0.7: Weak coupling, heartbeat may be unresponsive to load

Goal: The golden configuration should have correlation > 0.75.

Troubleshooting

Problem: Builds Fail for Certain Configs

Symptom: Error message during build phase

✗ Build failed for 1_1_0_1_1_1

Solution:

Check the build log:
```
cat run_logs/build_1_1_0_1_1_1.log
```
Look for compilation errors or missing definitions
Common causes:
- Missing header includes
- Circular dependencies between features
- Undefined macros

Next steps:

Modify the Makefile or source code to fix incompatibility
Re-run experiment (builds are skipped if config hasn't changed)

Problem: Runs Crash Mid-Execution

Symptom: Some runs show segfault or crash

✗ Configuration 1_1_0_1_1_1 crashed during execution

Solution:

Check individual run log:

tail -100 run_logs/1_1_0_1_1_1_run_1.log

Look for stack traces or error messages
Common causes:
- Buffer overflow in heartbeat metrics collection
- Memory leak in rolling window
- Race condition in heartbeat thread

Next steps:

If crash is in heartbeat code, check for thread safety issues

Run with GDB if deterministic:

gdb --args ./build/amd64/fastest/starforth --doe-experiment

Problem: Analysis Fails (Missing Dependencies)

Symptom: R script errors

Error: package 'tidyverse' not found

Solution:

# Install required R packages
R --quiet --no-save <<'EOF'
install.packages(c("tidyverse", "ggplot2", "gridExtra"))
quit()
EOF

# Then retry:
Rscript scripts/analyze_heartbeat_stability.R

Problem: Results Look Odd (All Configs Equal)

Symptom: All stability scores are identical or very similar

Possible causes:

Heartbeat data not being collected (check DoeMetrics struct)
Heartbeat thread disabled during build (check HEARTBEAT_THREAD_ENABLED=1)
Metrics not being extracted from VM (check metrics_from_vm())

Debug:

# Verify heartbeat was enabled in build
nm ./build/amd64/fastest/starforth | grep heartbeat
# Should show symbols

# Check a run log for metrics
tail -1 run_logs/1_0_1_1_1_0_run_1.log | tr ',' '\n' | nl
# Should show 60+ columns with numeric values

Golden Configuration Usage

Once identified, the golden configuration should be:

Documented: Create a reference config file

cat > docs/GOLDEN_CONFIG_HEARTBEAT.md <<'EOF'
# Golden Configuration (Phase 2 Heartbeat)

**Config:** [WINNER_HERE]
**Date:** [COMPLETION_DATE]
**Stability Score:** [SCORE]

Use this configuration as the default for:
- MamaForth production baseline
- PapaForth reference physics
- Future optimization benchmarks
EOF

Locked in build system: Add to Makefile

# Golden configuration from Phase 2 DoE (2025-11-20)
ENABLE_LOOP_1_HEAT_TRACKING ?= 1    # From golden config
ENABLE_LOOP_2_ROLLING_WINDOW ?= 0
ENABLE_LOOP_3_LINEAR_DECAY ?= 1
# ... etc

Used as baseline: Future experiments compare against this

# Experiment Phase 3: Compare new optimizations to golden
./scripts/run_comparison_doe.sh --baseline GOLDEN --test NEW_FEATURE

Archive & Handoff

After analysis:

# Create archive of results_run_01_2025_12_08
cd /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_20_HEARTBEAT_TOP5

tar czf ../2025_11_20_HEARTBEAT_TOP5.tar.gz \
    experiment_results_heartbeat.csv \
    configuration_manifest.txt \
    experiment_summary.txt

# Create analysis report
cat > ANALYSIS_SUMMARY.txt <<'EOF'
# StarForth Heartbeat DoE - Analysis Summary

**Golden Configuration:** [WINNER]
**Overall Stability Score:** [SCORE] / 100

## Metrics
- Jitter (CV): [VALUE] (target < 0.15)
- Convergence Rate: [VALUE] ticks
- Load Coupling: [VALUE] (target > 0.7)
- Settling Time: [VALUE] ticks

## Interpretation
[Details about why this config is optimal]

## Recommendations
[Future work based on this config]
EOF

# Archive everything
tar czf ../2025_11_20_HEARTBEAT_RESULTS_FINAL.tar.gz \
    *.csv *.png *.txt

echo "Archive ready: ../2025_11_20_HEARTBEAT_RESULTS_FINAL.tar.gz"

Next Steps (Phase 3)

Once the golden configuration is identified:

Lock it in code: Update CLAUDE.md with golden config
Create MamaForth baseline: Use golden config for next experiment
Plan Phase 3: Optimization experiments around golden config
Document learnings: What made this config optimal?

Success Criteria

✓ All 250 runs completed without segfault ✓ CSV file has 251 rows (header + 250 data) ✓ All heartbeat metrics are numeric (not NaN or Inf) ✓ Stability rankings are clear (no tie for first place) ✓ R analysis generates all 6 visualizations ✓ Golden config identified with p < 0.05 significance ✓ Golden config stability score ≥ 75

References

Design: docs/DOE_HEARTBEAT_EXPERIMENT_DESIGN.md
Script: scripts/run_factorial_doe_with_heartbeat.sh
Analysis: scripts/analyze_heartbeat_stability.R
Metrics Schema: docs/DOE_METRICS_SCHEMA.md

Questions? Review CLAUDE.md or examine run logs for details.

16 KiB Raw Blame History Unescape Escape

StarForth Heartbeat DoE Experiment – Execution Guide

Quick Start

What This Experiment Does

Phase 2: Focus on Heartbeat Stability

The 5 Elite Configurations

Pre-Execution Checklist

1. Build System Ready

2. Storage Space

3. System Stability

4. Heartbeat Thread Configuration

Execution Steps

Step 1: Launch the Experiment

Step 2: Monitor Execution

Step 3: Completion

Post-Execution Analysis

Step 1: Verify CSV Output

Step 2: Run R Analysis

Step 3: Review Generated Files

Interpreting Results

Stability Score (Primary Metric)

Jitter (CV – Coefficient of Variation)

Convergence Rate

Load-Response Coupling

Troubleshooting

Problem: Builds Fail for Certain Configs

Problem: Runs Crash Mid-Execution

Problem: Analysis Fails (Missing Dependencies)

Problem: Results Look Odd (All Configs Equal)

Golden Configuration Usage

Archive & Handoff

Next Steps (Phase 3)

Success Criteria

References

16 KiB

Raw Blame History