457 lines
12 KiB
Plaintext
457 lines
12 KiB
Plaintext
// Moved from docs/src/INTEGRATION_NOTES.adoc to docs/working/scratch/src/INTEGRATION_NOTES.adoc on 2026-06-16 (docs reorg Phase 2)
|
||
== Integration Scripts Documentation
|
||
:toc: left
|
||
:toc-title: Contents
|
||
:toclevels: 3
|
||
xref:./README.adoc[← Back to Documentation Index]
|
||
|
||
|
||
|
||
=== Overview
|
||
|
||
Two complementary sync scripts for bidirectional integration between
|
||
StarForth and StarshipOS.
|
||
|
||
=== Scripts
|
||
|
||
==== 1. `+integrator.sh+` (StarForth → StarshipOS) ✅ WORKING
|
||
|
||
* *Location*: `+StarForth/maint/integrator.sh+`
|
||
* *Direction*: StarForth → StarshipOS
|
||
* *Source*: StarForth root
|
||
* *Destination*: `+$STARSHIPOS_ROOT/l4/pkg/starforth/server+`
|
||
* *Quarantine*: `+StarForth/maint/quarantine+`
|
||
* *Path stripping*: Strips `+server/+` prefix from files
|
||
* *Status*: Fully debugged and tested (spent all day on this)
|
||
|
||
==== 2. `+integrator.sh+` (StarshipOS → StarForth) 🆕 NEW
|
||
|
||
* *Final Location*: `+StarshipOS/maint/integrator.sh+`
|
||
* *Direction*: StarshipOS → StarForth (reverse)
|
||
* *Source*: StarshipOS root
|
||
* *Destination*: `+$STARFORTH_ROOT+`
|
||
* *Quarantine*: `+StarshipOS/maint/quarantine+`
|
||
* *Path stripping*: Strips `+l4/pkg/starforth/server/+` prefix from
|
||
files
|
||
* *Status*: Generated based on working integrator.sh, needs testing
|
||
|
||
=== The Rules (Cap’t Bob Sez)
|
||
|
||
[arabic]
|
||
. *Never mkdir into target* — both repos must already exist
|
||
. *Blacklist = Law* — anything matching it is ignored
|
||
. *If file exists at same relative path in destination* → overwrite
|
||
(green)
|
||
. *Otherwise* → quarantine for review (orange)
|
||
|
||
=== How They Work
|
||
|
||
Both scripts: 1. Read files from `+maint/mergefiles.txt+` (or generate
|
||
from `+git diff --name-only HEAD~1+`) 2. Check each file against
|
||
`+maint/blacklist.txt+` 3. Strip appropriate path prefix to compute
|
||
destination path 4. If destination exists → overwrite (safe) 5. If
|
||
destination doesn’t exist → quarantine for Captain’s review 6. Refresh
|
||
git index when done
|
||
|
||
=== Next Steps
|
||
|
||
* [ ] Move `+integrator.sh+` to `+StarshipOS/maint/+`
|
||
* [ ] Test `+integrator.sh+` in StarshipOS
|
||
* [ ] Verify blacklist.txt and mergefiles.txt exist in StarshipOS/maint/
|
||
* [ ] Test round-trip sync (both directions)
|
||
|
||
=== Notes
|
||
|
||
* Both scripts are meant to be run from their respective repository
|
||
roots
|
||
* Quarantine prevents accidental file creation in unexpected locations
|
||
* The path stripping logic is the key difference between the two scripts
|
||
|
||
=== ⚠️ Special Cases
|
||
|
||
==== Documentation Directory (`+docs/+`)
|
||
|
||
*Issue*: Documentation needs bidirectional sync but has different paths:
|
||
- StarForth: `+docs/+` - StarshipOS: `+l4/pkg/starforth/docs/+`
|
||
|
||
*Current Status*: ⏳ *NOT YET INTEGRATED* - Manual handling required
|
||
*Captain’s Note*: Will be integrated into integrator.sh later with
|
||
custom path mapping logic
|
||
|
||
*Paths*: - `+/home/rajames/CLionProjects/StarForth/docs+` -
|
||
`+/home/rajames/CLionProjects/StarshipOS/l4/pkg/starforth/docs+`
|
||
|
||
*Future Enhancement*: Add special case handling in both integrators for
|
||
`+docs/+` directory with custom path transformations
|
||
|
||
'''''
|
||
|
||
=== Test & Repair Cycle (StarshipOS → StarForth)
|
||
|
||
==== Date: 2025-10-19
|
||
|
||
==== Status: ✅ PRODUCTION READY
|
||
|
||
This section documents the complete debugging process for
|
||
`+StarshipOS/maint/integrator.sh+`. Use this methodology to test and
|
||
debug the reverse script (`+StarForth/maint/integrator.sh+`).
|
||
|
||
'''''
|
||
|
||
==== 🐛 Bugs Found & Fixed
|
||
|
||
===== Bug #1: Blacklist Regex Matching (Line 43) - CRITICAL
|
||
|
||
*Problem*: Used `+grep -Fqx+` which treats patterns as *fixed strings*,
|
||
but `+blacklist.txt+` contains *regex patterns* like
|
||
`+(^|/)?Makefile($|[.].*)+`.
|
||
|
||
*Symptom*: Makefiles and blacklisted files were NOT being blocked.
|
||
|
||
*Original Code*:
|
||
|
||
[source,bash]
|
||
----
|
||
if grep -Fqx -f "$BLACKLIST" <<< "$FILE"; then
|
||
----
|
||
|
||
*Fix*:
|
||
|
||
[source,bash]
|
||
----
|
||
if grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' "$BLACKLIST") <<< "$FILE"; then
|
||
----
|
||
|
||
*Changes*: - `+-F+` removed (enables regex) - `+-x+` removed (allows
|
||
partial line matching) - `+-E+` added (extended regex) - `+sed+` filter
|
||
added (see Bug #2)
|
||
|
||
'''''
|
||
|
||
===== Bug #2: Empty Lines in Blacklist - CRITICAL
|
||
|
||
*Problem*: Line 5 in `+blacklist.txt+` is empty. Empty patterns in
|
||
`+grep -f+` match *everything*.
|
||
|
||
*Symptom*: ALL files were being blocked when blacklist was enabled.
|
||
|
||
*Test*:
|
||
|
||
[source,bash]
|
||
----
|
||
# This blocks EVERYTHING because of empty line:
|
||
echo "any_file.txt" | grep -Eq -f blacklist.txt && echo "BLOCKED"
|
||
|
||
# This works correctly:
|
||
echo "any_file.txt" | grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' blacklist.txt)
|
||
----
|
||
|
||
*Fix*: Filter out empty lines and comments using
|
||
`+sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d'+`
|
||
|
||
'''''
|
||
|
||
===== Bug #3: Case Sensitivity in mergefiles.txt - MINOR
|
||
|
||
*Problem*: `+mergefiles.txt+` listed `+testing.md+` but actual file is
|
||
`+TESTING.md+`.
|
||
|
||
*Symptom*: File skipped by line 40 (`+[[ -f "$SRC" ]] || continue+`).
|
||
|
||
*Fix*: Corrected filename in `+mergefiles.txt+`.
|
||
|
||
'''''
|
||
|
||
==== 📋 Testing Methodology
|
||
|
||
===== Phase 1: Basic Execution Test
|
||
|
||
[source,bash]
|
||
----
|
||
# Run with existing mergefiles.txt
|
||
bash maint/integrator.sh
|
||
----
|
||
|
||
*Expected*: Script runs without errors, processes files listed in
|
||
`+mergefiles.txt+`.
|
||
|
||
'''''
|
||
|
||
===== Phase 2: Blacklist Enforcement Test
|
||
|
||
Create comprehensive test file:
|
||
|
||
[source,bash]
|
||
----
|
||
cat > maint/mergefiles_blacklist_test.txt << 'EOF'
|
||
l4/pkg/starforth/server/Makefile
|
||
l4/pkg/starforth/server/Makefile.inc
|
||
l4/pkg/starforth/server/README.md
|
||
l4/pkg/starforth/server/src/panic.c
|
||
l4/pkg/starforth/server/src/core.c
|
||
l4/pkg/starforth/server/Control
|
||
maint/integrator.sh
|
||
EOF
|
||
|
||
# Create test file that should be quarantined
|
||
touch l4/pkg/starforth/server/src/core.c
|
||
|
||
# Clear quarantine
|
||
rm -rf maint/quarantine/*
|
||
|
||
# Swap mergefiles and run test
|
||
cp maint/mergefiles.txt maint/mergefiles.txt.backup
|
||
cp maint/mergefiles_blacklist_test.txt maint/mergefiles.txt
|
||
bash maint/integrator.sh
|
||
----
|
||
|
||
*Expected Results*: | File | Expected Action | Reason | |——|—————-|———|
|
||
| `+Makefile+` | 🔴 *BLOCKED* | Matches `+(^|/)?Makefile($|[.].*)+` | |
|
||
`+Makefile.inc+` | 🔴 *BLOCKED* | Matches `+(^|/)?Makefile($|[.].*)+` |
|
||
| `+README.md+` | 🟢 *OVERWRITE* | Not blacklisted, exists in StarForth
|
||
| | `+panic.c+` | 🔴 *BLOCKED* | Matches `+server/src/panic\.c$+` | |
|
||
`+core.c+` | 🟠 *QUARANTINE* | Not blacklisted, doesn’t exist in
|
||
StarForth | | `+Control+` | 🔴 *BLOCKED* | Matches
|
||
`+(^|/)?Control($|/)+` | | `+integrator.sh+` | 🟢 *OVERWRITE* | Not
|
||
blacklisted, exists in StarForth |
|
||
|
||
*Verify Blacklist*:
|
||
|
||
[source,bash]
|
||
----
|
||
# Test individual patterns
|
||
echo "l4/pkg/starforth/server/Makefile" | grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' maint/blacklist.txt) && echo "BLOCKED" || echo "NOT BLOCKED"
|
||
|
||
echo "l4/pkg/starforth/server/src/panic.c" | grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' maint/blacklist.txt) && echo "BLOCKED" || echo "NOT BLOCKED"
|
||
----
|
||
|
||
'''''
|
||
|
||
===== Phase 3: Path Stripping Verification
|
||
|
||
*Test*: Verify that `+l4/pkg/starforth/server/+` prefix is correctly
|
||
stripped.
|
||
|
||
[source,bash]
|
||
----
|
||
# Check where files land in destination
|
||
FILE="l4/pkg/starforth/server/README.md"
|
||
RELATIVE="${FILE#l4/pkg/starforth/server/}"
|
||
echo "Source: $FILE"
|
||
echo "Destination: $STARFORTH_ROOT/$RELATIVE"
|
||
----
|
||
|
||
*Expected*: - Source: `+l4/pkg/starforth/server/README.md+` -
|
||
Destination: `+/home/rajames/CLionProjects/StarForth/README.md+`
|
||
|
||
*Verify*:
|
||
|
||
[source,bash]
|
||
----
|
||
diff l4/pkg/starforth/server/README.md $STARFORTH_ROOT/README.md && echo "✓ Synced correctly"
|
||
----
|
||
|
||
'''''
|
||
|
||
===== Phase 4: Overwrite vs Quarantine Logic
|
||
|
||
*Rule 3*: If file exists at destination → *overwrite* (green) *Rule 4*:
|
||
If file doesn’t exist → *quarantine* (orange)
|
||
|
||
*Test Overwrite*:
|
||
|
||
[source,bash]
|
||
----
|
||
# README.md exists in StarForth root
|
||
test -f $STARFORTH_ROOT/README.md && echo "Exists → should OVERWRITE"
|
||
----
|
||
|
||
*Test Quarantine*:
|
||
|
||
[source,bash]
|
||
----
|
||
# core.c doesn't exist in StarForth
|
||
test -f $STARFORTH_ROOT/src/core.c || echo "Doesn't exist → should QUARANTINE"
|
||
|
||
# Verify it's quarantined
|
||
test -f maint/quarantine/l4/pkg/starforth/server/src/core.c && echo "✓ Correctly quarantined"
|
||
----
|
||
|
||
'''''
|
||
|
||
===== Phase 5: Auto-Generation from Git (Webhook Mode)
|
||
|
||
*Test*: Script generates `+mergefiles.txt+` from
|
||
`+git diff --name-only HEAD~1+` when file doesn’t exist.
|
||
|
||
[source,bash]
|
||
----
|
||
# Remove mergefiles.txt to trigger auto-generation
|
||
mv maint/mergefiles.txt maint/mergefiles.txt.manual
|
||
|
||
# Run script - should auto-generate
|
||
bash maint/integrator.sh
|
||
|
||
# Check what was generated
|
||
cat maint/mergefiles.txt
|
||
----
|
||
|
||
*Expected Output*:
|
||
|
||
....
|
||
[INFO] No mergefiles list found — generating diff from last commit.
|
||
....
|
||
|
||
*Verify*:
|
||
|
||
[source,bash]
|
||
----
|
||
git diff --name-only HEAD~1 # Should match mergefiles.txt content
|
||
----
|
||
|
||
'''''
|
||
|
||
==== ✅ Final Production Test
|
||
|
||
[source,bash]
|
||
----
|
||
# 1. Restore original mergefiles.txt
|
||
cp maint/mergefiles.txt.backup maint/mergefiles.txt
|
||
|
||
# 2. Clear quarantine
|
||
rm -rf maint/quarantine/*
|
||
|
||
# 3. Run production test
|
||
bash maint/integrator.sh
|
||
|
||
# 4. Verify sync
|
||
diff maint/integrator.sh $STARFORTH_ROOT/maint/integrator.sh && echo "✓ Synced"
|
||
----
|
||
|
||
*Expected Output*:
|
||
|
||
....
|
||
🛰️ Quark webhook engaged — StarshipOS → StarForth (Reverse Sync)
|
||
|
||
[Quarantining] l4/pkg/starforth/server/TESTING.md
|
||
[Overwriting] maint/integrator.sh
|
||
|
||
🧾 First Officer's Reverse Report:
|
||
• Destination base: /home/rajames/CLionProjects/StarForth
|
||
• Quarantine: /home/rajames/CLionProjects/StarshipOS/maint/quarantine
|
||
• Merge list: 2 files processed
|
||
|
||
Awaiting Cappy's signature: Captain Bob ✍️
|
||
....
|
||
|
||
'''''
|
||
|
||
==== 🔧 Debugging Commands Reference
|
||
|
||
===== Check Blacklist Patterns
|
||
|
||
[source,bash]
|
||
----
|
||
# View blacklist with line numbers
|
||
cat -n maint/blacklist.txt
|
||
|
||
# Test specific file against blacklist
|
||
FILE="l4/pkg/starforth/server/Makefile"
|
||
grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' maint/blacklist.txt) <<< "$FILE" && echo "BLOCKED" || echo "NOT BLOCKED"
|
||
----
|
||
|
||
===== Verify Path Stripping
|
||
|
||
[source,bash]
|
||
----
|
||
# For StarshipOS → StarForth
|
||
FILE="l4/pkg/starforth/server/src/vm.c"
|
||
RELATIVE="${FILE#l4/pkg/starforth/server/}"
|
||
echo "$RELATIVE" # Should output: src/vm.c
|
||
|
||
# For StarForth → StarshipOS (reverse)
|
||
FILE="server/src/vm.c"
|
||
RELATIVE="${FILE#server/}"
|
||
echo "$RELATIVE" # Should output: src/vm.c
|
||
----
|
||
|
||
===== Check File Existence
|
||
|
||
[source,bash]
|
||
----
|
||
# Check source
|
||
ls -la l4/pkg/starforth/server/TESTING.md
|
||
|
||
# Check destination
|
||
ls -la $STARFORTH_ROOT/TESTING.md
|
||
|
||
# Check quarantine
|
||
ls -la maint/quarantine/l4/pkg/starforth/server/TESTING.md
|
||
----
|
||
|
||
===== Monitor Git Index Refresh
|
||
|
||
[source,bash]
|
||
----
|
||
# Manual refresh (what script does)
|
||
cd $STARFORTH_ROOT && git update-index --really-refresh
|
||
----
|
||
|
||
'''''
|
||
|
||
==== 🎯 Apply to StarForth → StarshipOS Script
|
||
|
||
Use the same methodology to test `+StarForth/maint/integrator.sh+`:
|
||
|
||
*Key Differences*: 1. *Path stripping*: `+server/+` →
|
||
`+l4/pkg/starforth/server/+` 2. *Direction*: StarForth → StarshipOS 3.
|
||
*Blacklist patterns*: Different (StarForth-specific patterns)
|
||
|
||
*Same Fixes Needed*: - Change `+grep -Fqx+` → `+grep -Eq+` - Add `+sed+`
|
||
filter for empty lines and comments - Verify case sensitivity in
|
||
mergefiles.txt
|
||
|
||
*Test Pattern*:
|
||
|
||
[source,bash]
|
||
----
|
||
cd /home/rajames/CLionProjects/StarForth
|
||
|
||
# Phase 1: Basic test
|
||
bash maint/integrator.sh
|
||
|
||
# Phase 2: Blacklist test (create test mergefiles)
|
||
# Phase 3: Path stripping test
|
||
# Phase 4: Overwrite vs quarantine test
|
||
# Phase 5: Auto-generation test
|
||
----
|
||
|
||
'''''
|
||
|
||
==== 📊 Test Results Summary
|
||
|
||
[width="100%",cols="51%,26%,23%",options="header",]
|
||
|===
|
||
|Test Category |Status |Notes
|
||
|Basic Execution |✅ PASS |Script runs without errors
|
||
|
||
|Blacklist Enforcement |✅ PASS |Regex patterns work, empty lines
|
||
filtered
|
||
|
||
|Path Stripping |✅ PASS |`+l4/pkg/starforth/server/+` removed correctly
|
||
|
||
|Overwrite Logic |✅ PASS |Existing files overwritten (README.md, vm.h)
|
||
|
||
|Quarantine Logic |✅ PASS |New files quarantined (TESTING.md, core.c)
|
||
|
||
|Auto-Generation |✅ PASS |Generates from `+git diff HEAD~1+`
|
||
|
||
|Git Index Refresh |✅ PASS |StarForth index updated
|
||
|===
|
||
|
||
*Final Status*: 🚀 *PRODUCTION READY* for git push webhook
|
||
|
||
'''''
|
||
|
||
_Debugged: 2025-10-19_ _Captain Bob ✍️_
|