652 lines
14 KiB
Plaintext
652 lines
14 KiB
Plaintext
// Moved from docs/src/build-and-tooling/INIT_TOOLS.adoc to docs/working/scratch/src/build-and-tooling/INIT_TOOLS.adoc on 2026-06-16 (docs reorg Phase 2)
|
||
== StarForth INIT Tools
|
||
:toc: left
|
||
:toc-title: Contents
|
||
:toclevels: 3
|
||
xref:../README.adoc[← Back to Documentation Index]
|
||
|
||
|
||
|
||
*Tools for managing init.4th ↔ disk image synchronization*
|
||
|
||
'''''
|
||
|
||
=== Overview
|
||
|
||
StarForth provides a complete toolchain for managing the bidirectional
|
||
flow between `+./conf/init.4th+` ( version-controlled source of truth)
|
||
and disk images (`+.img+` files tracked via Git LFS). This enables
|
||
collaborative development where:
|
||
|
||
[arabic]
|
||
. Developers work in disk images (native Forth editing via `+LIST+`,
|
||
`+EDIT+`, etc.)
|
||
. Tools extract `+(- ... )+` annotated blocks into `+init.4th.bak+`
|
||
. Changes to `+init.4th.bak+` are reviewed in Git PRs
|
||
. Approved changes are applied back to disk images
|
||
|
||
'''''
|
||
|
||
=== Tools
|
||
|
||
==== `+tools/extract-init+` (C)
|
||
|
||
*Purpose:* Scan a disk image for blocks marked with `+(- ... )+` headers
|
||
and extract them to `+init.4th.bak+`
|
||
|
||
*Usage:*
|
||
|
||
[source,bash]
|
||
----
|
||
./tools/extract-init --img=disks/example.img --out=conf/init-4.4th \
|
||
[--fbs=1024] [--start=0] [--end=-1] \
|
||
[--loose] [--require-close] [--max=N]
|
||
----
|
||
|
||
*Options:*
|
||
|
||
* `+--img=PATH+` - Disk image to scan (required)
|
||
* `+--out=PATH+` - Output init.4th file (required)
|
||
* `+--fbs=N+` - Forth block size in bytes (default: 1024)
|
||
* `+--start=N+` - First block number to scan (default: 0)
|
||
* `+--end=N+` - Last block number to scan, -1 for EOF (default: -1)
|
||
* `+--loose+` - Allow BOM/whitespace before `+(-+` header
|
||
* `+--require-close+` - Require closing `+)+` in same block
|
||
* `+--max=N+` - Stop after extracting N blocks
|
||
|
||
*Output format:*
|
||
|
||
[source,forth]
|
||
----
|
||
(- StarForth INIT export )
|
||
(- Source: disks/example.img )
|
||
(- FBS: 1024 Range: 0 .. 499999 Mode: loose RequireClose: yes Max: unlimited )
|
||
|
||
Block 2048
|
||
(- Large Letter F )
|
||
: STAR 42 EMIT ;
|
||
...
|
||
|
||
Block 3000
|
||
(- Another block )
|
||
...
|
||
----
|
||
|
||
*Detection algorithm:*
|
||
|
||
[arabic]
|
||
. Read each block sequentially
|
||
. Check for `+(-+` at start (with optional BOM/whitespace if
|
||
`+--loose+`)
|
||
. If `+--require-close+`, verify closing `+)+` exists in same block
|
||
. Extract blocks that match criteria
|
||
|
||
'''''
|
||
|
||
==== `+tools/apply-init+` (C)
|
||
|
||
*Purpose:* Parse `+init.4th.bak+` and write its `+Block N+` sections to
|
||
specific blocks in a disk image
|
||
|
||
*Usage:*
|
||
|
||
[source,bash]
|
||
----
|
||
./tools/apply-init --img=disks/example.img --in=conf/init-4.4th \
|
||
[--fbs=1024] [--start=0] [--end=N] \
|
||
[--clip] [--dry-run] [--verify] [--verbose]
|
||
----
|
||
|
||
*Options:*
|
||
|
||
* `+--img=PATH+` - Disk image to modify (required)
|
||
* `+--in=PATH+` - init.4th source file (required)
|
||
* `+--fbs=N+` - Forth block size in bytes (default: 1024)
|
||
* `+--start=N+` - Guard: refuse writes to blocks < N (safety)
|
||
* `+--end=N+` - Guard: refuse writes to blocks > N (safety)
|
||
* `+--clip+` - Truncate block content if > FBS (default: error)
|
||
* `+--dry-run+` - Parse and show plan, don’t write
|
||
* `+--verify+` - Read back and compare after write
|
||
* `+--verbose+` - Show detailed write operations
|
||
|
||
*Parsing:*
|
||
|
||
[source,forth]
|
||
----
|
||
(- This is ignored - not a block header )
|
||
|
||
Block 2048
|
||
(- This is the block header comment )
|
||
: WORD1 ... ;
|
||
: WORD2 ... ;
|
||
|
||
(- These lines outside Block N are ignored )
|
||
|
||
Block 3000
|
||
(- Another block )
|
||
...
|
||
----
|
||
|
||
Lines outside `+Block N+` sections (like top-level comments) are
|
||
ignored. Only lines between `+Block N+` and the next `+Block M+` (or
|
||
EOF) are written to block N.
|
||
|
||
*Guards:*
|
||
|
||
* `+--start=1024+` prevents accidentally overwriting boot area (blocks
|
||
0-1023)
|
||
* `+--end=4095+` prevents writing beyond a certain range
|
||
* Both are optional; by default, no guards
|
||
|
||
*Verification:*
|
||
|
||
* With `+--verify+`, each block is re-read after write and compared
|
||
byte-for-byte
|
||
* Ensures disk write succeeded and data integrity maintained
|
||
|
||
'''''
|
||
|
||
==== `+scripts/update-init.sh+` (Interactive)
|
||
|
||
*Purpose:* User-friendly wrapper for `+extract-init+` with interactive
|
||
prompts
|
||
|
||
*Usage:*
|
||
|
||
[source,bash]
|
||
----
|
||
./scripts/update-init.sh
|
||
----
|
||
|
||
*Workflow:*
|
||
|
||
[arabic]
|
||
. Lists available disk images in `+./disks/+`
|
||
. Prompts user to select one
|
||
. Asks for extraction parameters (mode, FBS, range, etc.)
|
||
. Shows full command before execution
|
||
. Asks for confirmation
|
||
. Runs `+extract-init+` and writes to `+conf/init.4th+`
|
||
|
||
*Use case:* After editing blocks in a disk image, run this to update
|
||
`+init.4th.bak+` for Git commit.
|
||
|
||
'''''
|
||
|
||
==== `+scripts/apply-init.sh+` (Interactive)
|
||
|
||
*Purpose:* User-friendly wrapper for `+apply-init+` with interactive
|
||
prompts
|
||
|
||
*Usage:*
|
||
|
||
[source,bash]
|
||
----
|
||
./scripts/apply-init.sh
|
||
----
|
||
|
||
*Workflow:*
|
||
|
||
[arabic]
|
||
. Lists available disk images in `+./disks/+`
|
||
. Prompts user to select one
|
||
. Asks for safety guards, verification, etc.
|
||
. Shows full command before execution
|
||
. Asks for confirmation
|
||
. Runs `+apply-init+` to write `+conf/init.4th+` blocks to disk
|
||
|
||
*Use case:* After merging PR with init.4th changes, run this to update
|
||
disk images.
|
||
|
||
'''''
|
||
|
||
=== Workflow Patterns
|
||
|
||
==== Pattern 1: Extract from Disk → Git
|
||
|
||
*Scenario:* You’ve been working in a disk image and want to commit
|
||
changes to Git.
|
||
|
||
[source,bash]
|
||
----
|
||
# 1. Work in disk (using StarForth REPL)
|
||
./build/starforth --disk-img=disks/mywork.img
|
||
ok> 2048 LIST
|
||
ok> 2048 EDIT
|
||
... (edit block) ...
|
||
|
||
# 2. Extract marked blocks to init.4th
|
||
./scripts/update-init.sh
|
||
# (select disk, confirm extraction)
|
||
|
||
# 3. Review changes
|
||
git diff conf/init-4.4th
|
||
|
||
# 4. Commit
|
||
git add conf/init-4.4th
|
||
git commit -m "Add new INIT words: FOO and BAR"
|
||
git push
|
||
----
|
||
|
||
'''''
|
||
|
||
==== Pattern 2: Apply from Git → Disk
|
||
|
||
*Scenario:* A PR was merged with init.4th changes, and you need to
|
||
update your disk.
|
||
|
||
[source,bash]
|
||
----
|
||
# 1. Pull latest changes
|
||
git pull origin master
|
||
|
||
# 2. Apply to your disk
|
||
./scripts/apply-init.sh
|
||
# (select disk, confirm application)
|
||
|
||
# 3. Verify in REPL
|
||
./build/starforth --disk-img=disks/mywork.img
|
||
ok> 2048 LIST
|
||
ok> ( verify new code is there )
|
||
----
|
||
|
||
'''''
|
||
|
||
==== Pattern 3: Command-Line Automation (CI/CD)
|
||
|
||
*Extract:*
|
||
|
||
[source,bash]
|
||
----
|
||
./tools/extract-init \
|
||
--img=disks/production.img \
|
||
--out=conf/init-4.4th \
|
||
--fbs=1024 \
|
||
--start=0 \
|
||
--end=-1 \
|
||
--loose \
|
||
--require-close
|
||
----
|
||
|
||
*Apply:*
|
||
|
||
[source,bash]
|
||
----
|
||
./tools/apply-init \
|
||
--img=disks/production.img \
|
||
--in=conf/init-4.4th \
|
||
--fbs=1024 \
|
||
--start=1024 \
|
||
--clip \
|
||
--verify \
|
||
--verbose
|
||
----
|
||
|
||
'''''
|
||
|
||
=== Block Naming Convention
|
||
|
||
*Recommended disk naming:* `+<hostname>-<username>-<version>.img+`
|
||
|
||
Examples:
|
||
|
||
* `+workstation-alice-1.0.img+`
|
||
* `+laptop-bob-2.3.img+`
|
||
* `+starship-rajames-1.0.img+`
|
||
|
||
*Tracked in Git:*
|
||
|
||
* `+disks/*.img+` → Git LFS (binary, large)
|
||
* `+conf/init.4th+` → Git (text, diffable)
|
||
|
||
'''''
|
||
|
||
=== The `+(-+` Metadata Marker
|
||
|
||
==== Purpose
|
||
|
||
The `+(-+` word serves dual purposes:
|
||
|
||
[arabic]
|
||
. *Runtime:* Forth comment (like `+(+` but with dash)
|
||
. *Tooling:* Metadata marker for block extraction
|
||
|
||
==== Syntax
|
||
|
||
[source,forth]
|
||
----
|
||
(- <description> )
|
||
----
|
||
|
||
Example:
|
||
|
||
[source,forth]
|
||
----
|
||
Block 2048
|
||
(- String utilities: COUNT, COMPARE, SEARCH )
|
||
: COUNT DUP 1+ SWAP C@ ;
|
||
: COMPARE ... ;
|
||
----
|
||
|
||
==== Why `+(-+` instead of `+(+`?
|
||
|
||
[arabic]
|
||
. Distinguishes extraction-worthy blocks from regular comments
|
||
. Avoids false positives (many Forth blocks use `+(+` for normal
|
||
comments)
|
||
. Easy to search: `+grep "^(-"+` finds all metadata markers
|
||
. Doesn’t conflict with standard Forth syntax
|
||
|
||
==== Implementation
|
||
|
||
Located in `+src/word_source/starforth_words.c:102-120+`:
|
||
|
||
[source,c]
|
||
----
|
||
void starforth_word_paren_dash(VM *vm) {
|
||
/* Consume input from "(- " to first ")" */
|
||
int depth = 1;
|
||
while (vm->input_pos < vm->input_length && depth > 0) {
|
||
char c = vm->input_buffer[vm->input_pos++];
|
||
if (c == '(') depth++;
|
||
else if (c == ')') depth--;
|
||
}
|
||
if (depth > 0) {
|
||
log_message(LOG_WARN, "(- comment not terminated");
|
||
}
|
||
log_message(LOG_DEBUG, "(- comment parsed (init.4th metadata marker)");
|
||
}
|
||
----
|
||
|
||
Registered in both FORTH and STARFORTH vocabularies.
|
||
|
||
'''''
|
||
|
||
=== Safety Features
|
||
|
||
==== 1. Guard Rails
|
||
|
||
*Problem:* Accidentally overwriting boot area (blocks 0-1023) or system
|
||
ranges
|
||
|
||
*Solution:* `+--start+` and `+--end+` guards
|
||
|
||
[source,bash]
|
||
----
|
||
# Only allow writes to blocks 1024-4095
|
||
./tools/apply-init --img=disk.img --in=init-4.4th --start=1024 --end=4095
|
||
----
|
||
|
||
If init.4th contains `+Block 512+`, apply-init will refuse:
|
||
|
||
....
|
||
Refusing to write block 512 (< start guard 1024)
|
||
....
|
||
|
||
==== 2. Verification
|
||
|
||
*Problem:* Disk write might fail silently (bad sector, filesystem issue)
|
||
|
||
*Solution:* `+--verify+` flag
|
||
|
||
[source,bash]
|
||
----
|
||
./tools/apply-init --img=disk.img --in=init-4.4th --verify
|
||
----
|
||
|
||
After writing each block:
|
||
|
||
[arabic]
|
||
. Seek back to block start
|
||
. Read full block (1024 bytes)
|
||
. Compare with what was written
|
||
. Error if mismatch
|
||
|
||
==== 3. Dry Run
|
||
|
||
*Problem:* Want to see what would happen without committing
|
||
|
||
*Solution:* `+--dry-run+` flag
|
||
|
||
[source,bash]
|
||
----
|
||
./tools/apply-init --img=disk.img --in=init-4.4th --dry-run --verbose
|
||
----
|
||
|
||
Output:
|
||
|
||
....
|
||
[plan] Block 2048 (171 byte payload)
|
||
[plan] Block 3000 (56 byte payload)
|
||
[plan] Block 3001 (30 byte payload)
|
||
Sections parsed: 3; would write 3 block(s)
|
||
....
|
||
|
||
No actual writes occur.
|
||
|
||
==== 4. Content Length Checks
|
||
|
||
*Problem:* Block text exceeds FBS (1024 bytes)
|
||
|
||
*Solution:* Error by default, `+--clip+` to truncate
|
||
|
||
Without `+--clip+`:
|
||
|
||
....
|
||
Block 2048: content 1127 > FBS 1024 (use --clip to truncate)
|
||
ERROR: ...
|
||
....
|
||
|
||
With `+--clip+`:
|
||
|
||
....
|
||
Wrote block 2048 (1024/1024 bytes used, padded 0)
|
||
(Warning: 103 bytes truncated)
|
||
....
|
||
|
||
'''''
|
||
|
||
=== Building the Tools
|
||
|
||
*Automatic:*
|
||
|
||
Scripts (`+4th-update-init.sh+`, `+4th-apply-init.sh+`) check timestamps and
|
||
rebuild if source is newer than binary.
|
||
|
||
*Manual:*
|
||
|
||
[source,bash]
|
||
----
|
||
# extract-init
|
||
gcc -std=c99 -O2 -Wall -Wextra -Werror \
|
||
-o tools/extract-init tools/extract_init.c
|
||
|
||
# apply-init
|
||
gcc -std=gnu99 -D_FILE_OFFSET_BITS=64 -O2 -Wall -Wextra -Werror \
|
||
-o tools/apply-init tools/apply_init.c
|
||
----
|
||
|
||
*Requirements:*
|
||
|
||
* C99 compiler (gcc or clang)
|
||
* POSIX file APIs (fseeko, ftello)
|
||
* 64-bit file offset support
|
||
|
||
'''''
|
||
|
||
=== Testing
|
||
|
||
==== Unit Test (Round-Trip)
|
||
|
||
[source,bash]
|
||
----
|
||
# 1. Apply init.4th to disk
|
||
./tools/apply-init --img=test.img --in=conf/init-4.4th --fbs=1024 --verify
|
||
|
||
# 2. Extract back
|
||
./tools/extract-init --img=test.img --out=/tmp/extracted.4th --fbs=1024 --loose
|
||
|
||
# 3. Compare (ignoring metadata header)
|
||
tail -n +4 /tmp/extracted.4th > /tmp/extracted-body.4th
|
||
diff -u conf/init-4.4th /tmp/extracted-body.4th
|
||
|
||
# Should be identical (or only whitespace diffs)
|
||
----
|
||
|
||
==== Integration Test (Full Workflow)
|
||
|
||
[source,bash]
|
||
----
|
||
# 1. Edit a block in disk image
|
||
echo ": TEST 42 . ; " | ./build/starforth --disk-img=test.img <<EOF
|
||
2048 BLOCK
|
||
( edit commands... )
|
||
EOF
|
||
|
||
# 2. Extract to init.4th
|
||
./tools/extract-init --img=test.img --out=conf/init-4.4th --fbs=1024 --loose --require-close
|
||
|
||
# 3. Verify TEST word is in init.4th
|
||
grep "TEST" conf/init-4.4th
|
||
|
||
# 4. Apply to fresh disk
|
||
cp test.img test-fresh.img
|
||
./tools/apply-init --img=test-fresh.img --in=conf/init-4.4th --fbs=1024 --verify
|
||
|
||
# 5. Verify TEST word works
|
||
echo "TEST" | ./build/starforth --disk-img=test-fresh.img
|
||
# Should print: 42
|
||
----
|
||
|
||
'''''
|
||
|
||
=== Troubleshooting
|
||
|
||
==== "`Picked 0 blocks`"
|
||
|
||
*Cause:* No blocks found with `+(-+` headers
|
||
|
||
*Fix:*
|
||
|
||
[arabic]
|
||
. Check disk actually has blocks with `+(-+` at start
|
||
. Try `+--loose+` if blocks have BOM or leading whitespace
|
||
. Remove `+--require-close+` if `+)+` is on next line
|
||
|
||
==== "`Refusing to write block N`"
|
||
|
||
*Cause:* Block number violates guard rails
|
||
|
||
*Fix:*
|
||
|
||
[arabic]
|
||
. Check `+--start+` and `+--end+` values
|
||
. Ensure init.4th block numbers are in allowed range
|
||
. Remove guards if intentional
|
||
|
||
==== "`Content exceeds FBS`"
|
||
|
||
*Cause:* Block text is longer than 1024 bytes
|
||
|
||
*Fix:*
|
||
|
||
[arabic]
|
||
. Split block into multiple blocks
|
||
. Use `+--clip+` to truncate (NOT RECOMMENDED for code)
|
||
. Increase FBS if using custom block size
|
||
|
||
==== "`Verify failed at block N`"
|
||
|
||
*Cause:* Read-back doesn’t match what was written
|
||
|
||
*Fix:*
|
||
|
||
[arabic]
|
||
. Check disk image file permissions
|
||
. Verify filesystem not full
|
||
. Check for disk errors (fsck, SMART)
|
||
. Try without `+--verify+` to isolate issue
|
||
|
||
'''''
|
||
|
||
=== Advanced Usage
|
||
|
||
==== Custom Block Size
|
||
|
||
*Scenario:* Disk uses 2048-byte blocks
|
||
|
||
[source,bash]
|
||
----
|
||
# Extract
|
||
./tools/extract-init --img=bigdisk.img --out=init-4.4th --fbs=2048
|
||
|
||
# Apply
|
||
./tools/apply-init --img=bigdisk.img --in=init-4.4th --fbs=2048
|
||
----
|
||
|
||
*Note:* FBS must match between extract/apply and must match what
|
||
StarForth VM expects.
|
||
|
||
==== Partial Range Extraction
|
||
|
||
*Scenario:* Only extract blocks 1024-2047
|
||
|
||
[source,bash]
|
||
----
|
||
./tools/extract-init \
|
||
--img=disk.img \
|
||
--out=partial.4th \
|
||
--start=1024 \
|
||
--end=2047
|
||
----
|
||
|
||
==== Limited Extraction
|
||
|
||
*Scenario:* Extract only first 5 marked blocks
|
||
|
||
[source,bash]
|
||
----
|
||
./tools/extract-init \
|
||
--img=disk.img \
|
||
--out=init-4.4th \
|
||
--max=5
|
||
----
|
||
|
||
'''''
|
||
|
||
=== See Also
|
||
|
||
* link:INIT_SYSTEM.md[INIT System Documentation] - How INIT word loads
|
||
init.4th at boot
|
||
* link:BLOCK_STORAGE_GUIDE.md[Block Storage Guide] - 3-layer block
|
||
architecture
|
||
* https://git-lfs.github.com/[Git LFS Documentation] - Large file
|
||
storage
|
||
|
||
'''''
|
||
|
||
=== Code References
|
||
|
||
==== extract_init.c
|
||
|
||
* `+tools/extract_init.c:66-76+` - Header detection logic
|
||
* `+tools/extract_init.c:78-83+` - Block text trimming
|
||
* `+tools/extract_init.c:85-91+` - Closing `+)+` check
|
||
* `+tools/extract_init.c:193-212+` - Main extraction loop
|
||
|
||
==== apply_init.c
|
||
|
||
* `+tools/apply_init.c:85-98+` - Block header parsing
|
||
* `+tools/apply_init.c:114-155+` - Block write with verification
|
||
* `+tools/apply_init.c:193-219+` - Section finalization (guards, verify)
|
||
* `+tools/apply_init.c:221-236+` - Main parser loop
|
||
|
||
==== Shell Scripts
|
||
|
||
* `+scripts/update-init.sh+` - Interactive extract wrapper
|
||
* `+scripts/apply-init.sh+` - Interactive apply wrapper
|
||
|
||
'''''
|
||
|
||
*End of INIT Tools Documentation*
|