22 KiB
22 KiB
Phase 1 Governance Validation Protocol
Date Created: November 3, 2025 Protocol Version: 1.0 Status: Ready for Execution Type: Comprehensive System Validation
Protocol Objective
Validate that the Phase 1 GitHub governance workflow system functions correctly end-to-end, with all document types properly routed, approval chains working, and backlog access control enforced.
This protocol tests the complete governance system to ensure it is production-ready before proceeding to Phase 2 vault infrastructure.
Scope
This protocol covers validation of:
- Template Recognition - All 16 document types
- Submission Workflows - Document intake and validation
- Approval Chains - QA/PM approvals trigger correctly
- Routing Decisions - PM decision gate works
- Backlog Entry - Approved documents enter backlog correctly
- Vault Routing - Approved documents route to in_basket
- Gatekeeper Enforcement - Unauthorized access blocked
- Security Logging - All events logged to SEC_LOG.adoc
- Label Management - Labels apply and track correctly
- Error Handling - System provides clear feedback on problems
Out of Scope:
- Vault organization and archival (Phase 2)
- Jenkins CI/CD integration (Phase 2)
- Digital signature collection (Phase 2)
Test Environment Setup
Prerequisites
- StarForth GitHub repository access
- StarForth-Governance repository access (read/write)
- Test user accounts (non-owner) for bypass testing
- GitHub Actions access logs visible
- Repository labels created (verify in GitHub settings)
- Workflows deployed and active (check Actions tab)
Cleanup from Previous Testing
- Archive or delete test issues #147-153
- Clear any test data from in_basket
- Verify SEC_LOG.adoc is clean
Documentation Ready
- BACKLOG_GATEKEEPER_PROCEDURES.md available
- GOVERNANCE_WORKFLOW_IMPLEMENTATION.md available
- Template instructions reviewed
Validation Test Cases
TEST SUITE 1: Template Recognition & Classification
Test 1.1: ECO Template Recognition
Test ID: VAL-1.1
Document Type: ECO (Engineering Change Order)
Precondition: No issues with type:eco label exist
Steps:
1. Create new GitHub issue using eco.yml template
2. Fill in all required fields:
- Related ECR reference
- Acceptance criteria (checklist)
- Implementation constraints
- FMEA decision
- Priority level
- Effort estimate
- Success criteria
3. Submit issue
Expected Result:
- Issue created successfully
- Labels automatically applied: type:eco, status:approved
- Issue appears in repository
- No validation errors
Success Criteria:
- ✓ Issue visible in GitHub
- ✓ type:eco label present
- ✓ status:approved label present
- ✓ Issue number recorded for audit trail
Test 1.2: CAPA Template Recognition
Test ID: VAL-1.2
Document Type: CAPA (Corrective/Preventive Action)
Precondition: No CAPA issues exist
Steps:
1. Create new GitHub issue using capa.yml template
2. Fill in all required fields:
- What is the Problem? (substantive description)
- When did it happen? (timing)
- Why did it happen? (root cause)
- What is the impact? (severity assessment)
- What fix do we recommend?
- Evidence/Attachments
- Regression potential
3. Submit issue
Expected Result:
- Issue created successfully
- Labels automatically applied: type:capa, status:submitted
- Workflow comment posted with validation feedback
- No placeholder text accepted
Success Criteria:
- ✓ Issue visible in GitHub
- ✓ type:capa label present
- ✓ status:submitted label present
- ✓ CAPA submission workflow comment present
Test 1.3: CER Template Recognition (Sequential Approval)
Test ID: VAL-1.3
Document Type: CER (Continuous Engineering Report)
Precondition: No CER issues exist
Steps:
1. Create new GitHub issue using cer.yml template
2. Fill in Section 1 only (Engineering Protocol):
- Research Objective
- Methodology & Approach
- Acceptance Criteria
3. Submit issue (leave Sections 2 & 3 empty)
Expected Result:
- Issue created successfully
- Labels: type:cer, status:submitted, cer-awaiting-protocol-approval
- Workflow comment shows current stage: "Protocol ⏳ → Results ⏳ → Report ⏳"
- System awaits protocol approval before allowing results
Success Criteria:
- ✓ Issue visible in GitHub
- ✓ type:cer label present
- ✓ cer-awaiting-protocol-approval label present
- ✓ Sequential approval structure detected
- ✓ Workflow comment shows progress tracking
Test 1.4: DHR Template Recognition (Reference Document)
Test ID: VAL-1.4
Document Type: DHR (Design History Record - Reference)
Precondition: No DHR issues exist
Steps:
1. Create new GitHub issue using dhr.yml template
2. Fill in required fields:
- Release version
- Design snapshot content
- Architecture overview
- Key components list
3. Submit issue
Expected Result:
- Issue created successfully
- Labels automatically applied: type:dhr, status:approved
- Auto-vault workflow should trigger (may need slight delay)
- Reference document treated as already approved
Success Criteria:
- ✓ Issue visible in GitHub
- ✓ type:dhr label present
- ✓ status:approved label present (auto-applied)
- ✓ Auto-vault comment posted (or ready to post)
Test 1.5: Additional Document Types
Test ID: VAL-1.5
Document Types: DWG, ENG, SEC, IR, VAL, DTA
For each document type:
Steps:
1. Create GitHub issue using corresponding template
2. Fill in required fields specific to type
3. Submit issue
Expected Result (for each):
- Issue created with correct type:* label
- status:submitted label applied
- Document type correctly classified
Success Criteria:
- ✓ All document types can be submitted
- ✓ Correct labels applied for each type
- ✓ No submission errors or rejections
TEST SUITE 2: Approval Workflows
Test 2.1: CAPA QA Approval Workflow
Test ID: VAL-2.1
Starting State: CAPA issue with status:submitted label
Role: QA Approver
Steps:
1. Using QA/PM account, post approval comment on CAPA issue:
"✅ APPROVED
Stakeholder: Quality Lead
Role: QA Manager
Date: YYYY-MM-DD
Notes: Issue well documented and reproducible"
2. Apply label: approved-by-qualm
Expected Result:
- Approval label applied successfully
- Workflow detects approval
- Next stage triggers (PM decision gate)
- Issue transitions from submitted → approved state
Success Criteria:
- ✓ Approval comment visible
- ✓ approved-by-qualm label present
- ✓ Workflow comment indicates next step: PM routing decision
- ✓ No validation errors
Test 2.2: PM Approval & Routing Decision
Test ID: VAL-2.2
Starting State: CAPA with approved-by-qualm label
Role: PM
Steps:
1. Review CAPA issue
2. Apply one of:
- route:backlog (if this creates implementation work)
- route:vault (if this is reference/documentation)
Expected Result (for route:backlog):
- Label applied successfully
- Gatekeeper accepts it (approved item → allowed)
- Issue moved to development backlog
- Item available for developer pickup
Expected Result (for route:vault):
- Label applied successfully
- Vault routing workflow triggers
- Document prepared for in_basket
- Metadata file created
- Confirmation comment posted
Success Criteria:
- ✓ Routing label applied
- ✓ No gatekeeper blocking
- ✓ Appropriate workflow triggered
- ✓ Document status updated
Test 2.3: CER Sequential Approval
Test ID: VAL-2.3
Starting State: CER with Section 1 only
Role: PM/Approver
Steps (Phase 1 - Protocol):
1. Review Engineering Protocol section
2. Post approval comment
3. Apply label: cer-protocol-approved
Expected Result:
- Workflow detects protocol approval
- Progress updates to: "Protocol ✅ → Results ⏳ → Report ⏳"
- System waits for Section 2 (Experimental Data)
- Blocks report submission until results approved
Steps (Phase 2 - Results):
1. Add Section 2: Experimental Data & Results
2. Post approval comment
3. Apply label: cer-results-approved
Expected Result:
- Workflow detects results approval
- Progress updates to: "Protocol ✅ → Results ✅ → Report ⏳"
- System waits for Section 3 (Final Report)
Steps (Phase 3 - Report):
1. Add Section 3: Final Engineering Report
2. Post approval comment
3. Apply label: cer-report-approved
Expected Result:
- Workflow detects report approval
- All stages complete: "Protocol ✅ → Results ✅ → Report ✅"
- Status changes to cer-complete, approved-by-projm
- Issue ready for routing decision
Success Criteria:
- ✓ Each stage requires previous approval
- ✓ Progress feedback accurate
- ✓ System blocks advancement without approval
- ✓ Final state triggers routing workflow
TEST SUITE 3: Routing & Backlog Entry
Test 3.1: Document Routes to Development Backlog
Test ID: VAL-3.1
Starting State: Approved CAPA with route:backlog label applied
Expected Destination: Development backlog
Steps:
1. Apply route:backlog label to approved CAPA/CER/DWG issue
2. Observe workflow execution
3. Verify document appears in backlog
Expected Result:
- Label applied successfully
- Workflow processes routing
- Document available for developer assignment
- Issue remains open for tracking
- GitHub issue linked from backlog item
Success Criteria:
- ✓ route:backlog label present
- ✓ Document visible in backlog context
- ✓ Developer can pick up work
- ✓ Audit trail shows routing decision
Test 3.2: Document Routes to Governance Vault
Test ID: VAL-3.2
Starting State: Approved document with route:vault label
Expected Destination: StarForth-Governance/in_basket/
Steps:
1. Apply route:vault label to approved document
2. Observe vault-routing workflow execution
3. Check in_basket directory
4. Verify metadata file created
Expected Result:
- route:vault label applied
- Vault routing workflow triggers
- Document metadata file created in in_basket
- Confirmation comment posted to GitHub issue
- status:in-vault label applied to issue
Success Criteria:
- ✓ route:vault label present
- ✓ in_basket contains new metadata
- ✓ Confirmation comment visible
- ✓ status:in-vault label applied
- ✓ GitHub issue linked in metadata
Test 3.3: Reference Documents Auto-Vault
Test ID: VAL-3.3
Document Types: DHR, DMR, ART, MIN, REL, RMP
Expected Behavior: Auto-approval and auto-vault without review
Steps:
1. Create DHR/DMR/ART/MIN/REL/RMP issue
2. Observe auto-vault workflow
3. Verify automatic routing
Expected Result:
- Reference document detected
- Auto-approved with status:approved label
- Auto-routed with route:vault label
- Confirmation comment: "No review required for reference material"
- Document ready for vault processing
Success Criteria:
- ✓ status:approved applied automatically
- ✓ route:vault applied automatically
- ✓ governance-routed label applied
- ✓ Confirmation comment present
- ✓ No manual approval needed
TEST SUITE 4: Gatekeeper Enforcement
Test 4.1: Unauthorized Backlog Access Attempt
Test ID: VAL-4.1
Test User: Non-owner account (not rajames440)
Starting State: Regular issue without approval labels
Expected Behavior: Bypass detected and blocked
Steps:
1. Create test issue (no template, no type label)
2. Apply route:backlog label (unauthorized attempt)
3. Observe gatekeeper response
Expected Result:
- Gatekeeper workflow detects bypass
- Security warning comment posted
- IR (Incident Report) issue created
- PM assigned for review
- Event logged to SEC_LOG.adoc
- No backlog entry permitted
Success Criteria:
- ✓ Security warning comment present
- ✓ IR issue created with severity:major
- ✓ PM notified (assigned)
- ✓ SEC_LOG.adoc updated
- ✓ Document NOT in backlog
Test 4.2: Authorized Access - Approved Document
Test ID: VAL-4.2
Test User: Non-owner account
Starting State: Approved CAPA issue
Expected Behavior: Access permitted (approved path)
Steps:
1. CAPA issue with approval-by-qualm label exists
2. Apply route:backlog label (approved path)
3. Observe gatekeeper response
Expected Result:
- Gatekeeper validates approval evidence
- route:backlog label accepted
- No security warning posted
- No IR created
- Document enters backlog normally
- No SEC_LOG entry
Success Criteria:
- ✓ No security warning posted
- ✓ No IR created
- ✓ route:backlog label accepted
- ✓ Document in backlog
- ✓ SEC_LOG clean (no false positive)
Test 4.3: Owner Direct Backlog Access
Test ID: VAL-4.3
Test User: Owner account (rajames440)
Starting State: Any issue
Expected Behavior: Owner has direct backlog access
Steps:
1. Owner account applies route:backlog label to any issue
2. Observe gatekeeper response
Expected Result:
- Gatekeeper recognizes owner
- route:backlog label accepted
- No security warning posted
- No IR created
- Document enters backlog
Success Criteria:
- ✓ No security warning posted
- ✓ route:backlog accepted
- ✓ Document enters backlog
- ✓ Owner discretionary authority respected
Test 4.4: ECO Always Routes to Backlog
Test ID: VAL-4.4
Document Type: ECO
Expected Behavior: ECO always permitted to backlog without approval
Steps:
1. Create ECO issue
2. System should recognize type:eco
3. route:backlog should be automatically applied
4. No approval labels needed
Expected Result:
- ECO created with type:eco label
- route:backlog applied automatically or accepted without approval
- No gatekeeper blocking
- Document enters backlog
Success Criteria:
- ✓ type:eco recognized
- ✓ Backlog routing permitted without approval
- ✓ No security blocking
- ✓ ECO in backlog as expected
TEST SUITE 5: Security Audit Logging
Test 5.1: SEC_LOG.adoc Records All Events
Test ID: VAL-5.1
Expected Behavior: All security events logged immutably
Steps:
1. Execute tests 4.1-4.4 (gatekeeper tests)
2. Review SEC_LOG.adoc in governance repo
3. Verify all bypass attempts recorded
Expected Result:
- Unauthorized attempt from Test 4.1 logged
- Timestamp recorded (UTC)
- Actor username recorded
- Event type: "Unauthorized Backlog Access"
- Reference to created IR
- Severity: MAJOR
Success Criteria:
- ✓ SEC_LOG.adoc contains entry for each bypass
- ✓ Format: timestamp | actor | issue | severity | action
- ✓ All unauthorized attempts logged
- ✓ Authorized attempts not logged (no false positives)
- ✓ SEC_LOG is git-versioned (immutable)
Test 5.2: Incident Report Creation
Test ID: VAL-5.2
Starting State: Gatekeeper detected bypass (from Test 4.1)
Expected Behavior: IR issue created with full details
Steps:
1. Find created IR issue
2. Review IR content
3. Verify completeness
Expected Result:
- IR issue created with title: "IR: Unauthorized Backlog Access"
- Body contains:
- Report Type: Security / Governance Violation
- Severity: MAJOR
- Unauthorized User: @testuser
- Original Issue: #NNN
- Violation Method: description
- Expected Process: governance procedure
- Labels: type:ir, status:submitted, severity:major, security-violation
- Linked to original issue
Success Criteria:
- ✓ IR created automatically
- ✓ All required fields present
- ✓ severity:major applied
- ✓ PM can review and take action
- ✓ Full audit trail maintained
TEST SUITE 6: Label Management
Test 6.1: Type Label Exclusivity
Test ID: VAL-6.1
Expected Behavior: Each issue has exactly ONE type:* label
Steps:
1. Attempt to create issue with multiple type labels
(e.g., apply both type:capa and type:eco)
2. Verify enforcement
Expected Result:
- System detects multiple type labels
- Warning comment posted
- User instructed to use only one type
- Validation workflow enforces exclusivity
Success Criteria:
- ✓ Multiple type labels detected
- ✓ Warning comment posted
- ✓ Guidance provided to correct
Test 6.2: Status Label Progression
Test ID: VAL-6.2
Expected Behavior: Status labels track workflow progression
Steps:
1. Create issue (status:submitted)
2. Approve issue (add approval label)
3. Route issue (add route:* label)
4. Verify status progression
Expected Result:
- status:submitted applied on creation
- Approval labels track review status
- route:backlog or route:vault applied
- status:in-vault applied if routed to vault
- Status progression visible in label history
Success Criteria:
- ✓ Each stage has appropriate status label
- ✓ Labels document progression
- ✓ Audit trail shows status changes
TEST SUITE 7: Error Handling & User Feedback
Test 7.1: Incomplete Template Detection
Test ID: VAL-7.1
Expected Behavior: System detects incomplete submissions
Steps:
1. Create CAPA issue with minimal/placeholder text
2. Verify validation feedback
Expected Result:
- Submission workflow detects incomplete template
- Comment posted: "Template Incomplete"
- Guidance provided on required fields
- status:needs-info label applied
- User can resubmit after fixing
Success Criteria:
- ✓ Validation feedback clear and helpful
- ✓ Specific fields identified as missing
- ✓ Template example provided
- ✓ User can fix and retry
Test 7.2: Workflow Feedback Comments
Test ID: VAL-7.2
Expected Behavior: Workflows provide clear status feedback
Steps:
1. Create various document types
2. Apply labels and move through workflow
3. Review workflow comment feedback
Expected Result:
- Each workflow stage posts status comment
- Comments include:
- Current status
- What's needed next
- Who needs to act
- Timeline expectations
- Comments are clear and actionable
Success Criteria:
- ✓ Workflow comments visible
- ✓ Feedback is clear and specific
- ✓ Next steps explained
- ✓ Users understand status
Execution Plan
Phase A: Basic Functionality (Day 1)
- Test Suites 1-2 (Templates and Approvals)
- Verify all document types can be submitted
- Verify approval chains work
- Estimated Time: 2-3 hours
Phase B: Routing & Backlog (Day 2)
- Test Suite 3 (Routing and Backlog Entry)
- Verify documents route correctly
- Verify vault routing works
- Verify in_basket receives documents
- Estimated Time: 2-3 hours
Phase C: Security & Enforcement (Day 3)
- Test Suites 4-5 (Gatekeeper and Audit Logging)
- Verify bypass attempts blocked
- Verify authorized access allowed
- Verify SEC_LOG.adoc updated
- Estimated Time: 2-3 hours
Phase D: Labels & Feedback (Day 4)
- Test Suites 6-7 (Label Management and Feedback)
- Verify label enforcement
- Verify error handling
- Verify user feedback quality
- Estimated Time: 1-2 hours
Total Estimated Time: 7-11 hours across 4 days
Pass/Fail Criteria
PASS Conditions (All Must Be Met)
- ✅ All 16 document types can be submitted
- ✅ All approval workflows trigger correctly
- ✅ Documents route correctly to backlog/vault
- ✅ Gatekeeper blocks unauthorized access
- ✅ Authorized access is permitted
- ✅ All events logged to SEC_LOG.adoc
- ✅ Users receive clear feedback
- ✅ No critical errors or crashes
- ✅ Audit trail is complete
FAIL Conditions (Any One Fails)
- ❌ Document type not recognized
- ❌ Approval workflow doesn't trigger
- ❌ Document doesn't route correctly
- ❌ Gatekeeper allows unauthorized access
- ❌ Gatekeeper blocks authorized access
- ❌ Events not logged to SEC_LOG
- ❌ User feedback unclear or missing
- ❌ System errors or crashes
- ❌ Audit trail incomplete or corrupted
Success Metrics
| Metric | Target | How Measured |
|---|---|---|
| Template Recognition | 16/16 (100%) | All document types submit successfully |
| Approval Accuracy | 100% | All approvals trigger correct workflows |
| Routing Accuracy | 100% | Documents reach correct destination |
| Gatekeeper Effectiveness | 100% | 0 unauthorized bypasses, all authorizations accepted |
| Audit Logging | 100% | All events logged, no gaps |
| User Feedback | Clear & Actionable | Users understand status without questions |
| System Stability | No crashes | All tests complete without errors |
Reporting & Documentation
During Execution
- Record test results in execution log
- Screenshot successful workflows
- Document any issues encountered
- Note timing of each test
- Capture workflow comments
After Execution
- Complete test report (pass/fail for each test)
- Document any defects found
- Note any workflow improvements needed
- Collect metrics and statistics
- Get sign-off on results
Deliverables
- Test Execution Report
- Defect Report (if any)
- Metrics Summary
- Recommendations for Phase 2
- Sign-off from validator(s)
Rollback Plan
If critical issues are found during validation:
- Stop - Pause Phase 2 preparations
- Isolate - Identify failing workflows
- Analyze - Determine root cause
- Fix - Implement corrections
- Re-test - Validate fix
- Resume - Continue once critical issues resolved
Appendix: Test Data Templates
Test User Accounts Needed
- QA Approver account (for approval tests)
- PM account (for routing decisions)
- Test Developer account (for bypass attempts)
- Owner account (rajames440)
Test Documents to Create
- Sample CAPA issue
- Sample ECR issue
- Sample CER issue
- Sample DWG issue
- Sample SEC issue
- Sample IR issue
Test Checklists
- GitHub Actions logs accessible
- SEC_LOG.adoc writable by workflow
- in_basket directory accessible
- All workflows deployed (no syntax errors)
- All labels created in GitHub
- Test issues can be created
Sign-Off
Validation Protocol Prepared By: Claude Code Date: November 3, 2025 Version: 1.0 Status: Ready for Execution
To Execute This Protocol:
- Schedule 4-day validation window
- Prepare test environment (see Prerequisites)
- Follow test cases in order
- Document all results
- Report findings
Expected Outcome: Phase 1 system validated and confirmed production-ready for Phase 2 vault infrastructure setup.
This protocol is comprehensive and self-contained. It can be executed by anyone with GitHub/governance repo access and can be repeated as needed for regression testing.