LithosAnanake/docs/working/archive/quality/validation/phase-1-protocol.md

<!-- Moved from docs/04-quality/validation/phase-1-protocol.md to docs/working/archive/quality/validation/phase-1-protocol.md on 2026-06-16 (docs reorg Phase 2) -->
# Phase 1 Governance Validation Protocol

**Date Created:** November 3, 2025
**Protocol Version:** 1.0
**Status:** Ready for Execution
**Type:** Comprehensive System Validation

---

## Protocol Objective

Validate that the Phase 1 GitHub governance workflow system functions correctly end-to-end, with all document types properly routed, approval chains working, and backlog access control enforced.

This protocol tests the complete governance system to ensure it is production-ready before proceeding to Phase 2 vault infrastructure.

---

## Scope

This protocol covers validation of:

1. **Template Recognition** - All 16 document types
2. **Submission Workflows** - Document intake and validation
3. **Approval Chains** - QA/PM approvals trigger correctly
4. **Routing Decisions** - PM decision gate works
5. **Backlog Entry** - Approved documents enter backlog correctly
6. **Vault Routing** - Approved documents route to in_basket
7. **Gatekeeper Enforcement** - Unauthorized access blocked
8. **Security Logging** - All events logged to SEC_LOG.adoc
9. **Label Management** - Labels apply and track correctly
10. **Error Handling** - System provides clear feedback on problems

**Out of Scope:**
- Vault organization and archival (Phase 2)
- Jenkins CI/CD integration (Phase 2)
- Digital signature collection (Phase 2)

---

## Test Environment Setup

### Prerequisites
- [ ] StarForth GitHub repository access
- [ ] StarForth-Governance repository access (read/write)
- [ ] Test user accounts (non-owner) for bypass testing
- [ ] GitHub Actions access logs visible
- [ ] Repository labels created (verify in GitHub settings)
- [ ] Workflows deployed and active (check Actions tab)

### Cleanup from Previous Testing
- [ ] Archive or delete test issues #147-153
- [ ] Clear any test data from in_basket
- [ ] Verify SEC_LOG.adoc is clean

### Documentation Ready
- [ ] BACKLOG_GATEKEEPER_PROCEDURES.md available
- [ ] GOVERNANCE_WORKFLOW_IMPLEMENTATION.md available
- [ ] Template instructions reviewed

---

## Validation Test Cases

### TEST SUITE 1: Template Recognition & Classification

#### Test 1.1: ECO Template Recognition
```
Test ID: VAL-1.1
Document Type: ECO (Engineering Change Order)
Precondition: No issues with type:eco label exist

Steps:
1. Create new GitHub issue using eco.yml template
2. Fill in all required fields:
   - Related ECR reference
   - Acceptance criteria (checklist)
   - Implementation constraints
   - FMEA decision
   - Priority level
   - Effort estimate
   - Success criteria
3. Submit issue

Expected Result:
- Issue created successfully
- Labels automatically applied: type:eco, status:approved
- Issue appears in repository
- No validation errors

Success Criteria:
- ✓ Issue visible in GitHub
- ✓ type:eco label present
- ✓ status:approved label present
- ✓ Issue number recorded for audit trail
```

#### Test 1.2: CAPA Template Recognition
```
Test ID: VAL-1.2
Document Type: CAPA (Corrective/Preventive Action)
Precondition: No CAPA issues exist

Steps:
1. Create new GitHub issue using capa.yml template
2. Fill in all required fields:
   - What is the Problem? (substantive description)
   - When did it happen? (timing)
   - Why did it happen? (root cause)
   - What is the impact? (severity assessment)
   - What fix do we recommend?
   - Evidence/Attachments
   - Regression potential
3. Submit issue

Expected Result:
- Issue created successfully
- Labels automatically applied: type:capa, status:submitted
- Workflow comment posted with validation feedback
- No placeholder text accepted

Success Criteria:
- ✓ Issue visible in GitHub
- ✓ type:capa label present
- ✓ status:submitted label present
- ✓ CAPA submission workflow comment present
```

#### Test 1.3: CER Template Recognition (Sequential Approval)
```
Test ID: VAL-1.3
Document Type: CER (Continuous Engineering Report)
Precondition: No CER issues exist

Steps:
1. Create new GitHub issue using cer.yml template
2. Fill in Section 1 only (Engineering Protocol):
   - Research Objective
   - Methodology & Approach
   - Acceptance Criteria
3. Submit issue (leave Sections 2 & 3 empty)

Expected Result:
- Issue created successfully
- Labels: type:cer, status:submitted, cer-awaiting-protocol-approval
- Workflow comment shows current stage: "Protocol ⏳ → Results ⏳ → Report ⏳"
- System awaits protocol approval before allowing results

Success Criteria:
- ✓ Issue visible in GitHub
- ✓ type:cer label present
- ✓ cer-awaiting-protocol-approval label present
- ✓ Sequential approval structure detected
- ✓ Workflow comment shows progress tracking
```

#### Test 1.4: DHR Template Recognition (Reference Document)
```
Test ID: VAL-1.4
Document Type: DHR (Design History Record - Reference)
Precondition: No DHR issues exist

Steps:
1. Create new GitHub issue using dhr.yml template
2. Fill in required fields:
   - Release version
   - Design snapshot content
   - Architecture overview
   - Key components list
3. Submit issue

Expected Result:
- Issue created successfully
- Labels automatically applied: type:dhr, status:approved
- Auto-vault workflow should trigger (may need slight delay)
- Reference document treated as already approved

Success Criteria:
- ✓ Issue visible in GitHub
- ✓ type:dhr label present
- ✓ status:approved label present (auto-applied)
- ✓ Auto-vault comment posted (or ready to post)
```

#### Test 1.5: Additional Document Types
```
Test ID: VAL-1.5
Document Types: DWG, ENG, SEC, IR, VAL, DTA

For each document type:
Steps:
1. Create GitHub issue using corresponding template
2. Fill in required fields specific to type
3. Submit issue

Expected Result (for each):
- Issue created with correct type:* label
- status:submitted label applied
- Document type correctly classified

Success Criteria:
- ✓ All document types can be submitted
- ✓ Correct labels applied for each type
- ✓ No submission errors or rejections
```

---

### TEST SUITE 2: Approval Workflows

#### Test 2.1: CAPA QA Approval Workflow
```
Test ID: VAL-2.1
Starting State: CAPA issue with status:submitted label
Role: QA Approver

Steps:
1. Using QA/PM account, post approval comment on CAPA issue:
   "✅ APPROVED
   Stakeholder: Quality Lead
   Role: QA Manager
   Date: YYYY-MM-DD
   Notes: Issue well documented and reproducible"

2. Apply label: approved-by-qualm

Expected Result:
- Approval label applied successfully
- Workflow detects approval
- Next stage triggers (PM decision gate)
- Issue transitions from submitted → approved state

Success Criteria:
- ✓ Approval comment visible
- ✓ approved-by-qualm label present
- ✓ Workflow comment indicates next step: PM routing decision
- ✓ No validation errors
```

#### Test 2.2: PM Approval & Routing Decision
```
Test ID: VAL-2.2
Starting State: CAPA with approved-by-qualm label
Role: PM

Steps:
1. Review CAPA issue
2. Apply one of:
   - route:backlog (if this creates implementation work)
   - route:vault (if this is reference/documentation)

Expected Result (for route:backlog):
- Label applied successfully
- Gatekeeper accepts it (approved item → allowed)
- Issue moved to development backlog
- Item available for developer pickup

Expected Result (for route:vault):
- Label applied successfully
- Vault routing workflow triggers
- Document prepared for in_basket
- Metadata file created
- Confirmation comment posted

Success Criteria:
- ✓ Routing label applied
- ✓ No gatekeeper blocking
- ✓ Appropriate workflow triggered
- ✓ Document status updated
```

#### Test 2.3: CER Sequential Approval
```
Test ID: VAL-2.3
Starting State: CER with Section 1 only
Role: PM/Approver

Steps (Phase 1 - Protocol):
1. Review Engineering Protocol section
2. Post approval comment
3. Apply label: cer-protocol-approved

Expected Result:
- Workflow detects protocol approval
- Progress updates to: "Protocol ✅ → Results ⏳ → Report ⏳"
- System waits for Section 2 (Experimental Data)
- Blocks report submission until results approved

Steps (Phase 2 - Results):
1. Add Section 2: Experimental Data & Results
2. Post approval comment
3. Apply label: cer-results-approved

Expected Result:
- Workflow detects results approval
- Progress updates to: "Protocol ✅ → Results ✅ → Report ⏳"
- System waits for Section 3 (Final Report)

Steps (Phase 3 - Report):
1. Add Section 3: Final Engineering Report
2. Post approval comment
3. Apply label: cer-report-approved

Expected Result:
- Workflow detects report approval
- All stages complete: "Protocol ✅ → Results ✅ → Report ✅"
- Status changes to cer-complete, approved-by-projm
- Issue ready for routing decision

Success Criteria:
- ✓ Each stage requires previous approval
- ✓ Progress feedback accurate
- ✓ System blocks advancement without approval
- ✓ Final state triggers routing workflow
```

---

### TEST SUITE 3: Routing & Backlog Entry

#### Test 3.1: Document Routes to Development Backlog
```
Test ID: VAL-3.1
Starting State: Approved CAPA with route:backlog label applied
Expected Destination: Development backlog

Steps:
1. Apply route:backlog label to approved CAPA/CER/DWG issue
2. Observe workflow execution
3. Verify document appears in backlog

Expected Result:
- Label applied successfully
- Workflow processes routing
- Document available for developer assignment
- Issue remains open for tracking
- GitHub issue linked from backlog item

Success Criteria:
- ✓ route:backlog label present
- ✓ Document visible in backlog context
- ✓ Developer can pick up work
- ✓ Audit trail shows routing decision
```

#### Test 3.2: Document Routes to Governance Vault
```
Test ID: VAL-3.2
Starting State: Approved document with route:vault label
Expected Destination: StarForth-Governance/in_basket/

Steps:
1. Apply route:vault label to approved document
2. Observe vault-routing workflow execution
3. Check in_basket directory
4. Verify metadata file created

Expected Result:
- route:vault label applied
- Vault routing workflow triggers
- Document metadata file created in in_basket
- Confirmation comment posted to GitHub issue
- status:in-vault label applied to issue

Success Criteria:
- ✓ route:vault label present
- ✓ in_basket contains new metadata
- ✓ Confirmation comment visible
- ✓ status:in-vault label applied
- ✓ GitHub issue linked in metadata
```

#### Test 3.3: Reference Documents Auto-Vault
```
Test ID: VAL-3.3
Document Types: DHR, DMR, ART, MIN, REL, RMP
Expected Behavior: Auto-approval and auto-vault without review

Steps:
1. Create DHR/DMR/ART/MIN/REL/RMP issue
2. Observe auto-vault workflow
3. Verify automatic routing

Expected Result:
- Reference document detected
- Auto-approved with status:approved label
- Auto-routed with route:vault label
- Confirmation comment: "No review required for reference material"
- Document ready for vault processing

Success Criteria:
- ✓ status:approved applied automatically
- ✓ route:vault applied automatically
- ✓ governance-routed label applied
- ✓ Confirmation comment present
- ✓ No manual approval needed
```

---

### TEST SUITE 4: Gatekeeper Enforcement

#### Test 4.1: Unauthorized Backlog Access Attempt
```
Test ID: VAL-4.1
Test User: Non-owner account (not rajames440)
Starting State: Regular issue without approval labels
Expected Behavior: Bypass detected and blocked

Steps:
1. Create test issue (no template, no type label)
2. Apply route:backlog label (unauthorized attempt)
3. Observe gatekeeper response

Expected Result:
- Gatekeeper workflow detects bypass
- Security warning comment posted
- IR (Incident Report) issue created
- PM assigned for review
- Event logged to SEC_LOG.adoc
- No backlog entry permitted

Success Criteria:
- ✓ Security warning comment present
- ✓ IR issue created with severity:major
- ✓ PM notified (assigned)
- ✓ SEC_LOG.adoc updated
- ✓ Document NOT in backlog
```

#### Test 4.2: Authorized Access - Approved Document
```
Test ID: VAL-4.2
Test User: Non-owner account
Starting State: Approved CAPA issue
Expected Behavior: Access permitted (approved path)

Steps:
1. CAPA issue with approval-by-qualm label exists
2. Apply route:backlog label (approved path)
3. Observe gatekeeper response

Expected Result:
- Gatekeeper validates approval evidence
- route:backlog label accepted
- No security warning posted
- No IR created
- Document enters backlog normally
- No SEC_LOG entry

Success Criteria:
- ✓ No security warning posted
- ✓ No IR created
- ✓ route:backlog label accepted
- ✓ Document in backlog
- ✓ SEC_LOG clean (no false positive)
```

#### Test 4.3: Owner Direct Backlog Access
```
Test ID: VAL-4.3
Test User: Owner account (rajames440)
Starting State: Any issue
Expected Behavior: Owner has direct backlog access

Steps:
1. Owner account applies route:backlog label to any issue
2. Observe gatekeeper response

Expected Result:
- Gatekeeper recognizes owner
- route:backlog label accepted
- No security warning posted
- No IR created
- Document enters backlog

Success Criteria:
- ✓ No security warning posted
- ✓ route:backlog accepted
- ✓ Document enters backlog
- ✓ Owner discretionary authority respected
```

#### Test 4.4: ECO Always Routes to Backlog
```
Test ID: VAL-4.4
Document Type: ECO
Expected Behavior: ECO always permitted to backlog without approval

Steps:
1. Create ECO issue
2. System should recognize type:eco
3. route:backlog should be automatically applied
4. No approval labels needed

Expected Result:
- ECO created with type:eco label
- route:backlog applied automatically or accepted without approval
- No gatekeeper blocking
- Document enters backlog

Success Criteria:
- ✓ type:eco recognized
- ✓ Backlog routing permitted without approval
- ✓ No security blocking
- ✓ ECO in backlog as expected
```

---

### TEST SUITE 5: Security Audit Logging

#### Test 5.1: SEC_LOG.adoc Records All Events
```
Test ID: VAL-5.1
Expected Behavior: All security events logged immutably

Steps:
1. Execute tests 4.1-4.4 (gatekeeper tests)
2. Review SEC_LOG.adoc in governance repo
3. Verify all bypass attempts recorded

Expected Result:
- Unauthorized attempt from Test 4.1 logged
- Timestamp recorded (UTC)
- Actor username recorded
- Event type: "Unauthorized Backlog Access"
- Reference to created IR
- Severity: MAJOR

Success Criteria:
- ✓ SEC_LOG.adoc contains entry for each bypass
- ✓ Format: timestamp | actor | issue | severity | action
- ✓ All unauthorized attempts logged
- ✓ Authorized attempts not logged (no false positives)
- ✓ SEC_LOG is git-versioned (immutable)
```

#### Test 5.2: Incident Report Creation
```
Test ID: VAL-5.2
Starting State: Gatekeeper detected bypass (from Test 4.1)
Expected Behavior: IR issue created with full details

Steps:
1. Find created IR issue
2. Review IR content
3. Verify completeness

Expected Result:
- IR issue created with title: "IR: Unauthorized Backlog Access"
- Body contains:
  - Report Type: Security / Governance Violation
  - Severity: MAJOR
  - Unauthorized User: @testuser
  - Original Issue: #NNN
  - Violation Method: description
  - Expected Process: governance procedure
- Labels: type:ir, status:submitted, severity:major, security-violation
- Linked to original issue

Success Criteria:
- ✓ IR created automatically
- ✓ All required fields present
- ✓ severity:major applied
- ✓ PM can review and take action
- ✓ Full audit trail maintained
```

---

### TEST SUITE 6: Label Management

#### Test 6.1: Type Label Exclusivity
```
Test ID: VAL-6.1
Expected Behavior: Each issue has exactly ONE type:* label

Steps:
1. Attempt to create issue with multiple type labels
   (e.g., apply both type:capa and type:eco)
2. Verify enforcement

Expected Result:
- System detects multiple type labels
- Warning comment posted
- User instructed to use only one type
- Validation workflow enforces exclusivity

Success Criteria:
- ✓ Multiple type labels detected
- ✓ Warning comment posted
- ✓ Guidance provided to correct
```

#### Test 6.2: Status Label Progression
```
Test ID: VAL-6.2
Expected Behavior: Status labels track workflow progression

Steps:
1. Create issue (status:submitted)
2. Approve issue (add approval label)
3. Route issue (add route:* label)
4. Verify status progression

Expected Result:
- status:submitted applied on creation
- Approval labels track review status
- route:backlog or route:vault applied
- status:in-vault applied if routed to vault
- Status progression visible in label history

Success Criteria:
- ✓ Each stage has appropriate status label
- ✓ Labels document progression
- ✓ Audit trail shows status changes
```

---

### TEST SUITE 7: Error Handling & User Feedback

#### Test 7.1: Incomplete Template Detection
```
Test ID: VAL-7.1
Expected Behavior: System detects incomplete submissions

Steps:
1. Create CAPA issue with minimal/placeholder text
2. Verify validation feedback

Expected Result:
- Submission workflow detects incomplete template
- Comment posted: "Template Incomplete"
- Guidance provided on required fields
- status:needs-info label applied
- User can resubmit after fixing

Success Criteria:
- ✓ Validation feedback clear and helpful
- ✓ Specific fields identified as missing
- ✓ Template example provided
- ✓ User can fix and retry
```

#### Test 7.2: Workflow Feedback Comments
```
Test ID: VAL-7.2
Expected Behavior: Workflows provide clear status feedback

Steps:
1. Create various document types
2. Apply labels and move through workflow
3. Review workflow comment feedback

Expected Result:
- Each workflow stage posts status comment
- Comments include:
  - Current status
  - What's needed next
  - Who needs to act
  - Timeline expectations
- Comments are clear and actionable

Success Criteria:
- ✓ Workflow comments visible
- ✓ Feedback is clear and specific
- ✓ Next steps explained
- ✓ Users understand status
```

---

## Execution Plan

### Phase A: Basic Functionality (Day 1)
- [ ] Test Suites 1-2 (Templates and Approvals)
- [ ] Verify all document types can be submitted
- [ ] Verify approval chains work
- Estimated Time: 2-3 hours

### Phase B: Routing & Backlog (Day 2)
- [ ] Test Suite 3 (Routing and Backlog Entry)
- [ ] Verify documents route correctly
- [ ] Verify vault routing works
- [ ] Verify in_basket receives documents
- Estimated Time: 2-3 hours

### Phase C: Security & Enforcement (Day 3)
- [ ] Test Suites 4-5 (Gatekeeper and Audit Logging)
- [ ] Verify bypass attempts blocked
- [ ] Verify authorized access allowed
- [ ] Verify SEC_LOG.adoc updated
- Estimated Time: 2-3 hours

### Phase D: Labels & Feedback (Day 4)
- [ ] Test Suites 6-7 (Label Management and Feedback)
- [ ] Verify label enforcement
- [ ] Verify error handling
- [ ] Verify user feedback quality
- Estimated Time: 1-2 hours

**Total Estimated Time:** 7-11 hours across 4 days

---

## Pass/Fail Criteria

### PASS Conditions (All Must Be Met)
- ✅ All 16 document types can be submitted
- ✅ All approval workflows trigger correctly
- ✅ Documents route correctly to backlog/vault
- ✅ Gatekeeper blocks unauthorized access
- ✅ Authorized access is permitted
- ✅ All events logged to SEC_LOG.adoc
- ✅ Users receive clear feedback
- ✅ No critical errors or crashes
- ✅ Audit trail is complete

### FAIL Conditions (Any One Fails)
- ❌ Document type not recognized
- ❌ Approval workflow doesn't trigger
- ❌ Document doesn't route correctly
- ❌ Gatekeeper allows unauthorized access
- ❌ Gatekeeper blocks authorized access
- ❌ Events not logged to SEC_LOG
- ❌ User feedback unclear or missing
- ❌ System errors or crashes
- ❌ Audit trail incomplete or corrupted

---

## Success Metrics

| Metric | Target | How Measured |
|--------|--------|--------------|
| Template Recognition | 16/16 (100%) | All document types submit successfully |
| Approval Accuracy | 100% | All approvals trigger correct workflows |
| Routing Accuracy | 100% | Documents reach correct destination |
| Gatekeeper Effectiveness | 100% | 0 unauthorized bypasses, all authorizations accepted |
| Audit Logging | 100% | All events logged, no gaps |
| User Feedback | Clear & Actionable | Users understand status without questions |
| System Stability | No crashes | All tests complete without errors |

---

## Reporting & Documentation

### During Execution
- [ ] Record test results in execution log
- [ ] Screenshot successful workflows
- [ ] Document any issues encountered
- [ ] Note timing of each test
- [ ] Capture workflow comments

### After Execution
- [ ] Complete test report (pass/fail for each test)
- [ ] Document any defects found
- [ ] Note any workflow improvements needed
- [ ] Collect metrics and statistics
- [ ] Get sign-off on results

### Deliverables
1. Test Execution Report
2. Defect Report (if any)
3. Metrics Summary
4. Recommendations for Phase 2
5. Sign-off from validator(s)

---

## Rollback Plan

If critical issues are found during validation:

1. **Stop** - Pause Phase 2 preparations
2. **Isolate** - Identify failing workflows
3. **Analyze** - Determine root cause
4. **Fix** - Implement corrections
5. **Re-test** - Validate fix
6. **Resume** - Continue once critical issues resolved

---

## Appendix: Test Data Templates

### Test User Accounts Needed
- [ ] QA Approver account (for approval tests)
- [ ] PM account (for routing decisions)
- [ ] Test Developer account (for bypass attempts)
- [ ] Owner account (rajames440)

### Test Documents to Create
- [ ] Sample CAPA issue
- [ ] Sample ECR issue
- [ ] Sample CER issue
- [ ] Sample DWG issue
- [ ] Sample SEC issue
- [ ] Sample IR issue

### Test Checklists
- [ ] GitHub Actions logs accessible
- [ ] SEC_LOG.adoc writable by workflow
- [ ] in_basket directory accessible
- [ ] All workflows deployed (no syntax errors)
- [ ] All labels created in GitHub
- [ ] Test issues can be created

---

## Sign-Off

**Validation Protocol Prepared By:** Claude Code
**Date:** November 3, 2025
**Version:** 1.0
**Status:** Ready for Execution

**To Execute This Protocol:**
1. Schedule 4-day validation window
2. Prepare test environment (see Prerequisites)
3. Follow test cases in order
4. Document all results
5. Report findings

**Expected Outcome:** Phase 1 system validated and confirmed production-ready for Phase 2 vault infrastructure setup.

---

*This protocol is comprehensive and self-contained. It can be executed by anyone with GitHub/governance repo access and can be repeated as needed for regression testing.*