acceptance-test-runner

Category: Tools & Productivity | Uploader: bjornslib | Downloads: 0 | Version: v1.0（Latest）
This skill should be used when the user asks to "run acceptance tests", "validate against PRD", "test the outcomes", "execute E2E validation for PRD-XXX", "verify acceptance criteria", or when the validation-test-agent routes --mode=e2e requests. Executes acceptance tests and generates evidence-based validation reports.
Changelog: Source: GitHub https://github.com/bjornslib/cobuilder-harness
Directory Structure

Current level: .claude/skills/acceptance-test-runner/
📁 examples/
- 📄 validation-report.md 5.1 KB
📁 references/
- 📄 chrome-mcp-actions.md 8.5 KB
- 📄 report-template.md 4.1 KB
📄 SKILL.md 9.6 KB
SKILL.md

---
name: acceptance-test-runner
description: >
  This skill should be used when the user asks to "run acceptance tests",
  "validate against PRD", "test the outcomes", "execute E2E validation for PRD-XXX",
  "verify acceptance criteria", or when the validation-test-agent routes --mode=e2e
  requests. Executes acceptance tests and generates evidence-based validation reports.
title: "Acceptance Test Runner"
status: active
---

# Acceptance Test Runner

Execute acceptance test scripts and generate evidence-based validation reports that prove whether implementations meet PRD requirements.

## Purpose

Run acceptance tests generated by `acceptance-test-writer` and produce:
- Binary PASS/FAIL for each criterion
- Evidence (screenshots, API responses) for each test
- Gap analysis for failures with root cause hypotheses
- Actionable feedback for workers to fix issues

## When to Use

This skill is invoked:
- By `validation-test-agent` when `--mode=e2e` is specified
- Directly when validating against a PRD
- After worker reports "done" to verify implementation

## Input Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `--prd` | Yes | PRD identifier (e.g., `PRD-AUTH-001`) |
| `--criterion` | No | Specific criterion ID to run (e.g., `AC-user-login`) |
| `--task_id` | No | Task ID for traceability in reports |

Example invocations:
```
# Run all acceptance tests for a PRD
Skill("acceptance-test-runner", args="--prd=PRD-AUTH-001")

# Run specific criterion only
Skill("acceptance-test-runner", args="--prd=PRD-AUTH-001 --criterion=AC-user-login")

# With task traceability
Skill("acceptance-test-runner", args="--prd=PRD-AUTH-001 --task_id=TASK-123")
```

## Output Structure

Reports are stored within the same `acceptance-tests/` directory as the test definitions:

```
acceptance-tests/PRD-XXX/
├── manifest.yaml                  # Test suite metadata
├── AC-user-login.yaml            # Test definitions
├── AC-password-reset.yaml
└── runs/                          # Execution results
    ├── 2026-01-24T10-30-00Z.md    # Timestamped report
    └── evidence/
        ├── AC-user-login-success.png
        ├── AC-user-login-step2.png
        └── AC-api-auth-response.json
```

## Execution Workflow

### Step 1: Load Test Definitions

```bash
# Verify acceptance tests exist
ls acceptance-tests/${PRD_ID}/

# Load manifest
cat acceptance-tests/${PRD_ID}/manifest.yaml
```

If no acceptance tests found, return error:
```
ERROR: No acceptance tests found for ${PRD_ID}
Expected: acceptance-tests/${PRD_ID}/manifest.yaml
Action: Run acceptance-test-writer to generate tests first
```

### Step 2: Prepare Evidence Directory

```bash
# Create timestamped evidence directory
TIMESTAMP=$(date -u +"%Y-%m-%dT%H-%M-%SZ")
mkdir -p acceptance-tests/${PRD_ID}/runs/evidence
```

### Step 3: Execute Each Criterion

For each acceptance criterion, spawn a **Sonnet sub-agent**:

```python
Task(
    subagent_type="general-purpose",
    model="sonnet",  # Sonnet for capability
    description=f"Execute acceptance test: {criterion_id}",
    prompt=f"""Execute acceptance test for criterion: {criterion_id}

## Test Definition
{criterion_yaml_content}

## Your Mission
1. Execute each step in order
2. Capture evidence as specified
3. Report PASS or FAIL with details

## Tools Available
- **Browser tests**: Use chrome-devtools MCP
  - mcp__chrome-devtools__navigate
  - mcp__chrome-devtools__click
  - mcp__chrome-devtools__fill
  - mcp__chrome-devtools__screenshot
  - mcp__chrome-devtools__evaluate (for assertions)

- **API tests**: Use Bash with curl
  - curl for HTTP requests
  - jq for JSON parsing

## Evidence Capture
Save all evidence to: acceptance-tests/{prd_id}/runs/evidence/
- Screenshots: {criterion_id}-{{description}}.png
- API responses: {criterion_id}-{{description}}.json

## Return Format
Return a structured result:
```yaml
criterion_id: {criterion_id}
status: PASS | FAIL
duration_seconds: <number>
evidence_files:
  - path: "evidence/..."
    description: "..."
steps_executed:
  - step_id: step-1
    status: PASS | FAIL
    details: "..."
failure_reason: null | "Description of why it failed"
actual_outcome: "What actually happened"
```
"""
)
```

### Step 4: Aggregate Results

Collect results from all sub-agents:
- Count PASS/FAIL/SKIP
- Gather evidence paths
- Identify blocking failures

### Step 5: Generate Report

Create comprehensive markdown report at `acceptance-tests/{PRD_ID}/runs/{TIMESTAMP}.md`.

Report structure:
1. **Header**: PRD info, timestamp, environment
2. **Summary table**: PASS/FAIL counts with percentages
3. **Results by criterion**: Detailed results for each test
4. **What Works**: List of passing functionality
5. **What Doesn't Work**: List of failures
6. **Blocking Issues**: Criteria that must pass before closure
7. **Recommendations**: Actionable next steps
8. **Evidence Files**: Table linking evidence to criteria

See `references/report-template.md` for complete template.

### Step 6: Return Summary

Return concise summary to caller:
```yaml
prd_id: PRD-AUTH-001
overall_status: PASS | PARTIAL | FAIL
criteria_results:
  total: 5
  passed: 4
  failed: 1
  skipped: 0
blocking_failures:
  - AC-password-reset-complete
report_path: acceptance-tests/PRD-AUTH-001/runs/2026-01-24T10-30-00Z.md
```

## Browser Test Execution

For `validation_type: browser` tests, use chrome-devtools MCP:

### Action Mapping

| YAML Action | MCP Tool |
|-------------|----------|
| `navigate` | `mcp__chrome-devtools__navigate` |
| `fill` | `mcp__chrome-devtools__fill` |
| `click` | `mcp__chrome-devtools__click` |
| `assert_visible` | `mcp__chrome-devtools__evaluate` with querySelector |
| `assert_url` | `mcp__chrome-devtools__evaluate` with location.href |
| `screenshot` | `mcp__chrome-devtools__screenshot` |
| `wait_for_navigation` | Wait + verify URL changed |
| `wait_for_element` | Poll with evaluate until element exists |

### Example Browser Execution

```python
# Step: navigate to /login
mcp__chrome-devtools__navigate(url="http://localhost:3000/login")

# Step: fill email input
mcp__chrome-devtools__fill(selector="[data-testid='email-input']", value="test@example.com")

# Step: click button
mcp__chrome-devtools__click(selector="[data-testid='login-button']")

# Step: assert_visible with text check
result = mcp__chrome-devtools__evaluate(
    expression="document.querySelector('[data-testid=\"user-greeting\"]')?.textContent"
)
assert "Welcome" in result

# Step: capture screenshot
mcp__chrome-devtools__screenshot(path="acceptance-tests/PRD-AUTH-001/runs/evidence/AC-user-login-success.png")
```

## API Test Execution

For `validation_type: api` tests, use Bash with curl:

### Example API Execution

```bash
# Make request and capture response
curl -s -w "\n%{http_code}" \
  -X GET "http://localhost:8000/api/user/profile" \
  -H "Authorization: Bearer invalid-token" \
  -H "Content-Type: application/json" \
  > /tmp/response.txt

# Extract status code (last line)
STATUS=$(tail -n1 /tmp/response.txt)

# Extract body (all but last line)
BODY=$(sed '$d' /tmp/response.txt)

# Assert status
[ "$STATUS" = "401" ] && echo "PASS: Status is 401" || echo "FAIL: Expected 401, got $STATUS"

# Assert JSON field
echo "$BODY" | jq -e '.error == "Unauthorized"' && echo "PASS: Error field correct"

# Save evidence
echo "$BODY" | jq . > acceptance-tests/PRD-AUTH-001/runs/evidence/AC-api-auth-response.json
```

## Failure Analysis

When a criterion fails, the sub-agent should provide:

1. **Which step failed**: Identify exact step ID
2. **Expected vs Actual**: What should have happened vs what did
3. **Root cause hypothesis**: Best guess at why it failed
4. **Recommended action**: Specific fix suggestion

Example failure analysis:
```markdown
### Failure Analysis
- **Step 3 failed**: navigate to /reset-password/:token returned 404
- **Expected**: Page loads with password reset form
- **Actual**: 404 Not Found error page
- **Root Cause Hypothesis**: The `/reset-password/:token` route is not implemented in the frontend router
- **Recommended Action**:
  1. Add route in `app/router.tsx` for `/reset-password/:token`
  2. Create `ResetPasswordPage` component
  3. Re-run this acceptance test
```

## Integration with Validation Agent

The validation-test-agent invokes this skill for `--mode=e2e`:

```python
# In validation-test-agent routing logic
if mode == "e2e" and prd:
    if Path(f"acceptance-tests/{prd}").exists():
        Skill("acceptance-test-runner", args=f"--prd={prd} --task_id={task_id}")
    else:
        warn(f"No acceptance tests for {prd} - falling back to generic E2E")
```

## Additional Resources

### Reference Files

- **`references/report-template.md`** - Complete validation report template
- **`references/chrome-mcp-actions.md`** - Browser automation patterns

### Example Files

- **`examples/validation-report.md`** - Sample validation report

## Error Handling

### No Acceptance Tests Found
```
ERROR: No acceptance tests found for PRD-AUTH-001
Path checked: acceptance-tests/PRD-AUTH-001/
Action: Generate acceptance tests first with:
  Skill("acceptance-test-writer", args="--prd=PRD-AUTH-001 --source=<prd-path>")
```

### Browser Not Available
```
ERROR: Chrome DevTools MCP not available
Action: Ensure Chrome is running with debugging enabled
  or use mcp__chrome-devtools__launch_browser first
```

### Sub-Agent Timeout
```
WARN: Criterion AC-user-login timed out after 60s
Status: SKIP
Action: Check if application is running and responsive
```

## Checklist Before Returning

- [ ] All criteria executed (or marked as SKIP with reason)
- [ ] Evidence captured for each criterion
- [ ] Report generated at `acceptance-tests/{PRD_ID}/runs/{TIMESTAMP}.md`
- [ ] Summary returned to caller with overall status
- [ ] Blocking failures clearly identified
Comments 0
Latest | Hot
Please login before commenting.
No comments yet. Be the first one!
acceptance-test-runner

Directory Structure

SKILL.md

Report

Notice
