running-claude-smoke-tests

Category: Development & Coding | Uploader: swt-labs | Downloads: 0 | Version: v1.0（Latest）
This skill should be used when the user asks to run or verify a VBW smoke test with plain Claude Code (`claude`), capture visible `Skills:` output, validate `.vbw-planning/.skill-decisions.log`, or confirm consumer-smoke-worktree versus `--plugin-dir` provenance for local smoke coverage in this repo.
Changelog: Source: GitHub https://github.com/swt-labs/vibe-better-with-claude-code-vbw
Directory Structure

Current level: tree/main/.claude/skills/running-claude-smoke-tests/
📄 SKILL.md 13.4 KB
SKILL.md

---
name: running-claude-smoke-tests
description: 'This skill should be used when the user asks to run or verify a VBW smoke test with plain Claude Code (`claude`), capture visible `Skills:` output, validate `.vbw-planning/.skill-decisions.log`, or confirm consumer-smoke-worktree versus `--plugin-dir` provenance for local smoke coverage in this repo.'
argument-hint: '[smoke-goal]'
---

# Running Claude Code smoke tests

Use this skill when validating VBW behavior with plain Claude Code so future issue-fix runs do not have to rediscover the local CLI entrypoint, the consumer repo path, or the provenance checks. This is the plain-Claude counterpart to the CCR smoke-test workflow: keep the worktree and provenance rules the same, but replace `ccr code` with `claude`.

## Core rule

- Do **not** run the smoke test with the VBW plugin repo or plugin worktree as the working directory.
- Do **not** run the smoke test directly in the base consumer repo checkout.
- Create or reuse a **disposable consumer smoke worktree** and run `claude` there, or use a **sandbox repo** when the real consumer repo cannot exercise the flow.
- Use the candidate VBW checkout only as `--plugin-dir`.
- When documenting results, say: **"smoke-tested from a consumer smoke worktree or sandbox while loading the candidate plugin checkout via `--plugin-dir`"**.
- If the transcript path encodes the plugin repo or the base consumer repo checkout instead of the consumer smoke worktree or sandbox, treat the smoke result as suspect until proven otherwise.

## Expected inputs

- Candidate plugin checkout path, usually the current VBW worktree under test.
- Preferred consumer repo path, resolved from the VBW checkout with:
  - `bash scripts/resolve-debug-target.sh repo`
- Smoke worktree label such as `smoke-fix-issue-502` or `smoke-skill-audit`.
- Optional sandbox path such as `/tmp/vbw-skill-smoke-<issue>` for phase-routed flows the real consumer repo cannot exercise.

## Location roles

Keep these four locations distinct throughout the run:

- **Candidate plugin checkout** — the VBW repo checkout under test. Pass this path only through `--plugin-dir`.
- **Consumer repo** — the real target repo used to resolve the testing target and create or remove the smoke worktree. Do not run `claude` from this checkout.
- **Consumer smoke worktree** — the disposable worktree created from the consumer repo. This is the real cwd for consumer-repo smoke tests.
- **Sandbox repo** — a temporary repo used only when the consumer repo cannot exercise the required phase-routed flow.

## Workflow

Copy this checklist into the run if the task is multi-step:

```text
Claude Smoke Test Progress:
- [ ] Confirm candidate plugin checkout/worktree path
- [ ] Resolve real consumer repo path
- [ ] Create or enter the consumer smoke worktree
- [ ] Decide consumer worktree vs sandbox per flow
- [ ] Run smoke command(s) with `claude`
- [ ] Capture visible `Skills:` evidence when needed
- [ ] Capture `.skill-decisions.log` deltas
- [ ] Verify transcript/project path provenance
- [ ] Confirm the base consumer repo and plugin checkout did not become the smoke cwd
```

### 1. Confirm the candidate plugin path

Treat the candidate plugin checkout as the path passed to `--plugin-dir`. In issue-fix runs, this is usually the feature worktree, not the main plugin repo checkout.

### 2. Resolve the real consumer repo first

From the VBW checkout, resolve the preferred smoke target with:

```bash
bash scripts/resolve-debug-target.sh repo
```

- If it returns an absolute path, use that repo as the source checkout for the consumer smoke worktree.
- If it fails and the smoke target must represent real consumer-repo behavior, ask the user for the real consumer repo path.
- Use a dedicated sandbox repo only when the real consumer repo is unavailable or cannot exercise the targeted flow.
- If it resolves to the same path as the candidate plugin checkout, stop and correct the target. The smoke cwd must be the consumer smoke worktree or sandbox, never the candidate plugin checkout.

### 3. Create or enter the consumer smoke worktree

Use a disposable worktree derived from the consumer repo checkout so smoke artifacts do not land in the production checkout.

```bash
CONSUMER_REPO=/absolute/path/to/consumer-repo
SMOKE_LABEL=smoke-fix-issue-502
CONSUMER_NAME=$(basename "$CONSUMER_REPO")
CONSUMER_PARENT=$(cd "$CONSUMER_REPO/.." && pwd)
SMOKE_BASE="${CONSUMER_PARENT}/${CONSUMER_NAME}-worktrees"
SMOKE_WORKTREE="${SMOKE_BASE}/${SMOKE_LABEL}"

mkdir -p "$SMOKE_BASE"

if git -C "$CONSUMER_REPO" worktree list --porcelain | grep -Fqx "worktree $SMOKE_WORKTREE"; then
  :
elif [ -e "$SMOKE_WORKTREE" ]; then
  echo "Target smoke worktree path exists but is not a registered worktree: $SMOKE_WORKTREE" >&2
  exit 1
else
  git -C "$CONSUMER_REPO" worktree add --detach "$SMOKE_WORKTREE" HEAD
fi
```

Run all consumer-repo smoke commands from `SMOKE_WORKTREE`, never from `CONSUMER_REPO`.

### 4. Pick the right target for each flow

Use the **consumer smoke worktree** for flows that do not require special phase state:
- `/vbw:debug`
- `/vbw:fix`
- `/vbw:research`
- `/vbw:map`

Use a **sandbox repo** when the real consumer repo cannot exercise a phase-routed flow cleanly, for example:
- consumer repo has `next_phase_state=no_phases`
- you need `/vbw:qa` against a seeded phase
- you need `/vbw:vibe --execute` or `/vbw:vibe --verify` against controlled phase or UAT state

When you split coverage this way, say so explicitly in the verification summary.

### 5. Use `claude` for this variant

Use the plain Claude Code entrypoint for this variant:

```bash
claude --help
```

We have not runtime-validated this path here. Treat `claude` as the expected local equivalent of `ccr code`, and confirm any flag assumptions with `claude --help` before relying on them.

### 6. Run `claude` from the consumer smoke worktree or sandbox

Before each smoke command, explicitly `cd` into the consumer smoke worktree or sandbox repo. Do not rely on ambient shell state.

### 6.5. Shape sandbox repos for subagent-driven flows

When the smoke target can spawn subagents or task worktrees (for example `/vbw:debug`, `/vbw:fix`, or any flow that asks the agent to inspect repo files), the sandbox repo shape matters:

- Commit the **minimal seed fixtures** that define the bug premise before invoking `claude`.
- If the smoke depends on a file being present, absent, or containing exact text, make that file part of `HEAD` first so subagent worktrees and git-based checks see the same evidence.
- Keep runtime artifacts such as `.vbw-planning/` untracked unless the smoke specifically needs tracked planning state, but do not assume untracked repo fixtures will appear inside subagent worktrees.
- For snapshot-backed numbered selection flows (for example `/vbw:list-todos` followed by `/vbw:debug 1`), keep both commands in the **same persistent session**. Separate one-shot `-p` invocations do not share the numbered snapshot.
- When driving that persistent session through repeated `-p` calls, create the first call with `--session-id <uuid>` and continue later calls with `-r <uuid>` instead of repeating `--session-id`. Reusing `--session-id` on the follow-up call can fail with `Session ID ... is already in use.`

Good sandbox shape for an already-fixed debug smoke:

1. Create sandbox repo.
2. Commit the fixture file whose content establishes the premise.
3. Run `/vbw:init`.
4. Add the todo.
5. Run `/vbw:list-todos` and `/vbw:debug N` in the same session.

For print-mode automation, that means: first call uses `--session-id`, second call uses `-r` for the same UUID.

### 7. Pipe the prompt through stdin

If `claude -p "...prompt..."` reports an input error, switch to stdin and keep it that way for the run:

```bash
cd /absolute/path/to/consumer-smoke-worktree && \
printf '%s\n' '/vbw:fix Investigate a SwiftData save failure during tests.' \
  | claude -p --dangerously-skip-permissions --plugin-dir /absolute/path/to/candidate-plugin-checkout
```

Use `printf '%s\n'` instead of `echo` so punctuation and backslashes are preserved in a shell-portable way.

### 8. Add a smoke-test system prompt

For normal smoke checks, append a short system prompt that reduces side effects and keeps output easy to inspect:

```text
Smoke test only. Avoid code edits. Keep output concise. Focus on diagnosis and reporting.
```

For routing-only probes, make the stop condition explicit:

```text
Smoke test only. Stop after choosing skills and mode. Do not modify project files. Keep output concise.
```

### 9. Capture the right evidence

Always capture:
- command exit code
- transcript or project path proving the smoke cwd was the consumer smoke worktree or sandbox

Then capture the claim-matched evidence you actually need:
- **Visible-selection claim** → capture a visible `Skills:` line
- **Activation or logging claim** → capture `.vbw-planning/.skill-decisions.log` deltas
- **Domain-skill claim** → capture task-specific keyword matches when relevant, such as `SwiftData` or `swiftdata`

#### Standard smoke pattern

```bash
cd /absolute/path/to/consumer-smoke-worktree && \
printf '%s\n' '/vbw:research Research SwiftData persistence failures.' \
  | claude -p --dangerously-skip-permissions \
      --plugin-dir /absolute/path/to/candidate-plugin-checkout \
      --append-system-prompt 'Smoke test only. Avoid code edits. Keep output concise. Focus on diagnosis and reporting.'
```

#### Visible `Skills:` capture pattern

If plain output does not surface the `Skills:` line, use this dedicated stream-capture mode:

```bash
cd /absolute/path/to/consumer-smoke-worktree && \
printf '%s\n' '/vbw:fix Investigate a SwiftData save or schema failure.' \
  | claude -p --verbose --dangerously-skip-permissions \
      --plugin-dir /absolute/path/to/candidate-plugin-checkout \
      --append-system-prompt 'Smoke test only. Avoid code edits. Keep output concise. Focus on diagnosis and reporting.' \
      --output-format stream-json --include-partial-messages
```

Then search the captured output for `Skills:`.

#### Routing-only probe pattern

Use this when you only need mode or skill-selection behavior and do not want the workflow to keep going:

```bash
cd /absolute/path/to/sandbox-repo && \
printf '%s\n' '/vbw:vibe --execute 1' \
  | claude -p --permission-mode plan --dangerously-skip-permissions \
      --plugin-dir /absolute/path/to/candidate-plugin-checkout \
      --append-system-prompt 'Smoke test only. Stop after choosing skills and mode. Do not modify project files. Keep output concise.'
```

### 10. Verify provenance before trusting the result

Use these checks:
- The shell cwd for `claude` must be the consumer smoke worktree or sandbox.
- The transcript path should encode the consumer smoke worktree or sandbox path under `<claude-config-dir>/projects/...`, not the candidate plugin checkout or base consumer repo checkout.
  - Resolve `<claude-config-dir>` the same way this repo does: `CLAUDE_CONFIG_DIR` if set, else `$HOME/.config/claude-code` when that directory exists, else `$HOME/.claude`.
  - Consumer worktree example: `<claude-config-dir>/projects/-path-to-consumer-repo-worktrees-smoke-fix-issue-502/...`
  - Sandbox example: `<claude-config-dir>/projects/-private-tmp-vbw-skill-smoke-502/...`
- The candidate plugin checkout should appear only as `--plugin-dir`.

If the transcript path or cwd points at the plugin repo, do **not** document the smoke test as consumer coverage.

### 11. Verify repo cleanliness when provenance is in doubt

If you suspect the smoke ran in the wrong place:
- run `git status --short` in the base consumer repo checkout
- run `git status --short` in the consumer smoke worktree
- run `git status --short` in the main plugin repo checkout
- run `git status --short` in the candidate plugin checkout if it is a separate worktree
- inspect diffs for unexpected `.vbw-planning` changes

Expected smoke artifacts belong in the **consumer smoke worktree** or **sandbox**, not the base consumer repo checkout or the plugin repo.

## Decision points

- **`resolve-debug-target.sh repo` fails and you need real consumer coverage** → ask for the consumer repo path instead of guessing.
- **Need real consumer coverage** → create or reuse the consumer smoke worktree before running any `claude` command.
- **Real consumer repo cannot exercise `/vbw:vibe` or `/vbw:qa`** → use a sandbox for those flows and keep the consumer smoke worktree for `/vbw:debug`, `/vbw:fix`, `/vbw:research`, and `/vbw:map`.
- **Plain `-p` invocation reports missing input** → switch to stdin and keep it that way.
- **Need visible `Skills:` output** → use stream-json plus `--verbose` and search the capture.
- **Need routing proof without side effects** → use `--permission-mode plan` plus the routing-only smoke prompt.
- **Base consumer repo checkout, candidate plugin checkout, or main plugin repo shows unexpected `.vbw-planning` changes** → stop and investigate provenance before writing the verification summary.

## Completion checks

A smoke run is ready to cite only when all are true:
- The working directory was the consumer smoke worktree or sandbox.
- `--plugin-dir` pointed at the candidate plugin checkout.
- Required command outputs were captured for the claim being made.
- Provenance evidence was captured.
- `.skill-decisions.log` and/or visible `Skills:` evidence supports the specific claim being made.
- The base consumer repo checkout did not become the smoke cwd or collect smoke artifacts.
- Any sandbox use is called out explicitly.
- The plugin repo is not being misrepresented as the smoke-test cwd.
Comments 0
Latest | Hot
Please login before commenting.
Loading comments...
running-claude-smoke-tests

Directory Structure

SKILL.md

Report

Notice
