skill-auditor

Category: Testing & Security | Uploader: profburial | Downloads: 0 | Version: v1.0（Latest）

Background knowledge for auditing generated .claude/ skills and commands — how to evaluate quality, run benchmarks, and iteratively improve them using test cases and quantitative metrics.

Changelog: Source: GitHub https://github.com/profburial/compounding-skills

Directory Structure

Current level: plugins/compounding-skills/skills/skill-auditor/

📁 agents/
- 📄 analyzer.md 10.1 KB
- 📄 comparator.md 7.1 KB
- 📄 grader.md 8.8 KB
📁 assets/
- 📄 eval_review.html 6.9 KB
📁 eval-viewer/
- 📄 generate_review.py 16.0 KB
- 📄 viewer.html 43.9 KB
📁 references/
- 📄 schemas.md 11.8 KB
📁 scripts/
- 📄 __init__.py 0 B
- 📄 aggregate_benchmark.py 14.0 KB
- 📄 generate_report.py 12.5 KB
- 📄 improve_description.py 10.5 KB
- 📄 package_skill.py 4.1 KB
- 📄 quick_validate.py 3.9 KB
- 📄 run_eval.py 11.2 KB
- 📄 run_loop.py 13.4 KB
- 📄 utils.py 1.6 KB
📄 SKILL.md 6.5 KB

SKILL.md

---
name: skill-auditor
description: Background knowledge for auditing generated .claude/ skills and commands — how to evaluate quality, run benchmarks, and iteratively improve them using test cases and quantitative metrics.
user-invocable: false
---

# Skill Auditor

Evaluation and improvement infrastructure for skills and commands generated by `compounding-skills-setup`. Loaded by `compounding-skills-audit`.

## Purpose

This skill teaches Claude how to:

1. **Audit generated skills and commands** — evaluate whether they actually improve Claude's output quality
2. **Run structured evals** — test with realistic prompts, grade outputs, and aggregate benchmarks
3. **Iterate on improvements** — use human feedback and quantitative data to refine until they're genuinely useful

## Infrastructure

All tooling lives in this skill's bundled directories. The scripts, viewer, and agents are identical to the ones used by Anthropic's official skill-creator plugin — same playbook, same rigor.

### Scripts

| Script | Purpose |
|--------|---------|
| `scripts/run_eval.py` | Tests whether a skill description triggers Claude to invoke it |
| `scripts/aggregate_benchmark.py` | Aggregates grading results into statistical summaries |
| `scripts/run_loop.py` | Orchestrates the full eval + description improvement loop |
| `scripts/improve_description.py` | Uses Claude with extended thinking to optimize skill descriptions |
| `scripts/generate_report.py` | Generates HTML reports from optimization loop output |
| `scripts/quick_validate.py` | Validates SKILL.md frontmatter (name, description, format) |
| `scripts/package_skill.py` | Creates distributable .skill files (ZIP archives) |
| `scripts/utils.py` | Shared utilities (YAML frontmatter parsing) |

### Eval Viewer

| File | Purpose |
|------|---------|
| `eval-viewer/generate_review.py` | Generates interactive HTML viewer for reviewing eval outputs and benchmarks |
| `eval-viewer/viewer.html` | Rich HTML template with Outputs tab, Benchmark tab, and feedback collection |

### Agents

| Agent | Purpose |
|-------|---------|
| `agents/grader.md` | Evaluates assertions against execution outputs — produces structured grading.json |
| `agents/comparator.md` | Blind A/B comparison between two outputs without knowing which skill produced them |
| `agents/analyzer.md` | Analyzes benchmark patterns and explains why one version outperformed another |

### References

| File | Purpose |
|------|---------|
| `references/schemas.md` | JSON schemas for evals.json, grading.json, benchmark.json, timing.json, and all other data structures |

### Assets

| File | Purpose |
|------|---------|
| `assets/eval_review.html` | Interactive HTML template for reviewing and editing trigger eval sets |

## Evaluation Methodology

The audit follows the same methodology as Anthropic's skill-creator:

### Dual-Run Comparison

Every test case runs twice — once **with the skill/command** and once **without it** (baseline). This isolates the actual contribution. If Claude produces equally good output without it, the skill or command isn't pulling its weight.

### Structured Grading

The grader agent evaluates each run against predefined assertions. It goes beyond pass/fail — it extracts implicit claims, verifies them against outputs, and critiques whether the assertions themselves are discriminating enough to be useful.

### Benchmark Aggregation

Results are aggregated with mean, stddev, min, max across configurations. The analyzer surfaces patterns the numbers alone won't show: assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.

### Human-in-the-Loop Review

The eval viewer presents outputs side-by-side with benchmark data. The human reviews qualitatively (does this output actually look good?) while the benchmarks provide quantitative rigor. Feedback drives the next iteration.

## Workspace Layout

Results are organized by iteration and eval:

```
{name}-workspace/
├── iteration-1/
│   ├── eval-descriptive-name/
│   │   ├── eval_metadata.json
│   │   ├── with_skill/           (or with_command/ for commands)
│   │   │   ├── outputs/
│   │   │   ├── timing.json
│   │   │   └── grading.json
│   │   └── without_skill/        (or without_command/)
│   │       ├── outputs/
│   │       ├── timing.json
│   │       └── grading.json
│   ├── benchmark.json
│   └── benchmark.md
├── iteration-2/
│   └── ...
└── feedback.json
```

## Key Principles for Auditing

### 1. Test What Matters

Generated skills contain real code examples and project-specific conventions. Good test prompts should exercise whether those conventions actually guide Claude's behavior — not just whether Claude can complete the task at all.

Generated commands define multi-phase workflows. Good test prompts should exercise whether the workflow produces better results than ad-hoc execution — better plans, more thorough reviews, more structured output.

### 2. Baseline Is Essential

A skill or command that doesn't outperform the baseline is overhead. Claude is already capable — the artifact needs to demonstrably improve output quality, consistency, or adherence to project conventions to justify its context window cost.

### 3. Generalize, Don't Overfit

Test cases are examples, not the universe of usage. When improving based on test results, make changes that will generalize to the many prompts users will actually throw at it. Explain the *why* behind instructions rather than adding rigid rules that only fix the specific test case.

### 4. Lean Beats Comprehensive

If the benchmark shows a section isn't contributing to better outcomes, consider removing it. Every line costs context window space — unused instructions actively hurt by crowding out useful ones.

### 5. Commands Have Different Success Criteria

Skills are judged on whether code output follows conventions. Commands are judged on whether the *workflow* adds value:
- Does a `plan` command produce more thorough, actionable plans than ad-hoc planning?
- Does a `review` command catch issues that a general review misses?
- Does a `brainstorm` command lead to better-explored design decisions?
- Does a `work` command produce more consistent, tested implementations?
- Does a `compound` command extract genuinely useful patterns?

Command improvements often involve streamlining phases, improving question quality, or removing steps that don't contribute to better outcomes.

Comments 0

Latest | Hot

Please login before commenting.

No comments yet. Be the first one!

skill-auditor

Directory Structure

SKILL.md

Report

Notice