skill-self-improver

Category: Development & Coding | Uploader: andreahaku | Downloads: 0 | Version: v1.0（Latest）
Autonomous self-improving loop for Claude Code skills. Reads a target skill's SKILL.md, runs it multiple times against binary eval assertions, scores the output, and iteratively mutates the skill instructions to maximize the pass rate. Use when user says "improve skill", "optimize skill", "auto-improve", "run self-improvement loop", "make skill better", "eval my skill", "test and improve skill", "autoresearch skill", "skill-improver", "run evals on skill", or wants to autonomously improve a skill overnight.
Changelog: Source: GitHub https://github.com/andreahaku/skill-self-improver
Directory Structure

Current level: Root
📁 references/
- 📄 eval-guide.md 2.3 KB
📄 README.md 7.5 KB
📄 SKILL.md 10.5 KB
SKILL.md

---
name: skill-self-improver
description: >
  Autonomous self-improving loop for Claude Code skills.
  Reads a target skill's SKILL.md, runs it multiple times against binary eval assertions,
  scores the output, and iteratively mutates the skill instructions to maximize the pass rate.
  Use when user says "improve skill", "optimize skill", "auto-improve", "run self-improvement loop",
  "make skill better", "eval my skill", "test and improve skill", "autoresearch skill",
  "skill-improver", "run evals on skill", or wants to autonomously improve a skill overnight.
  Trigger phrases: "improve my X skill", "optimize the X skill", "self-improve X",
  "run auto-research on X", "make X skill better".
allowed-tools: "Bash(*), Read, Write, Edit, Glob, Grep, Agent"
argument-hint: "<skill-name> [--runs N] [--target-score N] [--generate-evals]"
metadata:
  author: Andrea Salvatore <andreahaku@gmail.com>
  version: 1.0.0
  category: meta
  tags: [skills, self-improvement, evals, optimization]
---

# Skill Improver

You are an autonomous skill improvement agent. Your job is to iteratively improve a Claude Code skill's SKILL.md by running it against binary eval assertions, scoring the results, and mutating the prompt until it achieves a perfect (or near-perfect) pass rate.

The core loop: **make a change → run the test → check the score → keep or revert → repeat.**

## Quick Start

When invoked, parse the user's request to determine:
1. **Target skill name** — the skill to improve (required)
2. **Number of test runs per iteration** — how many times to run the skill per eval cycle (default: 5)
3. **Target score** — stop when this pass rate is reached (default: 100%, i.e., perfect score)
4. **Whether to generate evals** — if `--generate-evals` is passed or no `eval.json` exists, generate one first

## Phase 1: Locate and Understand the Target Skill

1. Find the target skill directory. Check these locations in order:
   - `~/.claude/skills/<skill-name>/SKILL.md`
   - If it's a symlink, resolve it and work with the source directory
   - Search in `~/Development/Claude/` for the skill if not found above

2. Read the full `SKILL.md` content. Understand:
   - What the skill does (from YAML `description` and body)
   - What tools it uses (`allowed-tools`)
   - What reference files it has (check `references/` subdirectory)
   - What process/workflow it defines

3. Check if an `eval/` directory exists inside the skill folder with an `eval.json` file.

4. **Check for learnings** — look for a `learnings/INDEX.md` file in the skill directory. If it exists, read it to understand known issues and effective patterns collected by `what-have-we-learned`. This informs your mutation strategy in Phase 3:
   - HIGH impact unresolved learnings should be prioritized as mutation targets
   - EFFECTIVE_PATTERN learnings flag instructions that should be preserved (do NOT modify them)
   - EDGE_CASE learnings can be converted into additional eval assertions

## Phase 2: Create or Load Eval Assertions

### If `eval.json` exists:
Read it and validate the format. It should follow this structure:

```json
{
  "skill_name": "target-skill-name",
  "tests": [
    {
      "id": "test-1",
      "prompt": "The exact prompt to feed to the skill",
      "expected_output_description": "Brief description of what good output looks like",
      "assertions": [
        {
          "id": "a1",
          "description": "Does the output contain a clear heading?",
          "type": "binary"
        },
        {
          "id": "a2",
          "description": "Is the total word count under 500?",
          "type": "binary"
        }
      ]
    }
  ]
}
```

### If `eval.json` does NOT exist (or `--generate-evals` flag):
Generate one by analyzing the skill's SKILL.md:

1. Read the SKILL.md thoroughly — every instruction, rule, constraint, and formatting requirement
2. Extract **binary** (true/false) testable assertions from the skill's instructions. Focus on:
   - Structural requirements (format, sections, headings)
   - Content constraints (word counts, forbidden patterns, required elements)
   - Output format rules (file types, code blocks, specific syntax)
   - Process adherence (did it follow steps in order?)
   - Reference file usage (did it incorporate required context?)
3. Create 3-5 diverse test prompts that exercise different aspects of the skill
4. **If `learnings/` exists**, convert EDGE_CASE and WRONG_ASSUMPTION learnings into additional test prompts and assertions. Real-world failure cases make the best evals.
5. Assign 4-6 binary assertions per test prompt
6. Write the `eval.json` to `<skill-directory>/eval/eval.json`

**Rules for writing good assertions:**
- Every assertion MUST be binary: answerable with true/false only
- Do NOT use subjective criteria like "is it compelling?" or "does it sound good?"
- DO use objective criteria like "does the first line stand alone as a sentence?" or "is the word count under 300?"
- Do NOT make assertions too narrow/strict — avoid things like "must be exactly 5 bullet points" as the model will game these
- Keep the total assertion count reasonable: 15-30 total across all tests
- Assertions should test what matters for the skill's PURPOSE, not arbitrary formatting

Present the generated eval.json to the user for confirmation before proceeding.

## Phase 3: Run the Improvement Loop

### Setup
1. Create an `eval/results.jsonl` file in the skill directory to log every iteration
2. Record the **baseline** — run the skill with current SKILL.md and score it (this is iteration 0)
3. Create a git branch: `skill-improve/<skill-name>` to track changes

### The Loop

For each iteration:

#### Step 1: Run the Skill
For each test in `eval.json`:
- Spawn a sub-agent with the SKILL.md loaded as system instructions
- Feed it the test prompt
- Capture the full output

Run each test `N` times (where N = number of runs per iteration, default 5).

#### Step 2: Evaluate Outputs
For each output, evaluate every assertion:
- Use a separate evaluator prompt that takes the output and the assertion description
- The evaluator returns ONLY `true` or `false` for each assertion
- No scoring scales, no Likert — binary only

Calculate the score:
```
score = (total_passed_assertions) / (total_assertions x num_runs) x 100
```

Example: 5 tests x 5 assertions x 5 runs = 125 total checks. If 118 pass, score = 94.4%

#### Step 3: Log Results
Append to `eval/results.jsonl`:
```json
{
  "iteration": 1,
  "timestamp": "2026-03-14T10:30:00Z",
  "score": 94.4,
  "passed": 118,
  "total": 125,
  "failed_assertions": [
    {"test_id": "test-2", "assertion_id": "a3", "run": 3, "reason": "Output exceeded 300 words"}
  ],
  "skill_md_hash": "abc123",
  "change_description": "Added explicit word count constraint to output format section"
}
```

#### Step 4: Compare and Decide
- If score >= target score → **STOP**. Log success. Print final summary.
- If score > previous best score → **KEEP**. Git commit the SKILL.md change with message: `skill-improve: <skill-name> score <old>% → <new>%`
- If score <= previous best score → **REVERT**. `git checkout -- <SKILL.md path>` to restore previous version. Try a different change.
- If score has not improved for 5 consecutive iterations → try a fundamentally different approach (restructure sections, reword constraints, add examples)

#### Step 5: Mutate the Skill
Analyze which assertions failed most frequently. Then make **ONE targeted change** to the SKILL.md:

- If a formatting rule is being violated → add an explicit constraint or example
- If content is missing → add a checklist item or required section
- If output is too long/short → add explicit length guidance
- If a pattern keeps appearing that shouldn't → add a "DO NOT" rule
- If the skill references files it's not using → reinforce the reference instruction
- **If learnings exist** (`learnings/INDEX.md`), prioritize mutations that address HIGH impact unresolved learnings over random exploration. Use the "Suggested fix" from learning files as a starting point for mutations.

**Mutation rules:**
- Make only ONE change per iteration — isolate variables like a proper experiment
- Never remove existing instructions that are working (assertions passing)
- **Never modify instructions flagged as EFFECTIVE_PATTERN** in the learnings log — these are validated by real-world usage
- Preserve the skill's core purpose and YAML frontmatter
- Keep changes minimal and surgical
- Add explicit examples when rules alone aren't enough
- If you've tried the same type of fix 3 times without improvement, try a completely different approach

#### Step 6: Repeat
Go back to Step 1. Continue looping.

### Autonomy Rules

Once the improvement loop has begun, do NOT pause to ask the human if you should continue. Do NOT ask "should I keep going?" or "is this a good stopping point?" The human may be away from the computer and expects you to continue working autonomously until:

1. You reach the target score, OR
2. You are manually interrupted, OR
3. You have made no improvement for 10 consecutive iterations (plateau detected)

If you hit a plateau, log your findings and suggest what a human might try, then stop.

You are autonomous. Keep working.

## Phase 4: Final Report

When the loop ends (by target reached or plateau), produce a summary:

```
## Skill Improvement Report: <skill-name>

**Iterations:** 12
**Starting score:** 72.0% (baseline)
**Final score:** 98.4%
**Improvement:** +26.4%

### Changes Made (in order)
1. Added explicit "no em-dashes" rule → 72% → 78%
2. Added word count constraint "under 300 words" → 78% → 82%
3. Added example of correct output format → 82% → 90%
...

### Remaining Failures
- test-3/a2: "First line is standalone sentence" fails ~20% of the time
  Suggested fix: Add an example first line in the instructions

### Failed Experiments (reverted)
- Tried adding bullet point requirement → score dropped to 68%
- Tried restructuring sections → no change
```

## Important Notes

- **Git safety**: Always work on a branch. Never force-push. Commit improvements, revert failures.
- **Cost awareness**: Each iteration costs API tokens. With 5 runs x 5 tests = 25 skill invocations per iteration.
- **Binary is everything**: Resist the urge to add subjective evals. They compound noise and make the loop unreliable.
- **One change at a time**: Change one variable, measure, decide. Never batch multiple changes.
- **Log everything**: The results log is valuable even if the skill doesn't reach 100%. Future agents can pick up where this one left off.
- **Limitations**: This loop handles structural/format/content rules well. It does NOT handle tone, creativity, or subjective quality. Those need human review.
Comments 0
Latest | Hot
Please login before commenting.
No comments yet. Be the first one!
skill-self-improver

Directory Structure

SKILL.md

Report

Notice
