skill-optimizer

Category: Tools & Productivity | Uploader: hqhq1025 | Downloads: 0 | Version: v1.0（Latest）
Use when the user wants to analyze, audit, or improve their Agent Skills (SKILL.md files). Triggers on /optimize-skill, /skill-audit, 'optimize skills', 'analyze skills', 'check my skills', 'skill quality'. Also use proactively when the user mentions skills aren't triggering, skills feel broken, or asks why a skill didn't fire.
Changelog: Source: GitHub https://github.com/hqhq1025/skill-optimizer
Directory Structure

Current level: tree/main/skills/skill-optimizer/
📄 SKILL.md 12.4 KB
SKILL.md

---
name: skill-optimizer
description: "Use when the user wants to analyze, audit, or improve their Agent Skills (SKILL.md files). Triggers on /optimize-skill, /skill-audit, 'optimize skills', 'analyze skills', 'check my skills', 'skill quality'. Also use proactively when the user mentions skills aren't triggering, skills feel broken, or asks why a skill didn't fire."
---

## Rules

- **Read-only**: never modify skill files. Only output report.
- **All 8 dimensions**: do not skip any. If data is insufficient, report "N/A — insufficient session data" rather than omitting.
- **Quantify**: "you had 12 research tasks last week but the skill never triggered" beats "you often do research".
- **Suggest, don't prescribe**: give specific wording suggestions for description improvements, but frame as suggestions.
- **Show evidence**: for undertrigger claims, quote the actual user message that should have triggered the skill.
- **Evidence-based suggestions**: when suggesting description rewrites, cite the specific research finding that motivates the change (e.g., "front-load trigger keywords — MCP study shows 3.6x selection rate improvement").

## Overview

Analyze skills using **historical session data + static quality checks**, output a diagnostic report with P0/P1/P2 prioritized fixes. Scores each skill on a 5-point composite scale across 8 dimensions.

CSO (Claude/Agent Search Optimization) = writing skill descriptions so agents select the right skill at the right time. This skill checks for CSO violations.

## Usage

- `/optimize-skill` → scan all skills
- `/optimize-skill my-skill` → single skill
- `/optimize-skill skill-a skill-b` → multiple specified skills

## Data Sources

Auto-detect the current agent platform and scan the corresponding paths:

| Source | Claude Code | Codex | Shared |
|--------|------------|-------|--------|
| Session transcripts | `~/.claude/projects/**/*.jsonl` | `~/.codex/sessions/**/*.jsonl` | — |
| Skill files | `~/.claude/skills/*/SKILL.md` | `~/.codex/skills/*/SKILL.md` | `~/.agents/skills/*/SKILL.md` |

**Platform detection:** Check which directories exist. Scan all available sources — a user may have both Claude Code and Codex installed.

## Workflow

```
Identify target skills
        ↓
Collect session data (python3 scripts scan JSONL transcripts)
        ↓
Run 8 analysis dimensions
        ↓
Compute composite scores
        ↓
Output report with P0/P1/P2
```

### Step 1: Identify Target Skills

Scan skill directories in order: `~/.claude/skills/`, `~/.codex/skills/`, `~/.agents/skills/`. Deduplicate by skill name (same name in multiple locations = same skill). For each, read `SKILL.md` and extract:
- name, description (from YAML frontmatter)
- trigger keywords (from description field)
- defined workflow steps (Step 1/2/3... or ### sections under Workflow)
- word count

If user specified skill names, filter to only those.

### Step 2: Collect Session Data

Use python3 scripts via Bash to scan session JSONL files. Extract:

**Claude Code sessions** (`~/.claude/projects/**/*.jsonl`):
- `Skill` tool_use calls (which skills were invoked)
- User messages (full text)
- Assistant messages after skill invocation (for workflow tracking)
- User messages after skill invocation (for reaction analysis)

**Codex sessions** (`~/.codex/sessions/**/*.jsonl`):
- `session_meta` events → extract `base_instructions` for skill loading evidence
- `response_item` events → assistant outputs (workflow tracking)
- `event_msg` events → tool execution and skill-related events
- User messages from `turn_context` events (for reaction analysis)

**Note:** Codex injects skills via context rather than explicit `Skill` tool calls. Skill loading (present in `base_instructions`) does NOT equal active invocation. To detect actual use, search for skill-specific workflow markers (step headers, output formats) in `response_item` content within that session. A skill is "invoked" only if the agent produced output following the skill's defined workflow.

**Aggregated:**
- Per-skill: invocation count, trigger keyword match count
- Per-skill: user reaction sentiment after invocation
- Per-skill: workflow step completion markers

### Step 3: Run 8 Analysis Dimensions

**You MUST run ALL 8 dimensions.** The baseline behavior without this skill is to skip dimensions 4.2, 4.3, 4.5b, and 4.8. These are the most valuable dimensions — do not skip them.

#### 4.1 Trigger Rate

Count how many times each skill was actually invoked vs how many times its trigger keywords appeared in user messages.

**Claude Code:** count `Skill` tool_use calls in transcripts.
**Codex:** count sessions where the agent produced output following the skill's workflow markers (not merely loaded in context).

**Diagnose:**
- Never triggered → skill may be useless or trigger words wrong
- Keywords match >> actual invocations → undertrigger problem, description needs work
- High frequency → core skill, worth optimizing

#### 4.2 Post-Invocation User Reaction

**This dimension is critical and easy to skip. Do not skip it.**

After a skill is invoked in a session, read the user's next 3 messages. Classify:
- **Negative**: "no", "wrong", "never mind", "not what I wanted", user interrupts
- **Correction**: user re-describes their intent, manually overrides skill output
- **Positive**: "good", "ok", "continue", "nice", user follows the workflow
- **Silent switch**: user changes topic entirely (likely false positive trigger)

Report per-skill satisfaction rate.

#### 4.3 Workflow Completion Rate

**This dimension is critical and easy to skip. Do not skip it.**

For each skill invocation found in session data:
1. Extract the skill's defined steps from SKILL.md
2. Search the assistant messages in that session for step markers (Step N, specific output formats defined in the skill)
3. Calculate: how far did execution get?

Report: `{skill-name} (N steps): avg completed Step X/N (Y%)`

If a specific step is frequently where execution stops, flag it.

#### 4.4 Static Quality Analysis

Check each SKILL.md against these 14 rules:

| Check | Pass Criteria |
|-------|--------------|
| Frontmatter format | Only `name` + `description`, total < 1024 chars |
| Name format | Letters, numbers, hyphens only |
| Description trigger | Starts with "Use when..." or has explicit trigger conditions |
| Description workflow leak | Description does NOT summarize the skill's workflow steps (CSO violation) |
| Description pushiness | Description actively claims scenarios where it should be used, not just passive |
| Overview section | Present |
| Rules section | Present |
| MUST/NEVER density | Count ALL-CAPS directive words; >5 per 100 words = flag. Note: Meincke et al. (2025) found persuasion directives have inconsistent effects across models. Suggest converting to concrete bright-line rules with rationale, not mere emphasis. |
| Word count | < 500 words (flag if over) |
| Narrative anti-pattern | No "In session X, we found..." storytelling — skills should be instructions, not post-hoc reports |
| YAML quoting safety | description containing `: ` must be wrapped in double quotes, otherwise YAML parse failure makes skill invisible |
| Critical info position | Core trigger conditions and primary actions must be in the first 20% of SKILL.md, not buried in the middle (Lost in the Middle, Liu et al. TACL 2024: U-shaped attention curve) |
| Description 250-char check | Primary trigger keywords must appear within the first 250 characters of description (skill listing truncation point in most agents) |
| Trigger condition count | ≤ 2 trigger conditions in description is ideal; consistent with IFEval (Zhou et al. 2023) finding that LLMs struggle with multi-constraint prompts |

#### 4.5a False Positive Rate (Overtrigger)

Skill was invoked but user immediately rejected or ignored it.

#### 4.5b Undertrigger Detection

**This is the highest-value dimension.** Memento-Skills (arXiv:2603.18743) demonstrates that skills stored as structured files require accurate retrieval/routing to be effective — skills that are never retrieved cannot improve through their read-write learning loop, making undertriggering a compounding problem.

For each skill, extract its **capability keywords** (not just trigger keywords — what the skill CAN do). Then scan user messages for tasks that match those capabilities but where the skill was NOT invoked.

Example: user says "run these tasks in parallel" but parallel-runner was not triggered → undertrigger.

Report: which user messages SHOULD have triggered the skill but didn't, and suggest description improvements.

**Compounding Risk Assessment:**
For skills with chronic undertriggering (0 triggers across 5+ sessions where relevant tasks appeared), flag as "compounding risk" — undertriggered skills cannot self-improve through usage feedback, causing the gap to widen over time. Recommend immediate description rewrite as P0.

#### 4.6 Cross-Skill Conflicts

Compare all skill pairs:
- Trigger keyword overlap (same keywords in two descriptions)
- Workflow overlap (two skills teach similar processes)
- Contradictory guidance

#### 4.7 Environment Consistency

For each skill, extract referenced:
- File paths → check if they exist (`test -e`)
- CLI tools → check if installed (`which`)
- Directories → check if they exist

Flag any broken references.

#### 4.8 Token Economics

**This dimension is critical and easy to skip. Do not skip it.**

For each skill:
- Word count (from Step 1)
- Trigger frequency (from 4.1)
- Cost-effectiveness = trigger count / word count
- Flag: large + never-triggered skills as candidates for removal or compression

**Progressive Disclosure Tier Check:**
Evaluate each skill against the 3-tier loading model (Agent Skills spec):
- Tier 1 (frontmatter): ~100 tokens. Check: is description ≤ 1024 chars?
- Tier 2 (SKILL.md body): <500 lines recommended. Check: word count.
- Tier 3 (reference files): loaded on demand. Check: does skill use reference files for detailed content, or cram everything into SKILL.md?

Flag skills that put 500+ words in SKILL.md without using reference files as "poor progressive disclosure".

### Step 4: Composite Score

Rate each skill on a 5-point scale:

| Score | Meaning |
|-------|---------|
| 5 | Healthy: high trigger rate, positive reactions, complete workflows, clean static |
| 4 | Good: minor issues in 1-2 dimensions |
| 3 | Needs attention: significant gap in 1 dimension or minor gaps in 3+ |
| 2 | Problematic: never triggered, or negative user reactions, or major static issues |
| 1 | Broken: doesn't work, references missing, or fundamentally misaligned |

**Scored dimensions** (weighted average):
- Trigger rate: 25%
- User reaction: 20%
- Workflow completion: 15%
- Static quality: 15%
- Undertrigger: 15%
- Token economics: 10%

**Qualitative dimensions** (reported but not scored — no reliable numeric metric):
- 4.5a Overtrigger: reported as count + examples
- 4.6 Cross-Skill Conflicts: reported as conflict pairs
- 4.7 Environment Consistency: reported as pass/fail per reference

(If a scored dimension has no data — e.g., skill was never invoked so no user reaction — mark as "N/A" and redistribute weight.)

## Report Format

```markdown
# Skill Optimization Report
**Date**: {date}
**Scope**: {all / specified skills}
**Session data**: {N} sessions, {date range}

## Overview
| Skill | Triggers | Reaction | Completion | Static | Undertrigger | Token | Score |
|-------|----------|----------|------------|--------|--------------|-------|-------|
| example-skill | 2 | 100% | 86% | B+ | 1 miss | 486w | 4/5 |

## P0 Fixes (blocking usage)
1. ...

## P1 Improvements (better experience)
1. ...

## P2 Optional Optimizations
1. ...

## Per-Skill Diagnostics
### {skill-name}
#### 4.1 Trigger Rate
...
#### 4.2 User Reaction
...
(all 8 dimensions)
```

## Research Background
The analysis dimensions in this report are grounded in the following research:
- **Undertrigger detection**: Memento-Skills (arXiv:2603.18743) — skills as structured files require accurate routing; unrouted skills cannot self-improve via the read-write learning loop
- **Description quality**: MCP Description Quality (arXiv:2602.18914) — well-written descriptions achieve 72% tool selection rate vs. 20% random baseline (3.6x improvement)
- **Information position**: Lost in the Middle (Liu et al., TACL 2024) — U-shaped LLM attention curve
- **Format impact**: He et al. (arXiv:2411.10541) — format changes alone can cause 9-40% performance variance
- **Instruction compliance**: IFEval (arXiv:2311.07911) — LLMs struggle with multi-constraint prompts
Comments 0
Latest | Hot
Please login before commenting.
No comments yet. Be the first one!
skill-optimizer

Directory Structure

SKILL.md

Report

Notice
