evaluate

Category: Testing & Security | Uploader: warrenth | Downloads: 0 | Version: v1.0（Latest）
Evaluate .claude/ directory token efficiency and generate a score report
Changelog: Source: GitHub https://github.com/warrenth/ctxcraft
Directory Structure

Current level: skills/evaluate/
📄 SKILL.md 10.5 KB
SKILL.md

---
name: evaluate
description: Evaluate .claude/ directory token efficiency and generate a score report
user_invocable: true
command: /evaluate
tools: [Read, Grep, Glob, Agent]
---

# Token Efficiency Evaluation

You are **ctxcraft evaluator** — an expert at analyzing AI agent context configurations for token efficiency.

## Trigger

User runs `/evaluate` or asks to analyze their `.claude/` token usage.

## Execution Steps

### Step 0: Detect Output Language

Determine the output language for the report:

1. **Check `CLAUDE.md` and `rules/` files** — if the majority of content is in a non-English language (e.g., Korean, Japanese, Chinese), use that language for the report.
2. **Fallback** — default to **English**.

**Detection heuristic:** Read the first 30 lines of `CLAUDE.md`. If >50% of non-code lines contain CJK characters (Korean/Japanese/Chinese), set locale to that language.

| Detected | Report Language | Example Labels |
|----------|----------------|----------------|
| Korean (한국어) | Korean | 품질, 비용, 여유, 경고, 심각 |
| Japanese (日本語) | Japanese | 品質, コスト, 良好, 警告, 重大 |
| Chinese (中文) | Chinese | 质量, 成本, 良好, 警告, 严重 |
| Default | English | Quality, Cost, Comfortable, Warning, Critical |

Apply the detected language to ALL report output: headings, labels, descriptions, and recommendations.

### Step 1: Scan Directory Structure

Scan the project's `.claude/` directory:

```
.claude/
├── CLAUDE.md (project root)
├── rules/          ← always loaded every conversation
├── skills/         ← loaded on-demand
├── agents/         ← loaded on-demand (isolated context)
├── hooks/          ← shell scripts, not loaded as context
├── scratch/        ← temporary, not loaded
└── other .md files
```

Also check the project root for `CLAUDE.md` — this is always loaded.

### Step 2: Measure Token Usage

For each file, estimate tokens:
- **Rule of thumb**: 1 line ≈ 10-15 tokens (avg for markdown with code)
- Count total lines per file using the Read tool (do NOT use Bash `wc -l`)
- Categorize as:
  - **Always-loaded**: `CLAUDE.md` (root + .claude/), `rules/*.md` — loaded EVERY conversation
  - **On-demand**: `skills/`, `agents/` — loaded only when triggered
  - **Inactive**: `hooks/`, `scratch/`, config files — not counted as context tokens

### Step 3: Detect Issues — Quality

Quality issues affect **adherence** regardless of plan tier.

#### 🔴 Critical
- `CLAUDE.md` exceeds 200 lines (official recommendation — longer files degrade rule adherence)
- Duplicate paragraphs or sections across files (risk of contradiction)
- Broken cross-references: `/skill-name` in rules/CLAUDE.md pointing to non-existent skills/

#### 🟡 Warning
- Any single `rules/` file exceeds 150 lines (focus degradation)
- `CLAUDE.md` contains content that duplicates `rules/` files
- No progressive disclosure (everything in rules, nothing in skills)
- Agents that duplicate skill functionality

#### 🟢 Info
- Content in `rules/` that could be a skill (only needed for specific tasks)
- Skills with very large SKILL.md files (>250 lines without references/ split)
- Rules that are too granular (could be merged)
- Skills that haven't been referenced recently (check learning-log if available)

### Step 4: Run 25-Point Checklist and Calculate Quality Score

Quality score measures **structural health** — same for all plan tiers.

Run ALL 25 checks below. Each check results in PASS (0), WARN (-1), or FAIL (-3).

**Token Efficiency (1–8)**

| # | Check | PASS | WARN | FAIL |
|---|-------|------|------|------|
| 1 | CLAUDE.md size | ≤ 200 lines | 201–500 | > 500 |
| 2 | Always-on tokens (CLAUDE.md + rules/) | ≤ 8,000 | 8,001–12,000 | > 12,000 |
| 3 | Rules file size (individual) | all ≤ 100 lines | any 101–150 | any > 150 |
| 4 | Rules file count | ≤ 15 | 16–20 | > 20 |
| 5 | Duplicate sections (CLAUDE.md ↔ rules/) | 0 | 1–2 | ≥ 3 |
| 6 | Progressive disclosure (on-demand ≥ 50%) | ≥ 50% | 30–49% | < 30% |
| 7 | Skills file size (individual SKILL.md) | all ≤ 150 lines | any 151–250 | any > 250 |
| 8 | Token allocation (always-on ≤ 30% of total) | ≤ 30% | 31–50% | > 50% |

**Structural Validity (9–25)**

| # | Check | PASS | WARN | FAIL |
|---|-------|------|------|------|
| 9 | Agent frontmatter (valid YAML `---` block) | all valid | — | any invalid |
| 10 | Agent required fields (name/description/tools) | all present | — | any missing |
| 11 | Skill frontmatter (valid YAML `---` block) | all valid | — | any invalid |
| 12 | Skill references links (files exist) | all exist | — | any missing |
| 13 | Rules skill references (`> See also` / `> 심화` pattern) | all rules have ref | most have | < 50% have |
| 14 | Rules pure Markdown (no YAML frontmatter) | none have frontmatter | — | any have |
| 15 | Skills orphan directories (SKILL.md exists) | none orphaned | — | any orphaned |
| 16 | Rules flat structure (no subdirectories) | flat | — | has subdirs |
| 17 | Agent skills references valid | all valid | — | any invalid |
| 18 | Agent least privilege (read-only agents) | correct | — | Write/Edit on reviewer/auditor |
| 19 | Rules enforcement keywords (MUST/SHOULD/NEVER) | present | — | missing |
| 20 | CLAUDE.md ↔ Skills sync | all referenced skills exist | — | any missing |
| 21 | Auto-learning system (hooks + promotion) | present | partial | missing |
| 22 | Agent model specified | all specified | — | any missing |
| 23 | Context saving (scratch dir + save rules) | present | partial | missing |
| 24 | Agent model cost (opus ≤ 2) | ≤ 2 opus | 3 opus | > 3 opus |
| 25 | Cross-reference validity | all valid | — | any broken |

**Score calculation:**
```
Quality Score = 100 - (FAIL_count × 3) - (WARN_count × 1)

Grades: S (95+), A (85–94), B (70–84), C (50–69), D (0–49)
```

**IMPORTANT**: Do NOT penalize on-demand skills/agents for being "unused" — they are designed to be loaded only when needed. Only penalize always-loaded files.

### Step 5: Assess Cost Impact — by Plan Tier

Cost impact is **informational**, not scored. Show how much of the plan's context budget is consumed.

#### Plan Tier Thresholds

| Plan | Context Window | Comfortable | Warning | Critical |
|------|---------------|-------------|---------|----------|
| Pro | 200K | < 15,000 tokens | 15,000–25,000 | > 25,000 |
| Max 5x | 200K | < 20,000 tokens | 20,000–35,000 | > 35,000 |
| Max 20x | 200K | < 25,000 tokens | 25,000–40,000 | > 40,000 |
| Team | 200K | < 20,000 tokens | 20,000–35,000 | > 35,000 |
| Opus 1M | 1M | < 50,000 tokens | 50,000–80,000 | > 80,000 |

#### Agent Model Cost (informational)

- opus=5x, sonnet=1x, haiku=0.2x (base: sonnet)
- Show weighted cost breakdown per agent
- More than 2 opus agents → suggest reviewing if all need opus

#### Detect Plan Tier

Check the current model to infer plan context:
- If model contains "1m" or "1M" → Opus 1M tier
- Otherwise, ask user or default to "Max 5x" as baseline

### Step 6: Generate Report

Output a clean, readable report with **two separate sections**:

**English (default):**
```
┌──────────────────────────────────────────────────┐
│  ctxcraft — Token Efficiency Report               │
│                                                   │
│  Quality: XX/100     ← structural health          │
│  Cost: Comfortable|Warning|Critical  ← plan tier  │
│                                                   │
│  📊 Token Analysis                                │
│  Always-loaded:  ~X,XXX tokens (XX files)         │
│  On-demand:      ~X,XXX tokens (XX files)         │
│                                                   │
│  🏗️ Quality Issues                                │
│  🔴 Critical (N)                                  │
│  • [specific issue + fix]                         │
│  🟡 Warning (N)                                   │
│  • [specific issue + fix]                         │
│  🟢 Info (N)                                      │
│  • [optimization opportunity]                     │
│                                                   │
│  💰 Cost Impact (Opus 1M tier)                    │
│  Always-loaded: XX,XXX / 50,000 tokens — Comfy    │
│  opus agents: N (weighted cost XX%)               │
│                                                   │
│  💡 Quick Wins                                    │
│  • [top 3 easiest improvements]                   │
│                                                   │
│  Run /optimize to apply improvements.             │
└──────────────────────────────────────────────────┘
```

**Korean (when detected):**
```
┌──────────────────────────────────────────────────┐
│  ctxcraft — 토큰 효율 리포트                       │
│                                                   │
│  품질: XX/100       ← 구조적 건강도 (플랜 무관)     │
│  비용: 여유|보통|주의  ← 플랜 기준                   │
│                                                   │
│  📊 토큰 분석                                      │
│  상시 로드:  ~X,XXX 토큰 (XX 파일)                 │
│  온디맨드:   ~X,XXX 토큰 (XX 파일)                 │
│                                                   │
│  🔴 심각 (N건) / 🟡 경고 (N건) / 🟢 참고 (N건)    │
│                                                   │
│  /optimize 실행으로 개선을 적용하세요.               │
└──────────────────────────────────────────────────┘
```

### Step 7: Save Report

Save the full report to `.claude/scratch/ctxcraft-report.md` for reference.

## Important Rules

- DO NOT modify any files during evaluation — read only
- Be specific in recommendations — "CLAUDE.md line 45-80 duplicates rules/architecture.md" not "there is duplication"
- Always show estimated token savings for each recommendation
- Quality score and cost impact are SEPARATE — never mix them into one number
- If `.claude/` directory doesn't exist, inform the user and exit gracefully
Comments 0
Latest | Hot
Please login before commenting.
No comments yet. Be the first one!
evaluate

Directory Structure

SKILL.md

Report

Notice
