autorefine

分类: 调研与分析 | 上传者: surahli123 | 下载: 0 | 版本: v1.0（最新）
使用基于评估的自动研究来迭代和提高任何技能。结合了 v2.0 设计审核、Hamel 的三湾评估方法和 Karpathy 式突变优化。当您想要评估技能质量、从头开始构建评估、运行错误分析或通过实验优化技能时使用。
更新日志: Source: GitHub https://github.com/surahli123/autorefine-skill-improvement
目录结构

当前层级: tree/main/autorefine/
📁 hooks/
- 📄 install.sh 775 B
- 📄 pre-commit 694 B
📄 dashboard.html 12.0 KB
📄 README.md 8.0 KB
📄 references.md 30.5 KB
📄 SKILL.md 25.7 KB
📄 validate-host.sh 3.4 KB
SKILL.md

---
name: autorefine
description: Iterate and improve any skill using eval-grounded autoresearch. Combines v2.0 design audit, Hamel's Three Gulfs eval methodology, and Karpathy-style mutation optimization. Use when you want to assess skill quality, build evals from scratch, run error analysis, or optimize a skill through experiments.
---

# AutoRefine

Guided skill improvement pipeline. Point at a skill: `/autorefine path/to/my-skill/`

## Preflight

1. **Target exists.** Confirm the path contains a SKILL.md. If not, ask.
2. **Detect enhancements.** Search for Hamel's `eval-audit` and `error-analysis` skills. If found, note in state.json. These enhance but are NOT required.
3. **Report tier:** Full (Hamel's detected) or Basic (core methodology only).
4. **Choose pipeline depth:**
   - **Quick** — Context-aware. Routes based on workspace state (~15-30 min). See routing below.
   - **Standard** — Full pipeline (Phases 1-7). For skills needing eval methodology from scratch. ~60-90 min.
   - **Deep** — Standard + expanded fixture set (30+ fixtures). For critical skills requiring statistical rigor.

   **Quick tier routing (3 states):**
   ```
   State 1: No workspace exists
     → Quick Start path (~30 min)
     → "First time? Let's find what your skill actually does wrong."
   State 1b: Workspace exists with schema_version 2 (legacy v2.1), no quick_start field
     → Standard/Deep only (legacy workspace — Quick Start not available)
     → "This workspace was created before Quick Start. Use Standard or Deep."
   State 2: quick_start.completed = true, both gates still "pending"
     → Quick Returning (~15 min): Run Phase 1 (design audit), then skip to Phase 7 in Mini mode. Show directional warning at start.
     → Steps: (1) Run Phase 1 as normal. (2) Skip Phases 2-6. (3) Run Phase 7 — it auto-detects Mini mode from state. (4) Run Session Close.
     → "Your evals haven't been validated — results are still directional."
   State 3: Both gulf_1 and gulf_2 = "approved" in state.json
     → Quick Returning (~15 min): Run Phase 1 (design audit), then skip to Phase 7 in Full mode.
     → Steps: (1) Run Phase 1 as normal. (2) Skip Phases 2-6. (3) Run Phase 7 — it auto-detects Full mode from state. (4) Run Session Close.
   ```

   If workspace has approved gates: offer Quick as default. If quick_start_complete: offer Quick with directional note. Otherwise default to Standard (offer Quick Start as faster alternative).

## Initialize Workspace

If `autoresearch-<skill>/` doesn't exist: create it with `traces/` and `judges/` subdirectories. Generate these files (see `references.md > Workspace Schemas` for exact formats):
- `state.json` — pipeline state (schema_version:4 for new workspaces — see `references.md > Workspace Schemas`)
- `results.json` — experiment results for dashboard
- `results.tsv` — append-only experiment log
- `session-log.json` — per-session audit trail
- `changelog.md`, `eval-suite.md`, `error-analysis-traces.md` — empty, formatted in later phases
- Copy `dashboard.html` from this skill's directory, replace `{{SKILL_NAME}}`

If workspace exists **with** `state.json`: read it and print pipeline status. **Check for checkpoint:** if `state.json.checkpoint` is not null and has `next_action`, enter resume mode — read all files in `checkpoint.files_to_read_on_resume` (skip any missing files and note which were missing), print "Resuming from checkpoint: {next_action}", clear the checkpoint (set to null), and proceed from `next_action`. See `references.md > Checkpoint Schema > Resume Detection`. Rotate `session-log.json` (rename to `session-log-<session_start, colons→dashes>.json`, create fresh). If `session-log.json` missing (pre-v2 workspace), create it. Legacy workspaces (schema_version 2 or 3) are read-compatible — checkpoint fields default to null.

If workspace exists **without** `state.json`: back up to `autoresearch-<skill>-prev/` and create fresh.

## Pipeline Status

Print at every session start:
```
AutoRefine: <name>
================================================================
Quick Start                        [STATUS]
Gulf 1: Comprehension
  Phase 1: Design Audit          [STATUS]
  Phase 2: Eval Audit             [STATUS]
  Phase 3: Error Analysis         [STATUS]  [N/M traces]
  >>> Gate: Approve taxonomy      [STATUS] <<<
Gulf 2: Specification
  Phase 4: Expand Inputs           [STATUS]  [N fixtures]
  Phase 5: Write Judges            [STATUS]  [N code / N judge]
  Phase 6: Validate Judges         [STATUS]  [TPR/TNR]
  >>> Gate: Approve judges         [STATUS] <<<
Gulf 3: Generalization
  Phase 7: AutoResearch Loop      [STATUS]  [best score]
================================================================
> Gulf 1 builds the scorer. Gulf 3 uses the scorer.
> Skip Gulf 1 and you optimize against a fantasy.
```
STATUS values: `not started`, `in progress`, `complete`, `skipped`. Read from `state.json.phases`.

---

## Quick Start Path

Read when: Quick Start path active (State 1 from Preflight routing).

> **Preview mode.** Quick Start gives you a taste of autorefine in ~30 min. It does NOT validate Gulf 1 or Gulf 2 — bootstrap evals are directional, not calibrated. Run Standard to get validated results.

### QS Step 1: Phase 1 Design Audit (5 min)
Run Phase 1 as normal (see below). No changes.

*Why this step:* "This tells us what your skill *should* do based on best practices. But the real failures might be different — that's why Step 2 exists."

### QS Step 2: Mini Phase 3 — Observation (10 min)
Generate 5 inputs targeting the skill. Use Phase 1 findings as **focus areas** (not literal inputs — structural gaps like "missing gotchas" guide what scenarios to test, not what text to send). Sort findings by priority, take top 5 as focus areas, generate one diverse input per focus area that exercises the skill in a way where that gap would matter. If fewer than 5 gaps, fill remaining slots from the diversity spread (see `references.md > Quick Start > Mini Phase 3 Template`). For interactive or session-spanning skills, create synthetic output fixtures instead (same fallback as Standard Phase 3 Step 1).

Run the skill on each input (read the target SKILL.md and provide the input as if you were a user requesting the skill — capture the full output). If the skill errors on an input, record the trace as a Fail with note "skill execution error: [error message]" and continue to the next input. Present outputs one at a time for user judgment:
```
--- Trace 1/5 ---
Input: [summary]
Output: [skill output]
Pass or Fail? (one-line note if Fail)
```

Append to session-log.json: `{"phase":"quick_start","step":"observation","type":"mini_observation","detail":"Reviewed 5 traces: N pass, M fail"}`.

*Why this step:* "You're reading actual outputs — not imagining what could go wrong. Evals built from observation catch real failures. Evals built from imagination catch hypothetical ones."

**Graceful exit:** If Phase 1 found ≤1 issue AND all 5 traces pass: set `quick_start.completed = true` and `quick_start.graceful_exit = true` in state.json (prevents re-entry into Quick Start on next run). Tell the user: "Your skill looks clean on this sample. Quick Start needs failure signal to work. Try Standard for a deeper look, or skip autorefine." Stop here.

### QS Step 3: Bootstrap Eval Generator (3 min)
Auto-generate lightweight evals from Phase 1 findings (→ structural checks) + Mini Phase 3 failures (→ behavioral checks). Write each eval using the `references.md > Eval Suite Template` format (`EVAL N: [Name]...`). See `references.md > Quick Start > Bootstrap Eval Generator` for conversion rules and the simplified zero-shot judge template.

**Eval types:** Code-based where possible (deterministic). Agent-as-judge only for subjective criteria — use the zero-shot bootstrap template (NO few-shot examples, NO train split).

**Write judge files:** For each agent-as-judge eval, write the judge prompt to `judges/judge-E{N}-bootstrap-{name}.md` using the Bootstrap Judge Template from `references.md > Quick Start > Bootstrap Eval Generator`. For each code-based eval, write to `judges/code-E{N}-bootstrap-{name}.sh`. Phase 7 reads from `judges/` — if the files don't exist there, scoring will fail.

**Minimum floor:** 3 evals. If Phase 1 + Mini Phase 3 yield fewer, supplement with additional Phase 1 pattern checks.

**Labeling:** Tag every bootstrap eval in eval-suite.md with per-eval metadata: `Source: quick_start | Validated: false | Confidence: directional`. Apply this tag to EACH eval entry individually, not as a suite-level header.

Present all evals in a numbered list. User responds: "approved", "drop N", "change N to [description]", or "add [new eval]". Single interaction, not a gate.

*Why this step:* "These evals are rough — think of them as a first-draft relevance model. Useful for direction, not for production metrics. Run Standard to validate with TPR/TNR."

### QS Step 4: Mini Phase 7 — Targeted Mutations (10 min)
Generate 5-10 **fresh** scoring inputs — NOT the same as Mini Phase 3 traces (reusing = overfitting). Use the diversity spread from `references.md > Quick Start > Mini Phase 3 Template`, but target different Phase 1 gaps or different complexity levels than QS Step 2. Ensure zero overlap with the 5 QS Step 2 inputs. Save scoring inputs to `traces/qs-scoring-S01.md` through `traces/qs-scoring-S10.md`.

Run 2-3 mutations targeting the top Phase 1 + Mini Phase 3 findings. Score with bootstrap evals using simplified weighting: code evals = 1.0, agent-as-judge = 0.5 (flat discount, not empirical TPR/TNR).

Present each mutation to the user (diff, score change, proposed keep/discard). User confirms or overrides. Record in results.json + session-log.json.

**Time note:** Estimates assume skill execution < 30 sec per run. Slow skills may push total to ~35 min.

*Why this step:* "You're seeing your skill get better. But bootstrap evals are rough — a passing mutation might miss subtle regressions. Standard mode builds evals you can trust."

### QS Step 5: Results + Handoff (2 min)
Show before/after comparison. All results labeled "directional improvement, not validated."

**State update:** In state.json, set `quick_start.completed = true` with metadata (traces, evals, mutations, timestamp). Keep `gates.gulf_1` and `gates.gulf_2` as `"pending"`. Set `current_phase: 0` (integer — 0 means Quick Start complete; phases 1-7 use integers 1-7). Set `phases.design_audit: "complete"` (Phase 1 was run). Keep `schema_version` at 4 (do NOT downgrade).

**Handoff:** "Your skill is better. Here's what Standard gives you: validated evals with TPR/TNR, a full failure taxonomy, and confidence-weighted optimization. Everything you just built carries forward — Standard extends this workspace."

**Standard transition rules:** When Standard runs on a Quick Start workspace:
- Phase 1: Re-run (skill may have changed from mutations). Overwrite `design-audit.md`.
- Phase 2: Acknowledge existing bootstrap evals in eval-suite.md. Audit them alongside any other eval infrastructure.
- Phase 3: Start fresh with full 20+ traces. Reference QS mini observations as prior context but don't skip reviews. Overwrite `error-analysis-traces.md`.
- Phase 5: Keep bootstrap evals that align with the new failure taxonomy. Discard the rest. Write validated judges to replace bootstrap judges in `judges/`.
- Phase 6: Validate all judges (including promoted bootstrap ones). Update per-eval metadata: `Source: standard | Validated: true`.

---

## Phase 1: Design Audit

Read the target SKILL.md. Score 4 dimensions: **Gotchas**, **Voice**, **Progressive Disclosure**, **Scripts** (if any). For each Partial/Missing: quote the problem, recommend a fix, assign priority.

**Gotchas dimension — 3-stage detection:**
1. **Taxonomy scan:** Check skill against 6 gotcha categories (shell execution, path handling, state mutation, concurrent access, auth/secrets, external APIs). Skills touching NONE → score "N/A." Full category list: `references.md > Gotcha Taxonomy`.
2. **Static evidence:** For each matched category, cite the specific line(s) creating the risk. Example: "Line 47: `$B goto $URL` — no URL sanitization."
3. **Smoke probe:** For the top 2 highest-risk findings, construct a minimal test input and run the target skill with it. Narrate before each probe. Record as `confirmed` or `not_confirmed`. For session-spanning skills, skip probes and record as `suspected`. Details: `references.md > Gotcha Taxonomy > Smoke Probe Instructions`.

Remaining dimensions: `references.md > V2.0 Design Audit Rubric`.

**Output:** `design-audit.md` (Gotchas section includes taxonomy, evidence, confidence levels, and probe results). Append to session-log.json: `{"phase":"1","type":"design_audit","detail":"Scored 4 dims: Gotchas=X (N categories matched, M confirmed), Voice=X, Disclosure=X, Scripts=X"}`. Phase 3 inherits `suspected` gotcha items as targeting input. **State:** advance to Phase 2.

---

## Phase 2: Eval Audit

Check for existing eval infrastructure (eval-suite.md, evals.json, test fixtures, results files). Audit against 6 categories: error analysis grounding, evaluator design, judge validation, train/test split, labeled data count, maintenance process. If no evals exist, document: "No eval infrastructure. Phase 3 builds the foundation."

Detail on each category: `references.md > Eval Audit Categories`.

**Output:** `eval-audit-report.md`. **State:** advance to Phase 3.

---

## Phase 3: Error Analysis

Close the Gulf of Comprehension. **Most important phase. CANNOT BE AUTOMATED.**

**Step 1: Prepare fixtures.** If fewer than 15 diverse inputs exist, generate more covering different types, lengths, quality levels, edge cases, at least 2 with planted flaws. For session-spanning skills (tracing, monitoring), create synthetic output fixtures instead.

**Step 2: Run the skill.** Invoke on each fixture (15-25). Save outputs to `traces/trace-T01.md` through `traces/trace-T25.md`.

**Step 3: Smart sampling.** Select 8-10 traces using stratified sampling. On first run, infer dimensions from fixture properties (input length, source, planted flaw). On re-runs, use Phase 4 dimensions. Ensure every dimension value represented at least once. Show the user which traces were selected and why. Append to session-log.json: `{"phase":"3","type":"sampling","detail":"Selected N/M traces..."}`. Methodology: `references.md > Smart Sampling Methodology`.

**Step 4: Preliminary clustering.** Assign each sampled trace a category ID (C1, C2, C3...) by surface patterns (adapt to skill type). Target 3-5 clusters. If <3, skip consistency checks. Write clusters to `error-analysis-traces.md` header.

**Step 5: Human reviews.** Present sampled traces in batches of 5 using the worksheet format. For each trace, the user provides: Pass or Fail, and a note if Fail. Record in `error-analysis-traces.md` (columns: #, Fixture, Cluster, Pass/Fail, Notes). User can stop after ≥5 traces. Format details: `references.md > Batch Review Format`.

**Consistency check (after ≥5 reviews):** If same-cluster traces got different verdicts, flag it. Append: `{"phase":"3","type":"consistency_flag","detail":"T03 and T07 match C2, judged differently"}`. If user confirms both verdicts, log resolution.

**Mid-phase resume:** Track `traces_reviewed` and `sampled_trace_ids` in state.json.

**Step 6: Build failure taxonomy.** Cluster failure notes into categories — let them EMERGE. If <3 failures in sample, review additional traces. Write `failure-taxonomy.md`.

**Step 7: Generate eval suite.** Convert top failures into binary evals. Write `eval-suite.md`. Format: `references.md > Eval Suite Template`.

### Gate: Gulf 1 Exit
Generate `gate-report-gulf-1.md` with: sample stats, fail rate, categories, consistency flags, proposed evals. Append to session-log.json: `{"phase":"gate_1","type":"gate_decision","detail":"APPROVED"}` (or REJECTED). **Override logging:** if user removes evals or rejects categories, also append: `{"phase":"gate_1","type":"override","detail":"Removed E4","reason":"..."}`

**STOP. Wait for user approval.**

**State:** Mark Phase 3 complete. Record traces_reviewed, sampling_strategy, taxonomy.

---

## Phase 4: Expand Inputs

**Step 1:** Count labeled fixtures from Phase 3.
**Step 2:** Define 3 failure-prone dimensions from the taxonomy. Format: `references.md > Dimension Template`.
**Step 3:** Generate fixtures via dimension tuples. Draft 15-20 with user, LLM generates 10 more. Target 30-40 total.
**Step 4:** Split into train (~15%) / dev (~42%) / test (~43%). Train = few-shot examples only. Dev = iteration. Test = final measurement, never peek. Write `fixtures-manifest.md`.

**Output:** `fixtures-manifest.md`. **State:** advance to Phase 5.

---

## Phase 5: Write Judges

**Step 1: Classify evals** as code-based (counting, regex, field presence → bash/python) or agent-as-judge (semantic judgment → judge prompt file). Exhaust code options first. Write `eval-classification.md`.

**Step 2: Build code-based evaluators.** One-liner or short script per eval. Test on 3 dev fixtures.

**Step 3: Build agent-as-judge prompts.** Each judge has 4 components: task+criterion, Pass/Fail definitions, 3 few-shot examples (TRAIN split only — never dev/test), critique-before-verdict output format. The coding agent itself IS the judge — no external API needed. **Anti-rigidity rule:** Score on outcome achievement, not path matching. A judge that fails outputs for using different structure or wording than the reference is penalizing creativity, not catching errors. Full template: `references.md > Judge Prompt Template`.

**Step 4: Write to workspace.** Save to `judges/judge-E{N}-{name}.md` and `judges/code-E{N}-{name}.sh`.

**Output:** `eval-classification.md` + `judges/` directory. **State:** advance to Phase 6.

---

## Phase 6: Validate Judges

Calibrate agent-as-judge evaluators against human labels. Code-based evals skip (deterministic).

**Steps 1-4 (dev split — human reviews):** Run each judge on dev split. Present judge verdicts vs human labels using batch review format (`references.md > Batch Review Format`). Compute TPR and TNR. Inspect disagreements — False Pass → strengthen Fail defs; False Fail → clarify Pass defs. Iterate until stable. Formulas: `references.md > TPR/TNR Reference`.

**Step 5 (test split — automated, NO human review):** Final measurement on test split. Run once, record, do NOT iterate. The human does NOT see test examples — this preserves the holdout. Write `judge-validation-report.md`.

**Step 6: Generate judge confidence cards.** For each agent-as-judge eval, generate a confidence card showing TPR/TNR with interpretation, evidence examples, and known blind spots. Template: `references.md > Judge Confidence Card Template`. Walk the human through each card with narration. If TPR-TNR gap > 20 points, flag the asymmetry explicitly.

### Gate: Gulf 2 Exit
Generate `gate-report-gulf-2.md` with: classification, TPR/TNR per judge, code eval results. Append to session-log.json: `{"phase":"gate_2","type":"gate_decision","detail":"APPROVED"}` (or REJECTED). **Override logging:** if user rejects judges, also append: `{"phase":"gate_2","type":"override","detail":"...","reason":"..."}`

**STOP. Wait for user approval.**

**State:** Mark Phase 6 complete.

---

## Phase 7: AutoResearch Loop

The Karpathy-style mutation-test-keep/discard cycle. Requires `eval-suite.md` + judges.

**Mode detection (check in this order):**
1. If both `gates.gulf_1` = "approved" AND `gates.gulf_2` = "approved" → **Full mode** (validated evals, confidence-weighted scoring). Budget: ask user — Quick (3), Standard (5), Deep (8-10).
2. Else if `quick_start.completed` = true AND both gates = "pending" → **Mini mode** (bootstrap evals, simplified weighting, directional labeling). Budget: 2-3 experiments.
3. Else → STOP and tell the user to run Standard pipeline first.

**Mini mode differences:** Fresh scoring corpus (5-10 generated inputs, NOT reusing Quick Start traces). Simplified weighting: code evals = 1.0, agent-as-judge = 0.5. All results labeled "directional." No confidence weighting from Phase 6 (evals aren't validated).

**The Loop:**
1. Run baseline on all fixtures, score against all evals → Experiment 0. Record `eval_results` per eval.
2. LOOP:
   a. Analyze failures → hypothesize ONE change → **save backup** (`<skill>-optimized-prev.md`) → mutate a copy
   b. Score mutation against all evals → record `eval_results`
   c. **Regression check:** Compare current `eval_results` against prior kept experiments. If any eval that previously passed now fails → regression detected. Details: `references.md > Regression Check Schema`. Skip on experiments 0 and 1 (baseline has no prior kept experiments to compare against).
   d. **Present to user** — show mutation diff, score change, regression status, proposed keep/discard. One decision point.
   e. User accepts or overrides → record in results.json (with `eval_results` + `regression_check`) + results.tsv + changelog.md
   f. If discarded (regression or user choice): restore backup as current baseline
3. Repeat until all evals pass or budget exhausted

**Key rules:** One mutation per experiment. Mutate a copy (`<skill>-optimized.md`), not the original. If score improves, the mutated copy becomes the new baseline for the next experiment. Target baseline 60-80% (>90% = evals too easy). **Mutation types:** add, modify, OR delete. After every 2-3 additive mutations, try one subtractive mutation — remove instructions and measure if performance improves. Shorter skills often outperform bloated ones. Formats: `references.md > Results & Changelog Schemas`.

**Confidence-weighted scoring:** Weight each eval by its judge's validated TPR/TNR. Code evals = 1.0. Agent evals = (TPR+TNR)/2. Experiment score = weighted sum / sum of weights.

**User verdict confirmation:** After each experiment, present the score change, regression status, and proposed keep/discard to the user. If the user overrides (e.g., keeps despite regression, or discards despite clean score), log as `type: "judge_gap"` in session-log.json: `{"phase":"7","type":"judge_gap","experiment":N,"agent_verdict":"keep","user_verdict":"discard","reason":"..."}`. These indicate judge blind spots and feed the loop-back prompt. Regression overrides also log: `{"phase":"7","type":"regression",...,"user_action":"keep_override"}`.

**Eval dimension pruning:** After the loop completes, check results.json for evals that passed 100% across ALL experiments (baseline + mutations). Flag them: "E{N} passed every experiment. This eval may no longer discriminate — consider whether the model has outgrown it or the criterion is too lenient."

**Mutation history:** Record discarded experiments with the same detail as kept ones — full `changes[]` diff, `eval_results`, and `regression_check`. Discarded mutations have diagnostic value: patterns in what fails reveal what the skill actually needs.

**State:** Update after each experiment (include `eval_results` and `regression_check`). **Dashboard:** serve workspace with `python3 -m http.server 8080`.

---

## Loop-Back Prompt

After Phase 7, if session-log has ≥2 `judge_gap` entries:

> "You overrode N experiment verdicts, suggesting judge blind spots:
> - [reason 1]
> - [reason 2]
> Loop back to Phase 5 to refine judges, then re-run Phase 7? Or accept current results?"

If user loops back: Phase 5 enters append mode (`locked_judges` prevents modifying approved judges). Phase 6 validates only new judges. Phase 7 re-runs with expanded suite. Score may drop — this is more accurate measurement, not regression. Explain this to the user. Max 2 loop-backs per session. Increment `loop_iteration` in state.json.

---

## Session Close

Runs after Phase 7, when user stops mid-pipeline, or when user explicitly pauses. Minimum: session-log must have ≥3 entries for learning summary.

1. **Save checkpoint** — update `state.json.checkpoint` with current state and write `resume-prompt.txt` to workspace root. See `references.md > Checkpoint Schema`. Append to session-log: `{"type":"checkpoint",...}`.
2. **Synthesize** session-log.json into 3-5 bullet learning summary (what worked, what was overridden, patterns emerged)
3. **User curates** — present summary, ask for edits or approval
4. **Persist** to agent memory system (path from `state.json.memory_path`, or ask on first run and record). If no memory system exists, write to `autoresearch-<skill>/learnings.md` as fallback.
5. **Present resume prompt** — "You can resume anytime. Paste this into a new session:" followed by the resume prompt from `resume-prompt.txt`.
6. **Archive** — rename session-log.json to `session-log-<timestamp>.json`

If <3 entries: still save checkpoint (step 1), skip learning summary. Say: "Not enough data for a learning summary yet. Checkpoint saved — you can resume later."

**Phase boundary checkpoints:** At every phase completion (not just Session Close), update `state.json.checkpoint` and write `resume-prompt.txt`. This is automatic — no user interaction needed. Log to session-log.

---

## Gotchas

See `references.md > Gotchas` for the full list. Critical ones:

1. **Don't skip Gulf 1.** A 100% score on narrow evals is an artifact, not evidence.
2. **Error analysis cannot be automated.** Phase 3 requires the human to read outputs.
3. **session-log.json is best-effort.** If corrupted or missing, recreate and continue. Never blocks.
4. **Never run two sessions on same skill.** state.json has no locking.
5. **"Invoke" means "read and follow."** Not all agents support direct skill invocation.
6. **Quick Start is a preview, not validation.** Bootstrap evals are directional, not calibrated. Quick Start does NOT satisfy Gulf 1 or Gulf 2. Run Standard to validate results.
7. **Critical state lives in files, not conversation.** Keep mutation rationale, eval scores, and pipeline state in workspace files (state.json, results.json). Auto-compact at ~85% context erases conversation history — only the skill file (loaded via system prompt) survives.

---

## References

`references.md` — Templates, schemas, methodology rationale, detailed rubrics, gotchas.
登录后下载/点赞/收藏 ❤ 5 | ★ 0
autorefine

目录结构

SKILL.md

举报内容

提示
