autorefine

Category: Research & Analysis | Uploader: surahli123 | Downloads: 0 | Version: v1.0（Latest）
Iterate and improve any skill using eval-grounded autoresearch. Combines v2.0 design audit, Hamel's Three Gulfs eval methodology, and Karpathy-style mutation optimization. Use when you want to assess skill quality, build evals from scratch, run error analysis, or optimize a skill through experiments.
Changelog: Source: GitHub https://github.com/surahli123/autorefine-skill-improvement
Directory Structure

Current level: tree/main/autorefine/
📁 lib/
- 📄 __init__.py 10.1 KB
- 📄 _projected_pattern_extractor.py 39.1 KB
- 📄 _shared_text.py 719 B
- 📄 exemplar_source_loader.py 60.1 KB
- 📄 heuristic_extractor.py 3.2 KB
- 📄 meta_learnings_loader.py 41.1 KB
- 📄 mutation_candidate_relevance.py 23.2 KB
- 📄 research_corpus.py 204.5 KB
- 📄 search_family_regression.py 16.9 KB
- 📄 source_parser_adapters.py 21.8 KB
- 📄 structure_extractor.py 3.2 KB
- 📄 style_preferences_loader.py 13.6 KB
- 📄 tactic_extractor.py 3.1 KB
- 📄 target_skill_scoring_context.py 9.5 KB
📁 references/
- 📄 gulf1-comprehension.md 50.7 KB
- 📄 gulf2-specification.md 20.0 KB
- 📄 gulf3-generalization.md 93.5 KB
📁 scripts/
- 📄 build-search-silver.py 10.3 KB
- 📄 eval-search-family-regression.py 7.0 KB
- 📄 eval-search-metric.py 9.4 KB
- 📄 prepare-gulf-gate-pack.py 14.4 KB
- 📄 record.py 39.4 KB
- 📄 records-to-gulf1.py 18.7 KB
- 📄 run-campaign.py 47.0 KB
📁 tests/
- 📄 test_adapter_gulf_wiring.sh 2.8 KB
- 📄 test_build_search_silver_py.py 3.9 KB
- 📄 test_campaign_orchestrator_py.py 13.6 KB
- 📄 test_domain_adapter_schema.sh 2.6 KB
- 📄 test_env_fact_reviewer_gate.sh 2.0 KB
- 📄 test_eval_search_metric_py.py 4.7 KB
- 📄 test_exemplar_source_loader.py 8.8 KB
- 📄 test_experiment_contract_schema.sh 2.1 KB
- 📄 test_external_reference_collection_contract.sh 6.7 KB
- 📄 test_holdout_isolation_contract.sh 2.1 KB
- 📄 test_holdout_optimization_metrics.sh 4.5 KB
- 📄 test_input_set_identity.sh 6.2 KB
- 📄 test_iteration_directory.sh 11.7 KB
- 📄 test_iteration_run_initial_state.sh 4.4 KB
- 📄 test_iteration_run_record.sh 3.6 KB
- 📄 test_pattern_eval_strategy_selector.sh 5.0 KB
- 📄 test_phase7_skip_outcome.sh 2.2 KB
- 📄 test_preflight.sh 24.5 KB
- 📄 test_prepare_gulf_gate_pack_py.py 6.3 KB
- 📄 test_record_py.py 50.5 KB
- 📄 test_records_to_gulf1_py.py 23.0 KB
- 📄 test_research_intake_contract.sh 9.9 KB
- 📄 test_search_adapter_contract.sh 2.0 KB
- 📄 test_search_family_regression_py.py 16.5 KB
- 📄 test_skill_version_artifact.sh 4.5 KB
- 📄 test_trust_gate_contract.sh 2.3 KB
📄 dashboard.html 525.9 KB
📄 meta-learnings.md 3.1 KB
📄 README.md 2.4 KB
📄 references.md 418.3 KB
📄 SKILL.md 30.2 KB
📄 validate-host.sh 3.6 KB
SKILL.md

---
name: autorefine
description: Iterate and improve any skill using eval-grounded autoresearch. Combines v2.0 design audit, Hamel's Three Gulfs eval methodology, and Karpathy-style mutation optimization. Use when you want to assess skill quality, build evals from scratch, run error analysis, or optimize a skill through experiments.
---

# AutoRefine

Guided skill improvement pipeline. Point at a skill: `/autorefine path/to/my-skill/`

## Preflight

### Step 0: Environment Check (MANDATORY — runs first, < 15 seconds)

Fast-fail checks. If ANY fail, STOP immediately with an actionable error message. Do NOT retry or explore alternatives silently.

1. **Target skill path.** If user provided a path, use it. If not, ask: "Which skill should I improve? Provide the full path to the skill directory."
2. **Target readable.** Run: `head -5 [skill-path]/SKILL.md`. If this fails → STOP: "I can't read your skill at [skill-path]. If you're in a sandboxed environment, copy your skill into my working directory first: `cp -r [skill-path] ./skill-under-test/` then re-invoke with `./skill-under-test/`"
3. **Workspace location.** Ask the user (ONE question, wait for answer):
   ```
   Where should I create the AutoRefine workspace?
     a) /tmp/autorefine-[skill-name]/  <- recommended (safe, no repo interference)
     b) Next to your skill: [skill-parent]/autorefine-[skill-name]/
     c) Custom path
   ```
   Default to (a) if user says "whatever" or "default." NEVER create the workspace without this confirmation.
4. **Workspace writable.** Run: `mkdir -p [chosen-workspace] && touch [chosen-workspace]/.preflight-test && rm [chosen-workspace]/.preflight-test`. If this fails → STOP: "I can't write to [chosen-workspace]. Try option (a) /tmp/ which is always writable, or specify a different path."
5. **Skill import.** Copy the entire skill directory into the workspace: `cp -r [skill-path]/ [chosen-workspace]/skill-under-test/`. All subsequent reads and writes operate ONLY on `[workspace]/skill-under-test/`, never on the original skill path. This protects the user's real skill from accidental modification.
6. **Persist paths.** Record in state.json: `original_skill_path: [skill-path]`, `workspace_path: [chosen-workspace]`. These are needed for Session Close (Apply Back gate) and session resume.

After Step 0 completes, print:
```
Preflight passed
  Target skill: [skill-path]/SKILL.md
  Workspace: [chosen-workspace]/
  Working copy: [chosen-workspace]/skill-under-test/SKILL.md
  Original path saved for apply-back: [skill-path]
  Original skill is UNTOUCHED until you approve changes.
```

### Step 1: Detect & Configure

1. **Detect enhancements.** Search for Hamel's `eval-audit` and `error-analysis` skills. If found, note in state.json. These enhance but are NOT required.
2. **Report tier:** Full (Hamel's detected) or Basic (core methodology only).
3. **Choose pipeline depth:**
   - **Quick** — Context-aware. Routes based on workspace state (~15-30 min). See routing below.
   - **Standard** — Full pipeline (Phases 1-7). For skills needing eval methodology from scratch. ~60-90 min.
   - **Deep** — Standard + expanded fixture set (30+ fixtures). For critical skills requiring statistical rigor.

   **Quick tier routing (3 states):**
   ```
   State 1: No workspace exists
     -> Quick Start path (~30 min)
     -> "First time? Let's find what your skill actually does wrong."
   State 1b: Workspace exists with schema_version 2 (legacy v2.1), no quick_start field
     -> Standard/Deep only (legacy workspace -- Quick Start not available)
     -> "This workspace was created before Quick Start. Use Standard or Deep."
   State 2: quick_start.completed = true, both gates still "pending"
     -> Quick Returning (~15 min): Run Phase 1 (design audit), then skip to Phase 7 in Mini mode. Show directional warning at start.
     -> Steps: (1) Run Phase 1 as normal. (2) Skip Phases 2-6. (3) Run Phase 7 -- it auto-detects Mini mode from state. (4) Run Session Close.
     -> "Your evals haven't been validated -- results are still directional."
   State 3: Both gulf_1 and gulf_2 = "approved" in state.json
     -> Quick Returning (~15 min): Run Phase 1 (design audit), then skip to Phase 7 in Full mode.
     -> Steps: (1) Run Phase 1 as normal. (2) Skip Phases 2-6. (3) Run Phase 7 -- it auto-detects Full mode from state. (4) Run Session Close.
   ```

   If workspace has approved gates: offer Quick as default. If quick_start_complete: offer Quick with directional note. Otherwise default to Standard (offer Quick Start as faster alternative).

## Initialize Workspace

**Workspace path** was confirmed in Preflight Step 0. The workspace is at `[workspace]/` and the working copy of the skill is at `[workspace]/skill-under-test/`. (After Preflight, `[workspace]` = the path chosen in Step 0.3 and persisted in `state.json.workspace_path`.)

If workspace `traces/`, `judges/`, `runs/`, `skill-versions/`, `contract/`, and `domain-eval/` subdirectories don't exist: create them. Generate these files (see `references.md > Workspace Schemas` for exact formats):
- `state.json` — pipeline state (schema_version:4 for new workspaces — see `references.md > Workspace Schemas`)
- `results.json` — experiment results for dashboard
- `results.tsv` — append-only experiment log
- `session-log.json` — per-session audit trail
- `changelog.md`, `eval-suite.md`, `error-analysis-traces.md` — empty, formatted in later phases
- Copy `dashboard.html` from this skill's directory, replace `{{SKILL_NAME}}`

If workspace exists **with** `state.json`: read it, deserialize any persisted `phase1_context` (including `selected_skill_pattern` and `selected_eval_strategy_id`) plus any persisted `mutation_stage_split_access_policy`, `iteration_state`, `mid_session_preference_signals`, and `mid_session_preference_signals_path`, `selected_adapter_id`, `adapter_config_path`, and `active_experiment_contract_path` into the loaded run context, then normalize the active-loop `style_preferences` payload using `references.md > Style Preferences Payload` before printing pipeline status.
If workspace exists **with** `state.json`: read it, deserialize any persisted `phase1_context` into the loaded run context, and print pipeline status.
If workspace exists **with** `state.json`: read it, deserialize any persisted `phase1_context` (including `selected_skill_pattern` and `selected_eval_strategy_id`) plus any persisted `mutation_stage_split_access_policy` into the loaded run context, and print pipeline status.
If workspace exists **with** `state.json`: read it, deserialize any persisted `phase1_context` (including `selected_skill_pattern` and `selected_eval_strategy_id`) plus any persisted `mutation_stage_split_access_policy` and `iteration_state` into the loaded run context, and print pipeline status.

**Step A: Checkpoint recovery (runs FIRST on resume).**
If `state.json.checkpoint` is not null and has `next_action`, enter resume mode — read all files in `checkpoint.files_to_read_on_resume` (skip any missing files and note which were missing), deserialize `state.json.phase1_context`, `state.json.mutation_stage_split_access_policy`, `state.json.iteration_state`, `state.json.mid_session_preference_signals`, and `state.json.mid_session_preference_signals_path` into the loaded run context before routing the resume path, then rebuild the normalized `style_preferences` payload from `references.md > Style Preferences Payload`, and print "Resuming from checkpoint: {next_action}". Also deserialize `state.json.contract_status`, `state.json.contract_path`, `state.json.effectiveness_floor`, `state.json.selected_adapter_id`, `state.json.adapter_config_path`, `state.json.active_experiment_contract_path`, and `state.json.domain_eval_config_path` into the loaded run context before routing. These fields control Phase 0.5 entry, adapter-aware Phase 4-7 routing, run-scoped experiment contracts, and Session Close contract effectiveness reporting. Prefer `state.json.adapter_config_path` as the canonical runtime adapter pointer. If it is null and `state.json.domain_eval_config_path` is populated, hydrate `adapter_config_path` from the legacy alias before downstream routing. If `contract_status = "confirmed"`, validate the full contract artifact set before trusting downstream state. Check all 4 required files exist and are non-empty:
- `[workspace]/contract/success-examples.jsonl` (must exist, must have >= 3 valid JSONL rows each with required fields: `id`, `input`, `output_shape.description`, `actual_output`)
- `[workspace]/contract/failure-examples.jsonl` (>= 3 valid rows, required fields: `id`, `input`, `output_shape.description`, `actual_output`, `failure_reason`)
- `[workspace]/contract/do-not-trigger-examples.jsonl` (>= 3 valid rows, required fields: `id`, `input`, `expected_behavior`)
- `[workspace]/contract/inferred-contract.md` (must exist, must contain the 8 standard sections from `Inferred Contract Template`)

Also validate the canonical adapter config path if set (`state.json.adapter_config_path`, or the legacy alias `state.json.domain_eval_config_path` when `adapter_config_path` is null) — FULL file integrity check, not just config existence:
- `[workspace]/domain-eval/config.json` must exist, parse as valid JSON, and match the required fields in `Domain Eval Config Schema` (`domain_eval_version`, `metric_name`, `threshold_pass`, `threshold_concern`, `weight_multiplier`, `eval_script_path`, `author_confirmed = true`)
- The file at `config.json.eval_script_path` must exist and be readable. Do not attempt to execute it; just verify readability.
- The golden-set file at `config.json.golden_set_path` is OPTIONAL at checkpoint time (may be filled in before Phase 7). If present, verify it parses as valid JSONL and has >= 1 row with required fields (`id`, `input`, `expected_output`).

On ANY validation failure, stop with a blocking error naming the specific missing/malformed file(s):
- "Adapter config integrity check failed. `adapter_config_path` (or legacy `domain_eval_config_path`) is set but the following are missing or malformed: [list]. Recovery options: (a) restore the files, (b) re-run Phase 0.5 Step 7 to reconfigure (set both `adapter_config_path` and `domain_eval_config_path` to null in `state.json` and re-enter Phase 0.5 — the contract examples remain; only adapter config re-prompts), or (c) clear both config fields to null in `state.json` to proceed without adapter-aware evaluation (Phase 5 and Phase 7 will skip domain-metric scoring — graceful degradation to LLM-judge-only)."

Fail closed — do NOT silently clear adapter config fields to null. The user must explicitly choose.

Log `{"phase":"checkpoint","type":"domain_eval_integrity_check","status":"passed|failed","files_validated":2}` to session-log.json.

On ANY validation failure, stop with a blocking error naming the specific missing/malformed file(s):
- "Contract integrity check failed. `contract_status` is `"confirmed"` but the following files are missing or malformed: [list]. Recovery options: (a) restore the files from backup, (b) re-run Phase 0.5 contract wizard (set `contract_status` to `"not_started"` in `state.json` and re-enter), or (c) set `contract_status` to `"skipped"` in `state.json` to proceed without contract examples (downstream phases will use no-contract fallbacks)."

Fail closed — do NOT silently reset `contract_status` to `"skipped"` or continue with partial data. The user must explicitly choose a recovery path.

If all validations pass, continue with normal downstream routing. Log `{"phase":"checkpoint","type":"contract_integrity_check","status":"passed","files_validated":4}` to session-log.json on success. Restore `phase1_context.selected_skill_pattern` and `phase1_context.selected_eval_strategy_id` unchanged so later phases can read the chosen pattern + resolved downstream strategy from the loaded context rather than recomputing them. If the restored run-context pattern and `state.json.skill_pattern` mismatch, stop and rerun Phase 1 Step 0 instead of continuing. If the restored `selected_eval_strategy_id` is missing or no longer maps back to the restored pattern through `references.md > Skill Pattern Eval Strategy > Pattern-to-Evaluation-Strategy Selector`, stop and rerun strategy selection before continuing. If split-scoped Phase 7 work is active and the restored `mutation_stage_split_access_policy` is missing, read the same policy from `fixtures-manifest.md` or a stored Phase 4 `evaluation_metadata.config.mutation_stage_split_access_policy` snapshot, hydrate the loaded run context, and stop if the sources disagree. If `iteration_state` is present, treat it as the authoritative Phase 7 handoff record for whether the active `run_id` is in eval, mutate, test, or session_close; continue automatic progression from the persisted `next_action` until terminal success (`phase_status = "completed"`) or terminal failure (`phase_status = "blocked"`) without requiring manual phase handoff. Do not infer boundaries from directory scans while the persisted runner state is available. Then clear the checkpoint (set to null) while preserving every other serialized state field, including `phase1_context`, `mutation_stage_split_access_policy`, `iteration_state`, `mid_session_preference_signals`, and `mid_session_preference_signals_path`. See `references.md > Checkpoint Schema > Resume Detection`. Rotate `session-log.json` (rename to `session-log-<session_start, colons->dashes>.json`, create fresh). If `session-log.json` missing (pre-v2 workspace), create it. Legacy workspaces (schema_version 2 or 3) are read-compatible — checkpoint fields default to null. If checkpoint has `next_action` pointing to a Phase 7 experiment, **skip ambient learning entirely** (workspace copy must match the in-progress experiment state) and proceed from `next_action`.
Deserialize `state.json.phase1_context`, `state.json.mutation_stage_split_access_policy`, `state.json.iteration_state`, `state.json.selected_adapter_id`, `state.json.adapter_config_path`, and `state.json.active_experiment_contract_path` into the loaded run context before routing or resuming later phases.
deserialize `state.json.phase1_context`, `state.json.mutation_stage_split_access_policy`, `state.json.selected_adapter_id`, `state.json.adapter_config_path`, and `state.json.active_experiment_contract_path` into the loaded run context before routing the resume path.
deserialize `state.json.phase1_context`, `state.json.mutation_stage_split_access_policy`, `state.json.iteration_state`, `state.json.selected_adapter_id`, `state.json.adapter_config_path`, and `state.json.active_experiment_contract_path` into the loaded run context.
Restore `phase1_context.selected_skill_pattern` unchanged so later phases can read the chosen pattern from the loaded context rather than recomputing it.
Then clear the checkpoint (set to null) while preserving every other serialized state field, including `phase1_context`.

**Pattern-aware downstream entry:** When routing into any downstream phase or stage after Phase 1, initialize pattern-aware logic from the loaded `state.json.phase1_context.selected_skill_pattern`. Then restore `state.json.phase1_context.selected_eval_strategy_id` into that same loaded run context and treat the pair as the active downstream routing state for the current run. If the active context does not already hold them, read the same canonical IDs from the top-level `selected_skill_pattern` and `selected_eval_strategy_id` fields emitted in `design-audit.md`, hydrate the loaded run context, and continue. If the active context does not already hold it, read the same canonical ID from the top-level `selected_skill_pattern` field emitted in `design-audit.md`, hydrate the loaded run context, and continue. If only the pattern is available, resolve the missing strategy through `references.md > Skill Pattern Eval Strategy > Pattern-to-Evaluation-Strategy Selector` before continuing. After hydrating `selected_eval_strategy_id`, immediately open the matching row in `references.md > Skill Pattern Eval Strategy > Strategy Definitions` and route all downstream eval work through that strategy bundle. The selected strategy is the execution path for Quick Start bootstrap evals, Phase 2 eval audit, Phase 3/4 failure clustering and fixture expansion, Phase 5/6 judge design, and Phase 7 mutation analysis; do not fall back to the generic downstream path while a valid selector is present. Do not trigger Phase 1 pattern classification again during downstream phase/stage initialization; rerun Phase 1 Step 0 only if the persisted pattern is missing or inconsistent with `state.json.skill_pattern`.

**Adapter-aware downstream entry:** When routing into any downstream phase or stage after Phase 1, restore `state.json.selected_adapter_id`, `state.json.adapter_config_path`, and `state.json.active_experiment_contract_path` into the loaded run context if they are present. Pattern classification may suggest an adapter, but do not treat a suggestion as active until `selected_adapter_id` is explicitly set in state. Prefer `adapter_config_path` as the canonical runtime config pointer. If it is null and `domain_eval_config_path` exists, hydrate `adapter_config_path` from the legacy alias and continue. If `selected_adapter_id` is present but the canonical config path is missing or unreadable, stop and ask the user to either restore the adapter assets or explicitly downgrade to the LLM-judge-only path. Do not silently downgrade from an active adapter-aware run. If `active_experiment_contract_path` is present, treat that artifact as the authoritative run-scoped success contract for Phase 7 mutate/evaluate handoffs instead of reconstructing success criteria from conversation history.

**Split-aware downstream entry:** When routing into Phase 7 or Session Close after Phase 4, initialize split-aware logic from the loaded `state.json.mutation_stage_split_access_policy`. If the active context does not already hold it, read the exact policy from `fixtures-manifest.md` or a stored Phase 4 `evaluation_metadata.config.mutation_stage_split_access_policy` snapshot, hydrate the loaded run context, and stop if those sources disagree. Treat the restored object as the active gate for any downstream step that may read split-scoped datasets, per-input outputs, or version-comparison joins. Canonicalize any caller-provided split token before the gate check; if the raw token, an alias, or a delegated resolution path lands on `adversarial_holdout`, reject the read. Once active, route every Phase 7 split-scoped read through `references.md > Restricted Mutation-Stage Dataset Access Path` instead of reopening fixtures, scored inputs, or comparison payloads directly from the mutation loop. Resolve intermediate scoring splits through that accessor and the active policy only; do not branch on raw split IDs or bypass the accessor to recover the dev corpus. If `requested_operation = mutation_scoring` resolves to `adversarial_holdout`, explicitly deny the request and fail closed before reopening any stored dev corpus. The final-only evaluation stage is not part of this Phase 7 read path: do not trigger holdout validation while the mutation loop is still iterating. Trigger it only once after the loop reaches a terminal exit for the active `state.json.current_run_path`. At Session Close, reuse the existing variant-evaluation interface to score the completed version lineage on the holdout split instead of inventing a second holdout-only scorer. Do not start a Phase 7 dataset read until that policy is active.

**Meta-learnings bootstrap context:** If the session may enter Phase 7 or Session Close, bootstrap the meta-learnings context into the loaded run context before any mutation steering or resume-time cross-campaign reasoning. Resolve `state.json.meta_learnings_path` (or the default AutoRefine skill-directory copy), normalize the current target context (`skill_pattern`, `agent_target`, `scenario_target`, `scope_type`, `scope_ref`), then hydrate `{meta_learnings_path, target_context, parsed_meta_learnings}` using `references.md > Campaign Bootstrap Meta-Learnings Context`. Rebuild this object on every start/resume instead of persisting parsed entries in `state.json`. When a run output or report payload is serialized, preserve the same bootstrap envelope's reporting fields — `curator_source`, `curator_version`, `transfer_parameters`, and `transfer_traceability` — unchanged so downstream filters and exports can replay the same curation lineage.

**Style-preferences payload:** If the session may continue iterating inside Phase 7 or Session Close, rebuild the normalized `style_preferences` payload from `state.json.mid_session_preference_signals` plus `state.json.mid_session_preference_signals_path` using `references.md > Style Preferences Payload`, then keep that envelope in the loaded run context across eval, mutate, test, and session_close. Mid-loop stages should read `style_preferences.active_signals` as the machine-readable preference set and `style_preferences.resolved_preferences_path` only when they need the human-readable `[workspace]/preferences.md` wording; do not rescan raw override sources once the hydrated payload is available.

**Step B: Ambient learning (runs AFTER checkpoint recovery, only if NOT resuming mid-Phase-7).**

Guard: `state.json.original_skill_path` must exist and be readable. If unreadable (sandbox, deleted), skip ambient learning silently and continue.

1. Run `diff [original-skill-path]/SKILL.md [workspace]/skill-under-test/SKILL.md`. If the `diff` command fails (sandbox restriction), skip ambient learning and continue.
2. If no diff → skill unchanged. Continue.
3. If diff exists → size gate:
   - **Small diff (<=20 lines changed):** likely preference signal. Proceed to step 4.
   - **Large diff (>20 lines, <=50% of file):** warn: "Large diff detected (N lines). Treat as preference signal or new baseline?" If user says baseline → skip to step 5.
   - **Rewrite (>50% of file):** skip rule extraction. Log `{"type":"ambient_learning","skipped":true,"reason":"full_rewrite","diff_size":N}`. Go to step 5.
4. **Extract preference rules.** Show the diff to the user. Ask: "Should I learn from these edits? (y/n)". If yes, extract rules using this format:
   ```
   RULE: [one-sentence preference]
   EVIDENCE: [quote removed text] -> [quote added text] (max 2 lines each)
   CONFIDENCE: high (clear intent) | medium (inferred) | low (ambiguous)
   ```
   Only auto-log `high` and `medium` rules. Present `low` rules for user confirmation. Distinguish preference edits from bug fixes (if the user fixed a typo or corrected a factual error, that's a fix, not a preference — skip it). Log to `[workspace]/preferences.md` (separate from `learnings.md` used by Session Close) and session-log: `{"type":"ambient_learning","rules_extracted":N,"diff_size":N}`.
5. **Sync workspace copy.** Always update: `cp [original-skill-path]/SKILL.md [workspace]/skill-under-test/SKILL.md`. This ensures the next mutation cycle starts from the user's current version.

If workspace exists **without** `state.json`: back up the workspace to `[workspace]-prev/` and create a fresh workspace at `[workspace]/`.

## Pipeline Status

Print at every session start:
```
AutoRefine: <name>
================================================================
Contract                           [STATUS]
Effectiveness Floor                [STATUS]  [N pass / N concern / N fail]
Quick Start                        [STATUS]
Gulf 1: Comprehension
  Phase 1: Design Audit          [STATUS]
  Phase 2: Eval Audit             [STATUS]
  Phase 3: Error Analysis         [STATUS]  [N/M traces]
  >>> Gate: Approve taxonomy      [STATUS] <<<
Gulf 2: Specification
  Phase 4: Expand Inputs           [STATUS]  [N fixtures]
  Phase 5: Write Judges            [STATUS]  [N code / N judge]
  Phase 6: Validate Judges         [STATUS]  [TPR/TNR]
  >>> Gate: Approve judges         [STATUS] <<<
Gulf 3: Generalization
  Phase 7: AutoResearch Loop      [STATUS]  [best score]
================================================================
> Contract examples anchor your evals. Skip it to use generic eval generation.
> Gulf 1 builds the scorer. Gulf 3 uses the scorer.
> Skip Gulf 1 and you optimize against a fantasy.
```
STATUS values: `not started`, `in progress`, `complete`, `skipped`. Read from `state.json.phases`. Contract status read from `state.json.contract_status`; Effectiveness Floor status read from `state.json.effectiveness_floor.overall_status` (or `not started` if null).

---

## Gulf Routing

After Initialize Workspace and Pipeline Status, stay in `SKILL.md` as the single skill entrypoint/router and read the appropriate support file from `references/` based on pipeline state. **Only read the support file relevant to the current phase. Do NOT preload all support files.**

| Current Phase | Read | Contains |
|---------------|------|----------|
| Phase 0.5 (Contract) | `references/gulf1-comprehension.md` | Phase 0.5 Contract Collection Wizard |
| Quick Start, Phase 1-3 | `references/gulf1-comprehension.md` | Quick Start Path, Phase 1 (Design Audit + Pattern Classification), Phase 2 (Eval Audit), Phase 3 (Error Analysis), Gulf 1 Gate |
| Phase 4-6 | `references/gulf2-specification.md` | Phase 4 (Expand Inputs), Phase 5 (Write Judges + Eval Category Tags), Phase 6 (Validate Judges), Gulf 2 Gate |
| Phase 7, Session Close | `references/gulf3-generalization.md` | Phase 7 (AutoResearch Loop + Verdict Explanation Cards + Aggregation Explainer + Version Registry), Loop-Back Prompt, Session Close (+ Version Comparison) |

**Routing rules:**
- Starting fresh with no contract (state.json.contract_status is null or "not_started") → read `references/gulf1-comprehension.md` for Phase 0.5 first, then Phase 1
- Contract already confirmed (state.json.contract_status = "confirmed") → skip Phase 0.5, read `references/gulf1-comprehension.md` for Phase 1
- Contract skipped (state.json.contract_status = "skipped") → skip Phase 0.5, read `references/gulf1-comprehension.md` for Phase 1
- Contract in-progress (state.json.contract_status is "collecting" or "inferred") → resume Phase 0.5 mid-wizard, read `references/gulf1-comprehension.md`
- **Phase 0.5 trace recorder input:** `autorefine/scripts/record.py` + `autorefine/scripts/records-to-gulf1.py` provide an alternative Phase 0.5 ingestion path. Use when you have recorded real agent traces (Option D in Phase 0.5).
- Starting fresh or resuming in Phases 1-3 → read `references/gulf1-comprehension.md`
- Gulf 1 gate approved, entering Phases 4-6 → read `references/gulf2-specification.md`
- Gulf 2 gate approved (or Quick Start returning) → read `references/gulf3-generalization.md`
- **Quick Start returning (State 2 or 3):** read `references/gulf1-comprehension.md` for Phase 1, then read `references/gulf3-generalization.md` for Phase 7 + Session Close (two-file read)
- Loop-back from Phase 7 to Phase 5 → re-read `references/gulf2-specification.md`

---

## Phase 7

Phase 7 writes the Aggregation Explainer and persists `decision_breakdown`, including `components`, `weight_source`, `normalized_contribution`, `combined_score`, and the keep/discard threshold. As soon as scoring completes for the experiment, populate `decision_breakdown` with the intermediate per-eval math before user presentation.
As soon as scoring completes for the experiment, populate `decision_breakdown` with the intermediate per-eval math and the final aggregate used for the keep/discard threshold.
Every stored verdict must preserve a structured `evidence` array with `kind, source, locator` fields covering input excerpts, output excerpts, metrics, or artifact references. Phase 7 must validate that `evidence[]` contains at least one concrete evidence item; otherwise flag the verdict as invalid and reject it from scoring/storage.
Every material sub-decision must also preserve `supporting_items` with `evidence_refs` so the reader can inspect each individual decision instead of only the final Pass/Fail. Attach the extracted `supporting_items[]` to the same verdict object, do not carry supporting items forward to another eval, and preserve them in the final judge output written to `eval_results[]` and iteration `eval_results.json`.

---

## Gotchas

Critical rules (full list in `references.md > Gotchas`):

1. **Don't skip Gulf 1.** A 100% score on narrow evals is an artifact, not evidence.
2. **Error analysis cannot be automated.** Phase 3 requires the human to read outputs.
3. **session-log.json is best-effort.** If corrupted or missing, recreate and continue. Never blocks.
4. **Never run two sessions on same skill.** state.json has no locking.
5. **"Invoke" means "read and follow."** Not all agents support direct skill invocation.
6. **Quick Start is a preview, not validation.** Bootstrap evals are directional, not calibrated.
7. **Critical state lives in files, not conversation.** Auto-compact at ~85% context erases conversation history.
8. **Never write to the original skill path during the pipeline.** All work on `[workspace]/skill-under-test/`.
9. **Sandbox environments block cross-directory access.** Use `/tmp/` as workspace workaround.
10. **Agent-as-judge evals should not see mutation reasoning.** Tier 1 (subagent) = real isolation. Tier 2 (behavioral) = heuristic only.
11. **Workspace location must be confirmed, never assumed.** Never create without asking.

---

## References

`references.md` — Templates, schemas, methodology rationale, detailed rubrics, gotchas.
Read `references.md > Version Comparison Alignment` before surfacing version diffs or any per-input comparison payload.
Version diffs must run the comparison preflight first, require the exact same set of stable `input_id`s, surface `missing_from_left`, `missing_from_right`, `extra_in_left`, or `extra_in_right` on mismatch, mark that state as `invalid-comparison`, and must not emit normal comparison results when preflight fails.
After the preflight passes, surface the per-input comparison payload together with a shared-input outcome summary that reports trusted `improved` / `regressed` counts plus `unreliable` and `unchanged` counts across the joined `input_id` set.
Read `references.md > Judge Verdict Evidence Schema` when Phase 7 stores verdicts: preserve a structured `evidence` array with required `kind, source, locator` fields for input excerpts, output excerpts, metrics, or artifact references.
Phase 7 evidence storage uses a structured `evidence` array. Required fields: `kind, source, locator`. Supported evidence types: input excerpts, output excerpts, metrics, or artifact references.
Read `references.md > Discard Autopsy Heuristics` when Phase 7 discards an experiment so the discard autopsy classification is written back to `results.json`.
Comments 0
Latest | Hot
Please login before commenting.
Loading comments...
autorefine

Directory Structure

SKILL.md

Report

Notice
