skill-optimizer

分类: 数据与AI | 上传者: fastxyz | 下载: 0 | 版本: v1.0（最新）

对 SDK、CLI、MCP 和提示文档进行基准测试和优化，以便每个 LLM 模型都可以使用正确的参数可靠地调用正确的操作。在为项目设置技能优化器、运行基准测试、解释结果、优化 SKILL.md 文件或诊断配置问题时使用。在技能优化器存储库本身内部工作时也可以使用 - 用于针对模拟存储库运行、测试更改或理解代码库。 --- # Skill-optimizer 针对多个 LLM 对您的 SDK/CLI/MCP/提示文档进行基准测试，衡量它们是否使用正确的参数调用正确的操作，并迭代重写您的指导，直到每个模型都达到质量底线。 ## 上下文检测在执行任何操作之前，请弄清楚您所在的位置： 1. **查找“skill-optimizer.json”**（在 CWD 或父目录中）。如果找到，则您位于**配置的目标项目**中。在下面的所有命令中使用该文件路径作为“<config-path>”。 2. **查找 `src/cli.ts` 和带有 `"name": "skill-optimizer"`** 的 `package.json`。如果找到，则您位于 **优化器存储库本身**。您可以直接使用开发命令（`npm run build`、`npm test`、`npx tsx src/cli.ts`）。要对目标进行基准测试，请使用“mock-repos/”中的模拟存储库或将“--config”指向外部项目的配置。 3. **都没有找到** - 您处于**未配置的目标项目**中。在继续之前，请阅读“references/setup.md”来构建配置。 ## 快速参考 |任务|命令 | |------|---------| |初始化配置| `npx 技能优化器初始化 cli\|sdk\|mcp\|提示` | |初始化（非交互式）| `npx 技能优化器 init cli --yes` | |导入 CLI 命令 | `npx 技能优化器导入命令 --from ./src/cli.ts` | |导入（二进制刮擦）| `npx 技能优化器导入命令 --from my-cli --scrape` | |诊断配置| `npx Skill-optimizer doctor --config <配置路径>` | |自动修复配置| `npx 技能优化医生 --fix --config <配置路径>` | |试运行（无法学硕士通话）| `npx 技能优化器运行 -

更新日志: Source: GitHub https://github.com/fastxyz/skill-optimizer

目录结构

当前层级: tree/development/SKILL/

📁 references/
- 📄 benchmark.md 5.4 KB
- 📄 config.md 8.1 KB
- 📄 optimize.md 5.3 KB
- 📄 setup.md 4.8 KB
📄 SKILL.md 4.6 KB

SKILL.md

---
name: skill-optimizer
description: >
  Benchmark and optimize SDK, CLI, MCP, and prompt documentation so every LLM
  model can reliably call the right actions with correct arguments. Use when
  setting up skill-optimizer for a project, running benchmarks, interpreting
  results, optimizing SKILL.md files, or diagnosing configuration issues. Also
  use when working inside the skill-optimizer repository itself — for running
  against mock repos, testing changes, or understanding the codebase.
---

# skill-optimizer

Benchmark your SDK / CLI / MCP / prompt docs against multiple LLMs, measure whether they call the right actions with the right arguments, and iteratively rewrite your guidance until a quality floor is met across every model.

## Context Detection

Before doing anything, figure out where you are:

1. **Look for `skill-optimizer.json`** (in CWD or parent directories). If found, you are in a **configured target project**. Use that file path as `<config-path>` in all commands below.

2. **Look for `src/cli.ts` and a `package.json` with `"name": "skill-optimizer"`**. If found, you are in the **optimizer repo itself**. You can use dev commands directly (`npm run build`, `npm test`, `npx tsx src/cli.ts`). To benchmark a target, either use the mock repos in `mock-repos/` or point `--config` at an external project's config.

3. **Neither found** — you are in an **unconfigured target project**. Read `references/setup.md` to scaffold a config before proceeding.

## Quick Reference

| Task | Command |
|------|---------|
| Init config | `npx skill-optimizer init cli\|sdk\|mcp\|prompt` |
| Init (non-interactive) | `npx skill-optimizer init cli --yes` |
| Import CLI commands | `npx skill-optimizer import-commands --from ./src/cli.ts` |
| Import (binary scrape) | `npx skill-optimizer import-commands --from my-cli --scrape` |
| Diagnose config | `npx skill-optimizer doctor --config <config-path>` |
| Auto-fix config | `npx skill-optimizer doctor --fix --config <config-path>` |
| Dry run (no LLM calls) | `npx skill-optimizer run --dry-run --config <config-path>` |
| Run benchmark | `npx skill-optimizer run --config <config-path>` |
| Generate tasks only | `npx skill-optimizer generate-tasks --config <config-path>` |
| Run optimizer | `npx skill-optimizer optimize --config <config-path>` |
| Compare two runs | `npx skill-optimizer compare --baseline a.json --current b.json` |

`<config-path>` is the path to your `skill-optimizer.json` — typically `./skill-optimizer/skill-optimizer.json` after running `init`, or wherever you placed it.

## What Do You Need?

Read the reference file that matches your current goal:

| Goal | Reference |
|------|-----------|
| Set up skill-optimizer for a project (first time) | Read `references/setup.md` |
| Run a benchmark or understand results | Read `references/benchmark.md` |
| Automatically optimize a SKILL.md | Read `references/optimize.md` |
| Understand config options | Read `references/config.md` |

If you are in an **unconfigured project** (context detection case 3), start with `references/setup.md`.

## Key Concepts

**Surfaces** — The callable interface of your project: SDK methods, CLI commands, MCP tools, or prompt templates. Skill-optimizer discovers these via tree-sitter code analysis, manifest files, or markdown parsing.

**Static evaluation** — Benchmark evaluation never executes generated code. Actions are extracted from model responses via pattern matching and compared structurally against expected calls. This makes benchmarks safe and repeatable.

**Verdict gates** — Two thresholds must both pass for a benchmark to receive a PASS verdict: `perModelFloor` (each model individually meets a minimum score) and `targetWeightedAverage` (the weighted mean across all models meets a target). A single model below the floor fails the entire run.

**Safety boundary** — The optimizer never modifies your original SKILL.md. It creates versioned copies in `.skill-optimizer/skill-v{N}.md` and only accepts mutations that improve scores without dropping any model below the floor. It does not modify tracked source files, but the generated artifacts appear under `.skill-optimizer/` — add that directory to your `.gitignore`.

**LLM routing** — By default (`format: "pi"`), all benchmark calls route through [OpenRouter](https://openrouter.ai) and need `OPENROUTER_API_KEY`. You can also call providers directly: `format: "anthropic"` uses the Anthropic API directly (`ANTHROPIC_API_KEY`), and `format: "openai"` uses the OpenAI API directly (`OPENAI_API_KEY`), with optional Codex browser-login auth via `authMode: "codex"`. The model ID prefix must match the format — see `references/config.md` for the full mapping.

登录后下载/点赞/收藏 ❤ 29 | ★ 0

skill-optimizer

目录结构

SKILL.md

举报内容

提示