audit-evals

分类: 工具与效率 | 上传者: satyaborg | 下载: 0 | 版本: v1.0（最新）

为代理代码库设置评估或在更改后检查评估状态。确定准备情况，确定可以测试的内容并准备环境。在第一次启动评估、代码更改后返回或弄清楚下一步要做什么时使用。还可以通过“设置评估”、“我的代理准备好了吗？”、“评估状态”、“下一步应该做什么？”、“初始化评估”、“评估我的代理”、“测试我的代理”、“帮助我评估这个”、“开始评估”、“我从哪里开始”、“如何测试这个代理”、“检查我的设置”来触发。这是默认的入口点，每当用户想要评估代理并且您不确定从哪项技能开始时就可以使用它。

更新日志: Source: GitHub https://github.com/satyaborg/kensa

目录结构

当前层级: tree/main/skills/audit-evals/

📁 scripts/
- 📄 check_instrumentation.py 4.2 KB
- 📄 check_library.py 3.7 KB
📄 SKILL.md 6.1 KB

SKILL.md

---
name: audit-evals
description: >
  Set up evals for an agent codebase or check eval status after changes. Determines
  readiness, identifies what can be tested, and prepares the environment. Use when
  starting evals for the first time, returning after a code change, or figuring out
  what to do next. Also triggered by "set up evals", "is my agent ready?",
  "eval status", "what should I do next?", "init eval", "evaluate my agent",
  "test my agent", "help me eval this", "get started with evals", "where do I
  start", "how do I test this agent", "check my setup". This is the default entry
  point, use it whenever a user wants to evaluate an agent and you're unsure which
  skill to start with.
compatibility: Requires Python 3.10+, uv.
---

# Audit Eval

Set up and assess an agent codebase for kensa.

## Lifecycle

```
► Setup → Design → Execute → Diagnose → Iterate
```

This is the entry point, both for first-time setup and for returning users who need to figure out where they are. After this skill, the next step is `generate-scenarios` (if no scenarios exist) or `kensa eval` (if scenarios are ready).

## Check kensa is installed

```bash
python ${CLAUDE_SKILL_DIR}/scripts/check_library.py
```

If missing, ask the user before installing:
> "kensa is not installed. Add it as a project dependency?"

If approved:

```bash
python ${CLAUDE_SKILL_DIR}/scripts/check_library.py --install
```

## Determine current state

Use Glob to check what exists:
1. `Glob(".kensa/")`: does the directory exist at all?
2. `Glob(".kensa/scenarios/*.yaml")`: scenario files?
3. `Glob(".kensa/agents/*")`: agent entry points?
4. `Glob(".kensa/traces/*.jsonl")`: trace files?
5. `Glob(".kensa/results/*.json")`: result files from previous runs?
6. `Glob(".kensa/judges/*.yaml")`: judge prompt files?

Recommend a single next action based on findings:

- **No `.kensa/` directory**: Fresh start. Run `kensa init --blank` to scaffold the directory structure without example files. Then scan the codebase (below) and hand off to `generate-scenarios` to create real scenarios directly.
- **`.kensa/` exists but empty** (no scenarios, no traces): Blank scaffold from `kensa init --blank` or manual setup. Scan the codebase (below) and hand off to `generate-scenarios`.
- **Has `.kensa/` with only example scenario**: Init was run without `--blank`. Scan the codebase (below) to understand the real agent, then hand off to `generate-scenarios` to replace the example.
- **No scenarios, has traces**: Traces exist from manual runs. Run `kensa analyze` to surface patterns, then hand off to `generate-scenarios` with the analysis as context.
- **Has scenarios, no traces**: Scenarios are ready but never run. Recommend: `kensa eval`.
- **Has scenarios, has traces and results**: Full loop is active. Check both data stores:
  1. Run `kensa report --format json` and count scenarios with `status: "fail"`, `"error"`, or `"uncertain"`. This catches check failures and judge rejections that leave no trace anomaly.
  2. Run `kensa analyze` to check for trace-level anomalies (cost/latency outliers, errors, looping).
  If either surface has failures, recommend `diagnose-errors`. If both are clean, recommend `kensa eval`.
- **Has scenarios, has traces but no results**: Traces exist but judge hasn't run. Recommend: `kensa judge`.

State the recommendation clearly. Mention alternatives only if genuinely relevant.

## Scan codebase

Identify these five things. Use `Agent` with `subagent_type: "Explore"` for thorough scanning on larger codebases, or `Read` and `Grep` directly on small ones.

1. **Entry point**: how to run the agent
2. **LLM SDK**: anthropic, openai, langchain, etc.
3. **Tools**: functions the agent can call
4. **Behaviors**: from system prompts, docstrings, README
5. **Env vars**: required configuration

Record as working memory:
```
Entry point: src/agent.py
Run command: [uv, run, python, src/agent.py]
SDK: anthropic
Tools: [get_weather, search_web, calculate]
Behaviors: Answers user questions using web search and weather data
Env vars: ANTHROPIC_API_KEY
```

## Assess readiness

| Signal | Readiness | What you can deliver |
|--------|-----------|---------------------|
| Clear entry point, typed inputs, tool defs, docstrings | High | Strong baseline scenarios with targeted checks |
| Ambiguous entry point, no docstrings, implicit config | Medium | Generic smoke tests + recommendations |
| No clear agent boundary, hardcoded keys | Low | Minimal value from code. Push toward traces. |
| Custom HTTP clients, vendored SDKs, non-Python | Instrumentation risk | Flag early. May need manual instrumentation. |

Tell the user what you found honestly:
> "I scanned your codebase. Here's what I can infer and what I can't. Scenario quality will be limited by [X]. To improve: [concrete action]."

## Verify instrumentation

Instrumentation is automatic. The runner injects `sitecustomize.py` via `PYTHONPATH` before the agent runs. No code changes needed in the agent.

Ensure the matching SDK extra is installed:

| SDK | Install |
|-----|---------|
| `anthropic` | `uv add kensa[anthropic]` |
| `openai` | `uv add kensa[openai]` |
| `langchain` | `uv add kensa[langchain]` |

Verify with:

```bash
kensa doctor
```

If `kensa doctor` passes, instrumentation is ready. If it flags issues, fix them before proceeding.

## Next step

Hand off to `generate-scenarios` with the codebase scan context (entry point, tools, behaviors, env vars). The user's next action is to design test scenarios.

## Gotchas

- `instrument()` is idempotent. Agents that still have `from kensa import instrument; instrument()` work fine (no duplicate spans).
- `instrument()` warns when no SDK instrumentors are installed. Install the matching extra.
- `.env` is auto-loaded by the runner (walks up from cwd). Never read or print `.env` yourself: it holds secrets. Use `kensa doctor` to verify which env vars are set.
- Judge model resolution: `KENSA_JUDGE_MODEL` env var > `ANTHROPIC_API_KEY` (claude-sonnet-4-6) > `OPENAI_API_KEY` (gpt-5.4-mini) > error.
- For non-Python commands or `python -S`/`-I`, the escape hatch is `from kensa import instrument; instrument()` in the agent code.

登录后下载/点赞/收藏 ❤ 5 | ★ 0

audit-evals

目录结构

SKILL.md

举报内容

提示