eval

分类: 工具与效率 | 上传者: agentevals-dev | 下载: 0 | 版本: v1.0（最新）

根据黄金参考对座席行为进行评估和评分。每当用户想要运行评估、检查通过/失败状态、了解指标分数、比较会话的回归、验证代理行为或对文件或实时会话的跟踪进行评分时，请使用此技能。触发诸如“评估此跟踪”、“检查我的代理输出”、“我的代理是否做了正确的事情”、“比较运行”、“我的代理回归了吗”、“对会话 X 进行评分”、“针对黄金进行评估”、“运行评估”等短语。适用于本地跟踪文件和实时流会话。 --- 评估代理行为并解释分数的含义。 ## 确定输入类型首先，弄清楚要评估的内容： - **跟踪文件** — 用户提到 `.json` 或 `.jsonl` 文件路径 → 使用 `evaluate_traces` - **Sessions 与 Golden** — 用户有多个实时会话并希望进行回归测试 → 使用 `evaluate_sessions` - **单个实时会话** — 用户想要针对黄金评估集对一个会话进行评分 → 指导他们将 `evaluate_sessions` 与一个会话一起使用Golden ## 评估跟踪文件 1. 获取文件路径。检查扩展名： `.jsonl` → `trace_format: "otlp-json"` | `.json` → `"jaeger-json"`（默认） 2. 询问他们是否有黄金评估集 JSON。对于“tool_trajectory_avg_score”（默认指标），需要一个评估集 - 它提供了要进行比较的预期工具调用序列。如果他们还没有，请解释这一点并建议从“hallucinations_v1”开始，或者询问他们是否想首先从参考运行中创建一个黄金集。 3. 使用文件、格式和评估集调用“evaluate_traces”。 4. 以分数表形式呈现结果（请参阅下面的分数解释）并解释失败的原因。 ## 评估会话（回归测试）此工作流程要求服务器使用“--dev”标志运行（启用 WebSocket 和会话流）。普通的“agentevasserve”不会有会话。如果您从下面的任何工具中收到连接错误，请告诉用户：``bash uv run agentevalsserve --dev``

更新日志: Source: GitHub https://github.com/agentevals-dev/agentevals

目录结构

当前层级: tree/main/.claude/skills/eval/

📁 evals/
- 📄 evals.json 3.2 KB
📄 SKILL.md 3.6 KB

SKILL.md

---
name: eval
description: >
  Evaluate and score agent behavior against a golden reference. Use this skill
  whenever the user wants to run evaluation, check pass/fail status, understand
  metric scores, compare sessions for regressions, validate agent behavior, or
  score a trace from a file or a live session. Trigger on phrases like "eval this
  trace", "check my agent output", "did my agent do the right thing", "compare runs",
  "did my agent regress", "score session X", "evaluate against golden", "run evals".
  Works with both local trace files and live streaming sessions.
---

Evaluate agent behavior and explain what the scores mean.

## Determine the input type

First, figure out what to evaluate:
- **Trace file(s)** — user mentions a `.json` or `.jsonl` file path → use `evaluate_traces`
- **Sessions vs golden** — user has multiple live sessions and wants regression testing → use `evaluate_sessions`
- **Single live session** — user wants to score one session against a golden eval set → guide them to use `evaluate_sessions` with one session as golden

## Evaluating trace files

1. Get the file path(s). Check the extension:
   `.jsonl` → `trace_format: "otlp-json"` | `.json` → `"jaeger-json"` (default)

2. Ask if they have a golden eval set JSON. For `tool_trajectory_avg_score` (the
   default metric), an eval set is required — it provides the expected tool call
   sequence to compare against. If they don't have one yet, explain this and suggest
   starting with `hallucinations_v1`, or ask if they want to create a golden set from
   a reference run first.

3. Call `evaluate_traces` with the file(s), format, and eval set.

4. Present results as a score table (see Score interpretation below) and explain failures.

## Evaluating sessions (regression testing)

This workflow requires the server to be running with the `--dev` flag (which enables
WebSocket and session streaming). Plain `agentevals serve` will not have sessions.
If you get a connection error from any tool below, tell the user:

```bash
uv run agentevals serve --dev
```

1. Call `list_sessions` to show available sessions.

2. Help the user identify the "golden" session — the reference run that represents
   correct behavior. The server derives the eval set from it automatically.

3. Call `evaluate_sessions(golden_session_id=...)`. This scores all other completed
   sessions against the golden.

4. Present a comparison table:
   ```
   Session             | Score | Status  | Delta
   session-abc (golden)| 1.00  | —       | baseline
   session-def         | 0.85  | PASSED  | -0.15
   session-ghi         | 0.40  | FAILED  | -0.60 ⚠️
   ```

5. Explain regressions specifically: which tools the golden called that a failing
   session skipped, or unexpected extra calls. Concrete tool names are more useful
   than just quoting the score.

## Score interpretation

| Score | Meaning |
|-------|---------|
| 1.0 | Exact match — right tools, right order |
| 0.7–0.9 | Minor deviations (extra call or slightly different args) |
| 0.5–0.7 | Partial match — some turns correct, others missing or wrong tool calls |
| 0.0–0.5 | Major divergence — most tool calls don't match golden |

**Important:** `evalStatus: PASSED` does **not** mean the agent did well — it only means
the score met the configured threshold. Without a configured threshold, every session
shows PASSED regardless of score. Focus on the numeric score, not the status label.

## After results

If the user wants to understand *what the agent did* step by step (not just the score),
suggest `/inspect` to get a readable narrative of a session.

登录后下载/点赞/收藏 ❤ 110 | ★ 0

eval

目录结构

SKILL.md

举报内容

提示