benchflow

Category: Tools & Productivity | Uploader: benchflow-ai | Downloads: 0 | Version: v1.0（Latest）

Run agent benchmarks, create tasks, analyze results, and manage agents using BenchFlow. Use when asked to benchmark an AI coding agent, run a benchmark suite, create tasks, view trajectories, or compare agent performance.

Changelog: Source: GitHub https://github.com/benchflow-ai/benchflow

Directory Structure

Current level: Root

📁 references/
- 📄 create-task.md 4.6 KB
- 📄 dogfood.md 3.0 KB
- 📄 review-and-test.md 2.9 KB
- 📄 smoke-test.md 3.1 KB
📁 tasks/
- 📁 benchflow-knowledge/
  - 📁 environment/
    
    📁 benchflow/
    
    📁 references/
    
    📄 create-task.md 4.6 KB
    
    📄 SKILL.md 5.7 KB
    
    📄 Dockerfile 192 B
  - 📁 tests/
    
    📄 test.sh 1.5 KB
  - 📄 instruction.md 874 B
  - 📄 task.toml 307 B
- 📁 create-simple-task/
  - 📁 environment/
    
    📁 benchflow/
    
    📁 references/
    
    📄 create-task.md 4.6 KB
    
    📄 SKILL.md 5.7 KB
    
    📄 Dockerfile 192 B
  - 📁 tests/
    
    📄 test.sh 1.5 KB
  - 📄 instruction.md 715 B
  - 📄 task.toml 314 B
- 📁 run-job-and-metrics/
  - 📄 instruction.md 670 B
  - 📄 task.toml 315 B
- 📁 run-single-task/
  - 📄 instruction.md 528 B
  - 📄 task.toml 302 B
📄 SKILL.md 6.7 KB

SKILL.md

---
name: benchflow
description: Run agent benchmarks, create tasks, analyze results, and manage agents using BenchFlow. Use when asked to benchmark an AI coding agent, run a benchmark suite, create tasks, view trajectories, or compare agent performance.
user-invocable: true
allowed-tools:
  - Read
  - Write
  - Edit
  - Bash
---

# BenchFlow — Agent Benchmarking

BenchFlow runs AI coding agents against tasks in sandboxed environments and
scores their output. It combines Harbor (environments, verifier) with ACP
(multi-turn agent communication).

Arguments passed: `$ARGUMENTS`

---

## Dispatch on arguments

### No args or `status` — show current state

1. Check if benchflow is installed: `pip show benchflow`
2. Check if `.env` exists with API keys
3. Check available agents: `benchflow agents`
4. Show recent job results if any exist in `jobs/`
5. Point to next action based on state

### `run <task-path>` — run a single task

```bash
source .env
benchflow run -t <task-path> -a claude-agent-acp -e daytona -m claude-haiku-4-5-20251001
```

Or via SDK:
```python
import asyncio
from benchflow import SDK

async def main():
    sdk = SDK()
    result = await sdk.run(
        task_path="<task-path>",
        agent="claude-agent-acp",
        model="claude-haiku-4-5-20251001",
        environment="daytona",
    )
    print(f"Reward: {result.rewards}, Tools: {result.n_tool_calls}")

asyncio.run(main())
```

API keys are auto-inherited from `os.environ`. No need to pass `agent_env`.

### `job <tasks-dir>` — run a benchmark suite

```bash
benchflow job -t <tasks-dir> -a claude-agent-acp -e daytona -c 64
```

Or via YAML config:
```bash
benchflow job -f examples/configs/tb2-haiku.yaml
```

YAML format (benchflow-native):
```yaml
tasks_dir: .ref/terminal-bench-2
jobs_dir: jobs/tb2-haiku
agent: claude-agent-acp
model: claude-haiku-4-5-20251001
environment: daytona
concurrency: 64
max_retries: 1
```

Harbor-compatible YAML also works:
```yaml
jobs_dir: jobs
n_attempts: 2
orchestrator:
  n_concurrent_trials: 8
environment:
  type: daytona
agents:
  - name: claude-agent-acp
    model_name: anthropic/claude-haiku-4-5-20251001
datasets:
  - path: .ref/terminal-bench-2
```

Multi-turn (adds a recheck prompt):
```yaml
tasks_dir: .ref/terminal-bench-2
jobs_dir: jobs/tb2-multiturn
agent: claude-agent-acp
model: claude-haiku-4-5-20251001
environment: daytona
concurrency: 64
prompts:
  - null  # uses instruction.md
  - "Review your solution. Check for errors, test it, and fix any issues."
```

### `metrics <jobs-dir>` — analyze results

```bash
benchflow metrics jobs/tb2-haiku/
benchflow metrics jobs/tb2-haiku/ --json
```

SDK:
```python
from benchflow import collect_metrics
metrics = collect_metrics("jobs/tb2-haiku", benchmark="TB2", agent="claude-agent-acp")
print(metrics.summary())
```

### `view <trial-dir>` — view a trajectory

```bash
benchflow view jobs/tb2-haiku/<trial-name>/
```

Opens HTML viewer at `http://localhost:8888`.

### `create-task` — create a new benchmark task

See `skills/benchflow/references/create-task.md` for the full guide.

### `smoke-test` — verify system works

See `skills/benchflow/references/smoke-test.md` for quick and full smoke tests.

### `dogfood` — end-to-end testing prompt

See `skills/benchflow/references/dogfood.md` for the full dogfood script.

### `review` — code review and fix workflow

See `skills/benchflow/references/review-and-test.md` for the council review workflow:
1. Launch 4 subagent reviewers in parallel (SDK, ACP, Process, Tests)
2. Synthesize findings, prioritize by severity
3. Fix bugs, run tests
4. Smoke test on SkillsBench tasks
5. Post progress to Discord

Quick structure:
```
my-task/
├── task.toml          # timeouts, resources, metadata
├── instruction.md     # what the agent should do
├── environment/
│   └── Dockerfile     # sandbox setup
├── tests/
│   └── test.sh        # verifier → writes to /logs/verifier/reward.txt
└── solution/          # optional reference solution
```

### `agents` — list available agents

```bash
benchflow agents
```

| Agent | Status | Skills |
|-------|--------|--------|
| `claude-agent-acp` | Working | `~/.claude/skills/` |
| `pi-acp` | Working | `~/.claude/skills/` |
| `openclaw` | Working (via shim) | copies to `<workspace>/skills/` |
| `codex-acp` | Registered | needs OPENAI_API_KEY |
| `gemini` | Registered | needs GOOGLE_API_KEY |

### `compare` — multi-agent comparison

```python
import asyncio
from benchflow import Job, JobConfig

async def main():
    for agent in ["claude-agent-acp", "pi-acp", "openclaw"]:
        job = Job(
            tasks_dir="path/to/tasks",
            jobs_dir=f"jobs/compare-{agent}",
            config=JobConfig(agent=agent, environment="daytona", concurrency=64),
        )
        result = await job.run()
        print(f"{agent}: {result.passed}/{result.total} ({result.score:.1%})")

asyncio.run(main())
```

---

## Setup

```bash
pip install benchflow    # or: pip install -e . (from source)
source .env              # ANTHROPIC_API_KEY, DAYTONA_API_KEY
```

## Environments

| Environment | Concurrency | Setup |
|-------------|-------------|-------|
| `daytona` | 64+ | Set `DAYTONA_API_KEY` in `.env` |
| `docker` | ~4 | Docker must be running locally |

Use `daytona` for benchmarks. Docker is limited by network exhaustion.

## Skills in tasks

Two approaches for deploying skills:

### Baked into Docker image (existing tasks)
```dockerfile
COPY skills /root/.claude/skills
```

### Runtime deployment via `--skills-dir` (new)
```bash
benchflow run -t task-dir -a claude-agent-acp --skills-dir skills/ -e daytona
```

SDK:
```python
result = await sdk.run(
    task_path="task-dir",
    agent="claude-agent-acp",
    skills_dir="skills/",  # uploaded to /skills, symlinked to agent paths
    environment="daytona",
)
```

YAML config:
```yaml
tasks_dir: path/to/tasks
skills_dir: skills/
agent: claude-agent-acp
```

Skills are uploaded to `/skills/` in the sandbox and symlinked to:
- `~/.claude/skills/` (claude-agent-acp, pi-acp)
- `~/.gemini/skills/` (gemini)
- `openclaw`: shim copies from `.claude/skills/` → `<workspace>/skills/`

## Output structure

```
jobs/{job_name}/{trial_name}/
├── result.json              # rewards, agent, timing
├── prompts.json             # prompts sent
├── trajectory/
│   └── acp_trajectory.jsonl # tool calls + agent thoughts
└── verifier/
    ├── reward.txt           # reward value
    └── ctrf.json            # test results
```

## Tips

- Use `claude-haiku-4-5-20251001` for testing. Use Sonnet for real benchmarks.
- Jobs resume — re-running the same `jobs_dir` skips completed tasks.
- `None` in prompts list gets replaced with `instruction.md` content.
- Partial rewards work (verifier can write `0.5` to reward.txt).

Comments 0

Latest | Hot

Please login before commenting.

No comments yet. Be the first one!

benchflow

Directory Structure

SKILL.md

Report

Notice