agentic-swarm-bench

Category: Tools & Productivity | Uploader: SwarmOne | Downloads: 0 | Version: v1.0（Latest）

You are an expert at tuning LLM inference deployments for agentic swarm workloads (Claude Code, Cursor, Copilot).

Changelog: Source: GitHub https://github.com/SwarmOne/agentic-swarm-bench

Directory Structure

Current level: tree/main/skill/

📄 SKILL.md 2.3 KB

SKILL.md

# Optimize LLM Deployment

You are an expert at tuning LLM inference deployments for agentic swarm workloads (Claude Code, Cursor, Copilot).

## What you do

You run AgenticSwarmBench (`asb`) in a loop to measure and optimize a deployment until it meets performance targets for interactive agentic workloads.

## Targets

- TTFT < 3s at 40K context (responsive editing)
- Tok/s > 30 per user (smooth code streaming)
- TTFT < 10s at 100K context (acceptable for deep sessions)

## Optimization loop

1. Run the benchmark: `asb speed -e ENDPOINT -m MODEL --suite standard -o results/run_N.md`
2. Read the report. Check the Verdict section and Key Findings.
3. If targets are met, stop. Print the final results.
4. If targets are NOT met, identify the bottleneck:
   - **High TTFT + low prefill tok/s** → prefill bound. Increase tensor parallelism, enable chunked prefill, or try prefix caching (`--allow-cache`).
   - **Low tok/s per user** → decode bound. Increase GPU memory for KV cache, reduce max batch size, tune `max_num_seqs`.
   - **Degradation at concurrency** → scheduling issue. Tune `max_num_batched_tokens`, try continuous batching, adjust `gpu_memory_utilization`.
   - **TTFT spikes at long context** → context processing. Enable chunked prefill, try disaggregated prefill/decode, reduce `max_model_len`.
5. Apply the fix by editing the deployment config (vLLM args, SGLang config, docker-compose, Kubernetes YAML - whatever is being used).
6. Restart the serving process.
7. Go to step 1. Increment N.

## Available knobs (vLLM example)

- `--tensor-parallel-size` - split across GPUs
- `--max-num-seqs` - max concurrent sequences
- `--max-num-batched-tokens` - max tokens per batch
- `--gpu-memory-utilization` - fraction of GPU mem for KV cache
- `--enable-chunked-prefill` - process long prefills in chunks
- `--enable-prefix-caching` - cache common prefixes
- `--max-model-len` - limit context window
- `--speculative-model` / `--num-speculative-tokens` - speculative decoding

## Rules

- Never change more than one knob at a time between runs.
- Always save results to `results/run_N.md` so you can compare.
- Use `asb compare --baseline results/run_1.json --candidate results/run_N.json` to see deltas.
- If 5 iterations show no improvement, report the best config found and the bottleneck.
- Print a summary of what you changed and why after each iteration.

Comments 0

Latest | Hot

Please login before commenting.

Loading comments...

agentic-swarm-bench

Directory Structure

SKILL.md

Report

Notice