Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .agents/skills/benchup-agent-eval/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
name: benchup-agent-eval
description: Run a round of the agent evaluation of the benchup CLI and MCP tools (crates/benchup-cli/evals) — one subagent per case, a results table compared with earlier rounds, and findings sorted into tool bugs, guide gaps and core limits. Use when asked to evaluate, test or "run the cases/battery/round" for the benchup CLI, MCP or agent guide, or after changing agent tools, their descriptions or docs/agent-guide.md.
---

# Agent evaluation round

The cases, rules and earlier results are in `crates/benchup-cli/evals/` (read its README.md first). A round spends many tokens: run it only when the user asked for it, and say how many agents it will start.

1. **Regression first.** `cargo test -p benchup-cli --test evals`. If it fails, stop and report: a reference solution no longer applies, which is a regression or a reference to update.
2. **Build and prepare.** `cargo build --release -p benchup-cli`, then `python3 crates/benchup-cli/evals/setup.py --round N` (N = the last round in `results/` plus one; `--cases 01,13` for a subset). It prints each case's folder and prompt file.
3. **Launch one subagent per case, in parallel, in the background.** Give each the content of its prompt file verbatim, with nothing added: no hints, no knowledge of the code, no expectations. The agents must learn the tools only from what the tools say.
4. **Check the results yourself; do not trust the reports alone.** For each case, run `benchup call <file> check_model` and `benchup render <file> --view iso` (look at the image) on the file the agent left, and compare with the request.
5. **Write `crates/benchup-cli/evals/results/round-N.md`:**
- a table per case: commands (errors), result ✅ / ⚠️ / ❌ checked in step 4, and the same columns from the previous round;
- which of each case's `focus` tools the agent actually used, and where it fell back to a workaround;
- the findings, sorted into: tool bugs and unclear errors or descriptions (fix in `benchup-tools`); guide gaps or mistakes (`docs/agent-guide.md`); core limits (an issue, with a minimal reproduction); missing features.
6. **Report to the user** with the table and the proposed fixes, and wait before changing code. After fixing, update reference batches if a tool changed on purpose, and add a reference for any new case an agent solved well.

Keep the cases comparable across rounds: do not edit a case's request or seed; add a new case instead.
1 change: 1 addition & 0 deletions .claude/skills
2 changes: 1 addition & 1 deletion crates/benchup-cli/evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ The CLI and MCP tools are meant to be used by coding agents that have never seen
There are two layers:

1. **Regression, on every change** — `cargo test -p benchup-cli --test evals`. Replays the reference solution of each case (`reference/<case>/*.jsonl`, batches that agents or maintainers got right) with the real binary and checks the result with `check_model`. It needs no agent and takes a few seconds. When a tool changes on purpose, update the reference batch.
2. **Agent rounds, on demand** — before a release, or after changing tools, descriptions or the [agent guide](../../../docs/agent-guide.md). One agent per case, run in parallel; each one reports what it did, how many commands it needed and where it got stuck. Not in CI: it costs tokens and the results vary from run to run. To run one:
2. **Agent rounds, on demand** — before a release, or after changing tools, descriptions or the [agent guide](../../../docs/agent-guide.md). One agent per case, run in parallel; each one reports what it did, how many commands it needed and where it got stuck. Not in CI: it costs tokens and the results vary from run to run. With Claude Code or Codex, the `benchup-agent-eval` skill in [`.agents/skills`](../../../.agents/skills/benchup-agent-eval/SKILL.md) runs a round (`.claude/skills` links to it); by hand:

```sh
cargo build --release -p benchup-cli
Expand Down
Loading