From 937e871ab153e0ee7f70737a4bb5e2bbd7767918 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=81lvaro=20Ortiz?= Date: Sat, 14 Nov 2026 12:27:30 +0100 Subject: [PATCH] Add a skill that runs an agent evaluation round .agents/skills/benchup-agent-eval tells Claude Code or Codex how to run a round of crates/benchup-cli/evals: the regression test first, one subagent per case with its prompt verbatim, an independent check of every result with check_model and a render, and results/round-N.md with the findings sorted into tool bugs, guide gaps, core limits and missing features. .claude/skills links to .agents/skills so Claude Code finds it. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/benchup-agent-eval/SKILL.md | 20 ++++++++++++++++++++ .claude/skills | 1 + crates/benchup-cli/evals/README.md | 2 +- 3 files changed, 22 insertions(+), 1 deletion(-) create mode 100644 .agents/skills/benchup-agent-eval/SKILL.md create mode 120000 .claude/skills diff --git a/.agents/skills/benchup-agent-eval/SKILL.md b/.agents/skills/benchup-agent-eval/SKILL.md new file mode 100644 index 0000000..2ef4947 --- /dev/null +++ b/.agents/skills/benchup-agent-eval/SKILL.md @@ -0,0 +1,20 @@ +--- +name: benchup-agent-eval +description: Run a round of the agent evaluation of the benchup CLI and MCP tools (crates/benchup-cli/evals) — one subagent per case, a results table compared with earlier rounds, and findings sorted into tool bugs, guide gaps and core limits. Use when asked to evaluate, test or "run the cases/battery/round" for the benchup CLI, MCP or agent guide, or after changing agent tools, their descriptions or docs/agent-guide.md. +--- + +# Agent evaluation round + +The cases, rules and earlier results are in `crates/benchup-cli/evals/` (read its README.md first). A round spends many tokens: run it only when the user asked for it, and say how many agents it will start. + +1. **Regression first.** `cargo test -p benchup-cli --test evals`. If it fails, stop and report: a reference solution no longer applies, which is a regression or a reference to update. +2. **Build and prepare.** `cargo build --release -p benchup-cli`, then `python3 crates/benchup-cli/evals/setup.py --round N` (N = the last round in `results/` plus one; `--cases 01,13` for a subset). It prints each case's folder and prompt file. +3. **Launch one subagent per case, in parallel, in the background.** Give each the content of its prompt file verbatim, with nothing added: no hints, no knowledge of the code, no expectations. The agents must learn the tools only from what the tools say. +4. **Check the results yourself; do not trust the reports alone.** For each case, run `benchup call check_model` and `benchup render --view iso` (look at the image) on the file the agent left, and compare with the request. +5. **Write `crates/benchup-cli/evals/results/round-N.md`:** + - a table per case: commands (errors), result ✅ / ⚠️ / ❌ checked in step 4, and the same columns from the previous round; + - which of each case's `focus` tools the agent actually used, and where it fell back to a workaround; + - the findings, sorted into: tool bugs and unclear errors or descriptions (fix in `benchup-tools`); guide gaps or mistakes (`docs/agent-guide.md`); core limits (an issue, with a minimal reproduction); missing features. +6. **Report to the user** with the table and the proposed fixes, and wait before changing code. After fixing, update reference batches if a tool changed on purpose, and add a reference for any new case an agent solved well. + +Keep the cases comparable across rounds: do not edit a case's request or seed; add a new case instead. diff --git a/.claude/skills b/.claude/skills new file mode 120000 index 0000000..2b7a412 --- /dev/null +++ b/.claude/skills @@ -0,0 +1 @@ +../.agents/skills \ No newline at end of file diff --git a/crates/benchup-cli/evals/README.md b/crates/benchup-cli/evals/README.md index 309e81b..48b8161 100644 --- a/crates/benchup-cli/evals/README.md +++ b/crates/benchup-cli/evals/README.md @@ -5,7 +5,7 @@ The CLI and MCP tools are meant to be used by coding agents that have never seen There are two layers: 1. **Regression, on every change** — `cargo test -p benchup-cli --test evals`. Replays the reference solution of each case (`reference//*.jsonl`, batches that agents or maintainers got right) with the real binary and checks the result with `check_model`. It needs no agent and takes a few seconds. When a tool changes on purpose, update the reference batch. -2. **Agent rounds, on demand** — before a release, or after changing tools, descriptions or the [agent guide](../../../docs/agent-guide.md). One agent per case, run in parallel; each one reports what it did, how many commands it needed and where it got stuck. Not in CI: it costs tokens and the results vary from run to run. To run one: +2. **Agent rounds, on demand** — before a release, or after changing tools, descriptions or the [agent guide](../../../docs/agent-guide.md). One agent per case, run in parallel; each one reports what it did, how many commands it needed and where it got stuck. Not in CI: it costs tokens and the results vary from run to run. With Claude Code or Codex, the `benchup-agent-eval` skill in [`.agents/skills`](../../../.agents/skills/benchup-agent-eval/SKILL.md) runs a round (`.claude/skills` links to it); by hand: ```sh cargo build --release -p benchup-cli