Skip to content

Add a skill that runs an agent evaluation round of the benchup tools - #64

Merged
dannycoulombe merged 3 commits into
BenchUp3D:mainfrom
furilo:agent-eval-skill
Nov 14, 2026
Merged

dannycoulombe merged 3 commits into
BenchUp3D:mainfrom
furilo:agent-eval-skill

Conversation

@furilo

@furilo furilo commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

Builds on #63: review that one first. Until #63 is merged this PR also shows its two commits; the change here is only the last commit.

Adds a skill that runs a round of the agent evaluation in crates/benchup-cli/evals, added in #63:

  • .agents/skills/benchup-agent-eval/SKILL.md. .agents/skills is a location shared by agents (Codex reads it), which fits a repository that uses AGENTS.md. The skill tells the agent to:
    1. run the regression test first, and stop if it fails;
    2. build the CLI and prepare the round with setup.py;
    3. start one subagent per case with its prompt verbatim, without hints;
    4. check every result itself with check_model and a render, instead of trusting the reports;
    5. write results/round-N.md, comparing with the previous round and sorting the findings into tool bugs, guide gaps, core limits and missing features;
    6. report and wait before changing code.
  • .claude/skills is a symlink to ../.agents/skills, so Claude Code finds the same skill without a copy.
  • The evals README mentions the skill.

A round costs tokens: the skill says to run one only when asked, and to state how many agents it will start. Rounds 5 and 6 in #63 were run this way.

🤖 Generated with Claude Code

@furilo

furilo commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

@dannycoulombe in this PR I propose to include an initial skill, in an .agents dir (so agent-agnostic), to help with reviews/tests. We could include more related to the dev process.

@furilo
furilo force-pushed the agent-eval-skill branch 2 times, most recently from 3daddd7 to 55bc85f Compare November 14, 2026 12:21
.agents/skills/benchup-agent-eval tells Claude Code or Codex how to run a
round of crates/benchup-cli/evals: the regression test first, one
subagent per case with its prompt verbatim, an independent check of every
result with check_model and a render, and results/round-N.md with the
findings sorted into tool bugs, guide gaps, core limits and missing
features. .claude/skills links to .agents/skills so Claude Code finds it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
furilo and others added 2 commits November 14, 2026 15:50
Resolve the scene.rs conflict by keeping joint_geometry and main's
joint_kind comment. Main added pocket joints (biscuit, Domino, groove,
loose tongue): joint_geometry reports only their region and depth, since
their pocket sizes are already in params and a tenon-style cut does not
describe them, and get_estimate uses the app's default minutes for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Take main's handling of pocket joints in joint_geometry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dannycoulombe
dannycoulombe merged commit e4547cf into BenchUp3D:main Nov 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants