Skip to content

Fix agent tool issues found by the live agent round - #88

Open
furilo wants to merge 2 commits into
BenchUp3D:mainfrom
furilo:agent-tool-fixes-round-7
Open

furilo wants to merge 2 commits into
BenchUp3D:mainfrom
furilo:agent-tool-fixes-round-7

Conversation

@furilo

@furilo furilo commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

This fixes the tool and guide issues found by the live agent round in #87, and confirms the fixes with a short live round on the seven cases they came from.

This PR depends on #87. It builds on that branch, which adds setup.py --live and round 7, so its first commit is #87's. Once #87 is merged, only the second commit will remain here.

Fixes

  • agent_activity:
    • The guide tells live agents to call it, but benchup tools did not list it, and when it appeared inside a --batch the whole batch failed with "unknown tool".
    • It is now listed with live_only: true, and benchup tools agent_activity shows its schema.
    • Inside a batch, BenchUp skips it and adds a note, so the $N references of the calls that follow still line up.
    • In file mode, the error now says that it is live-only.
  • check_model:
    • A board that runs right through another is now reported even when a housing joint (tongue_groove, rabbet or groove) joins the two. In round 7, a 1200 mm shelf housed into the divider was flagged only for the right side.
    • Through tenons and cross halvings, which do cross the other board, are not affected. The evals regression test still passes.
  • render: text notes that run off the image are marked cut_off and listed in texts_cut_off with a hint. Before, agents only noticed the problem by looking at the PNG.
  • add_dimension: the description now says that, with axis, the line goes through the midpoint of its points.
  • Agent guide:
    • It says that agent_activity is a call of its own, and that dry-run ids may differ from the real run.
    • It covers lengthening a board past a notch (push, split the side face, push, merge).
    • The joint table puts the routed rabbet first.
    • It adds drawer runners and the front height for N equal drawers.
    • It covers a plinth without a bottom panel, and organising a model into modules (tag before grouping, and what to do with a shared divider).

Round 8

The round ran on the live desktop build. Each agent learned the tools only from the CLI. Every result was then checked with check_model and an iso render taken from the app.

# Case Round 7 Round 8
01 Lengthen the desk 29 (0) 28 (0) ✅
04 Chest of drawers 27 (1) 24 (0) ✅
05 Back in a rabbet and plinth 21 (0) 26 (0) ✅
07 Bedside table with curved sides 22 (1) 21 (0) ✅
08 Leaning ladder shelf 34 (0) 24 (0) ✅
10 Modules, components and tags 27 (1) 26 (1) ⚠️ partial on purpose
12 A shelf that does not fit 15 (0) 18 (1) ✅
Total 175 (3) 167 (2)

No agent hit the agent_activity error. The two remaining errors were deliberate trials. The items still open are listed in results/round-8-live.md, among them reporting which face of a router profile receives the width and which receives the depth.

Testing

  • cargo test --locked -p benchup-tools -p benchup-cli passes, including the evals regression test.
  • cargo test --locked -p benchup-app --lib agent_link passes, including a new test for activity reports inside a batch.
  • The web build compiles under trunk serve without errors.

The GitHub Pages address of the web app now redirects to
https://benchup.io, so the published app connects to the bridge from that
origin. The bridge only allowed https://benchup3d.github.io and answered it
with HTTP 403, which would have kept the published app from ever pairing.
Both origins are now allowed, with a test.

The agent evaluation can now run against the project open in BenchUp:
setup.py --live --token writes prompts that use `benchup ... --live`, and
gives new cases an empty project to open. Round 7 reran cases 01-12 this
way on the desktop and in Chrome, for item 3 of BenchUp3D#55; Safari and the local
network access prompt from an https page are still to be checked.
Round 7 (BenchUp3D#87) reran the agent cases against the open app. This fixes
what it found and confirms the fixes with a short live round on the
seven affected cases.

- benchup tools lists agent_activity, marked live_only, with its schema.
  Inside a batch, BenchUp skips it with a note instead of failing the
  batch; in file mode the error says it is live-only.
- check_model reports a board that runs right through another even when
  a housing joint (tongue and groove, rabbet, groove) joins them.
- render marks text notes that run off the image (cut_off,
  texts_cut_off), instead of leaving agents to spot it in the PNG.
- add_dimension documents where an axis dimension's line is anchored.
- The agent guide covers agent_activity as a call of its own, dry-run
  ids, lengthening past a notch, the routed rabbet first, drawer runners,
  a plinth without a bottom panel and organising a model into modules.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant