Skip to content

Turn @deval into a tool-calling agent with capture - #1

Merged
lanly-dev merged 4 commits into
mainfrom
feature/agent-loop
Oct 8, 2026
Merged

lanly-dev merged 4 commits into
mainfrom
feature/agent-loop

Conversation

@lanly-dev

Copy link
Copy Markdown
Owner

What

Turns the @deval chat participant from a plain chat forwarder into a real
tool-calling agent, and captures every run for benchmarking.

Changes

  • Agent loop (src/agent/loop.ts) — dependency-free core: model turns,
    tool dispatch with PreToolUse/PostToolUse pairing, 10-turn cap. Takes the
    model, tools, and capture sink as ports, so the same code runs in the
    extension host, in unit tests (fakes), and in benchmarks (scripted model).
  • Tools (src/agent/tool-schemas.ts, src/agent/tools.ts) — three
    read-only tools (deval_readFile, deval_listFiles, deval_grep) declared
    in package.json (languageModelTools) and registered via vscode.lm,
    with workspace-escape guards and output caps. Read-only on purpose: no
    confirmation UX needed for v1.
  • Adapter (src/agent/vscode-adapter.ts) — translates loop messages to
    vscode.LanguageModelChatMessages, streams text to the chat, executes tools
    via vscode.lm.invokeTool so the UI shows invocations.
  • Capture writer (src/events/event-writer.ts) — writes each run to
    .deepeval/vscode-agent-events/<session-id>.jsonl in the format
    event-log.ts reads, so DeepEval: Evaluate Captured Local Agent Run
    scores the agent with no extra wiring.
  • Participant (src/extension.ts) — runs the loop per request, reports
    progress, summarizes the run when done.
  • Tests — 9 new unit tests (loop core, writer); new deterministic
    benchmark benchmarks/agent-loop-benchmark.test.ts (scripted model through
    observe(), ContainsAllMetric + ToolCorrectnessMetric, no API key).
  • Docs — README (@deval agent section, repo layout) and CHANGELOG.

Verification

  • npm run check-types — clean
  • npm run lint — clean
  • npm run benchmark — all suites pass, 100% metric pass rate (incl. the new
    agent-loop case)
  • 9 new mocha unit tests — passing

Follow-ups (not in this PR)

  • The old reference agent (todoAppAgent) and the scaffold command are now
    redundant with @deval as the system under test — candidates for removal.
  • writeFile / runCommand tools behind user confirmation (v2).
  • Extension-host tests for the new modules (xvfb needed; not run here).

The chat participant ran a single model request and streamed the answer.
It now runs an agentic loop (src/agent/loop.ts) against the Chat model with
three read-only tools (deval_readFile, deval_listFiles, deval_grep)
registered via vscode.lm and declared in package.json.

Every run is captured to .deepeval/vscode-agent-events/<session-id>.jsonl
via the new event-writer, in the same format the Local-harness hook writes,
so the existing captured-run benchmark scores the agent's trajectory
unchanged. The loop core is dependency-free: unit tests drive it with fake
ports, and benchmarks/agent-loop-benchmark.test.ts measures it with a
scripted model through observe().
The project is developed without CI for now; the README's CI section is
replaced with the equivalent local verification commands.
Removes the old features now that @deval is the system under test:
- src/agent-harness.ts (reference todo agent); the AgentRun contract
  moves into src/agent/loop.ts, which is what produces it
- benchmarks/todo-benchmark.test.ts
- src/capture-hook.ts, .github/hooks/, and the Install Capture Hook
  command; the agent loop captures runs automatically via event-writer
- the Scaffold Sample Benchmark command

Keeps the two run commands, the deepeval patch, and the captured-run
benchmark, which scores the agent loop's captures. Renames the evaluate
command to 'DeepEval: Evaluate Captured Agent Run' and rewrites the
README around the one feature.
Mirrors the lanly-dev/vscode-audio-lab layout: extension.ts only
registers the two commands and the chat participant, while the suite
runner moves to src/suites.ts and the participant to
src/agent/participant.ts. Test imports updated accordingly.
@lanly-dev
lanly-dev merged commit 21916e0 into main Oct 8, 2026
@lanly-dev
lanly-dev deleted the feature/agent-loop branch October 8, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant