Repository navigation
Turn @deval into a tool-calling agent with capture - #1
Merged
Merged
Conversation
The chat participant ran a single model request and streamed the answer. It now runs an agentic loop (src/agent/loop.ts) against the Chat model with three read-only tools (deval_readFile, deval_listFiles, deval_grep) registered via vscode.lm and declared in package.json. Every run is captured to .deepeval/vscode-agent-events/<session-id>.jsonl via the new event-writer, in the same format the Local-harness hook writes, so the existing captured-run benchmark scores the agent's trajectory unchanged. The loop core is dependency-free: unit tests drive it with fake ports, and benchmarks/agent-loop-benchmark.test.ts measures it with a scripted model through observe().
The project is developed without CI for now; the README's CI section is replaced with the equivalent local verification commands.
Removes the old features now that @deval is the system under test: - src/agent-harness.ts (reference todo agent); the AgentRun contract moves into src/agent/loop.ts, which is what produces it - benchmarks/todo-benchmark.test.ts - src/capture-hook.ts, .github/hooks/, and the Install Capture Hook command; the agent loop captures runs automatically via event-writer - the Scaffold Sample Benchmark command Keeps the two run commands, the deepeval patch, and the captured-run benchmark, which scores the agent loop's captures. Renames the evaluate command to 'DeepEval: Evaluate Captured Agent Run' and rewrites the README around the one feature.
Mirrors the lanly-dev/vscode-audio-lab layout: extension.ts only registers the two commands and the chat participant, while the suite runner moves to src/suites.ts and the participant to src/agent/participant.ts. Test imports updated accordingly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Turns the
@devalchat participant from a plain chat forwarder into a realtool-calling agent, and captures every run for benchmarking.
Changes
src/agent/loop.ts) — dependency-free core: model turns,tool dispatch with
PreToolUse/PostToolUsepairing, 10-turn cap. Takes themodel, tools, and capture sink as ports, so the same code runs in the
extension host, in unit tests (fakes), and in benchmarks (scripted model).
src/agent/tool-schemas.ts,src/agent/tools.ts) — threeread-only tools (
deval_readFile,deval_listFiles,deval_grep) declaredin
package.json(languageModelTools) and registered viavscode.lm,with workspace-escape guards and output caps. Read-only on purpose: no
confirmation UX needed for v1.
src/agent/vscode-adapter.ts) — translates loop messages tovscode.LanguageModelChatMessages, streams text to the chat, executes toolsvia
vscode.lm.invokeToolso the UI shows invocations.src/events/event-writer.ts) — writes each run to.deepeval/vscode-agent-events/<session-id>.jsonlin the formatevent-log.tsreads, soDeepEval: Evaluate Captured Local Agent Runscores the agent with no extra wiring.
src/extension.ts) — runs the loop per request, reportsprogress, summarizes the run when done.
benchmark
benchmarks/agent-loop-benchmark.test.ts(scripted model throughobserve(),ContainsAllMetric+ToolCorrectnessMetric, no API key).@devalagent section, repo layout) and CHANGELOG.Verification
npm run check-types— cleannpm run lint— cleannpm run benchmark— all suites pass, 100% metric pass rate (incl. the newagent-loop case)
Follow-ups (not in this PR)
todoAppAgent) and the scaffold command are nowredundant with
@devalas the system under test — candidates for removal.writeFile/runCommandtools behind user confirmation (v2).xvfbneeded; not run here).