Deval

A VS Code extension with its own benchmarkable coding agent. Chat with @deval and it runs a real tool-calling loop against your LLM model; every run is captured, and DeepEval TypeScript suites score the runs — trajectory and tool use — with deterministic metrics.

It gives you one thing: an agent you can measure.

1. The @deval agent (src/agent/) — a tool-calling loop on vscode.lm with read-only workspace tools. No API key needed; the model comes from the LLM model picker.
2. Automatic capture — every run is written to .deval/vscode-agent-events/<session-id>.jsonl.
3. Benchmarks — DeepEval suites that score the loop (scripted model) and score real captured runs (trajectory, tool resolution, policy).

The bundled suites use deterministic metrics only, so they run offline with no API key.

Commands

• Deval: Score Agent Run — tests your @deval agent's behavior with 11 trajectory checks.
• Deval: Benchmark Model — Benchmark wired model, focus on LLM model with 15 cases; samples per prompt configurable (1–5, default 1), reports pass@1 / pass@k per case.
• Deval: Show Scoreboard — Show the record runs.

Settings

• deval.devalCommand: executable used to launch the DeepEval runner. Defaults to npx.
• deval.packageManager: package manager used to install deepeval + vitest — npm, pnpm, yarn, or bun. Defaults to npm, and is used only by Install dependencies when a run needs it.
• deval.samplesPerPrompt: how many times each benchmark prompt is asked (1–5). Defaults to 1; above 1 enables pass@k scoring at proportionally more model calls.

These are deliberately separate settings. deval.devalCommand names the runner (npx deepeval test run …), so it cannot be reused to install: npx install --save-dev deepeval vitest makes npx look for an executable called install and fail with could not determine executable to run.

Release Notes

0.0.1

• Initial release of Deval