feat(agents): add OpenCode harness with opt-in [opencode] extra - #115
Open
mohsen-uipath wants to merge 8 commits into
Open
feat(agents): add OpenCode harness with opt-in [opencode] extra#115mohsen-uipath wants to merge 8 commits into
mohsen-uipath wants to merge 8 commits into
Conversation
New agent kind `opencode`, registered through the existing plugin SPI and selectable from task YAML (agent.type: opencode) or the CLI (-D agent.type=opencode). Drives `opencode run --format json` non-interactively and reduces its nd-JSON event stream into the standardized event protocol; EventCollector builds the TurnRecord, so no telemetry is assembled by hand. Telemetry and cost: - Per-step token buckets with the input convention arbitrated per step from the stream's own `total` (flat vs nested); unverifiable or contradictory shapes warn once per turn instead of silently mis-booking a bucket. - Real per-call cost from step_finish.cost; the rate card fills the gaps (cost omitted, or $0 reported for tokens the card prices above zero) so run totals are never understated. openrouter/ model prefixes normalize to the bare rate-card keys (mirrored in evalboard/lib/pricing.ts). - The reconciliation invariant holds by construction: summing the four buckets across TurnRecord.messages equals token_usage exactly. - Tool names normalize to the canonical vocabulary (bash -> Bash, ...) so one criterion scores identically across harnesses. Failure paths per the Agent contract: AgentCrashError with a crashed=True partial TurnRecord on pending_turn, TurnTimeoutError on deadline, cooperative should_stop honored at event granularity, and a clean exit that recognized no events crashes loudly instead of scoring as an empty success. stderr is drained concurrently and every post-exit read is bounded (the CLI's server child holds the inherited pipes open). Opt-in install: the [opencode] extra is deliberately empty — OpenCode is a Node CLI (npm install -g opencode-ai) and the harness imports no third-party Python package; a missing binary fails at start() with the install command. Validated live end-to-end (SUCCESS backed by real telemetry: turns, tokens, tools, cost, exact reconciliation). Documented at docs/agents/OPENCODE.md and wired into the docs nav and generated index surfaces.
… test every failure path Closes the three deliberately-deferred hardening items: - The post-EOF reap in _settle_turn was unbounded: a CLI that closed its stream but never exited hung the turn past its deadline, the one window where turn_timeout went unenforced. The reap now gets the deadline's remainder (TurnTimeoutError on expiry) or a fixed grace when no deadline is configured (AgentCrashError naming the wedge). - kill()/kill_sync() signaled only the CLI pid, orphaning the server child that opencode run leaves holding the pipes — a slow process leak across a batch. Each invocation now runs in its own session (start_new_session), and teardown sweeps the spawned process groups with SIGKILL. OpenCode persists sessions on disk, so --session continuity survives the sweep. Verified live: zero leftover opencode processes after a real run. - The failure paths were the least-tested code in the file. Eleven new tests cover: deadline expiry mid-stream and post-EOF (TurnTimeoutError, partial parked, single TIMEOUT terminal event, iteration rollback), the no-deadline wedge (AgentCrashError), external CancelledError (partial parked, CRASHED terminal event, cancellation re-raised), kill_sync from the watchdog thread, process-group sweep on stop/cooperative-stop, tool error/permission-denied capture, and the orphan-result branch. Live smoke re-run under the new teardown: SUCCESS with exact reconciliation, normalized tools, heavy cache traffic booked correctly, zero warnings, zero leaked processes.
…F exit grace The constant gained a second consumer in the bounded-reap change (the exit grace in _settle_turn when no turn deadline is configured); the comment still described only the SIGTERM->SIGKILL role. Comment text only.
mohsen-uipath
requested review from
akshaylive,
bai-uipath,
tmatup and
uipreliga
as code owners
August 14, 2026 17:30
…rate entry deepseek-v4-flash-0731 is unusable on this OpenRouter account (every serving provider is excluded by the account's data policy), so the checked-in smoke task failed out of the box while all real validation ran on deepseek-v4-pro anyway. Standardize every reference — smoke task, docs examples, config docstring, tests — on v4-pro, and drop the now-orphaned flash-0731 rate-card entry plus its evalboard parity listing (v4-pro was already priced on main). The smoke task now passes as checked in, with no model override. Verified live: SUCCESS 1.000, no data-policy error.
# Conflicts: # README.md # docs/index.md # docs/llms.txt # mkdocs.yml
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
- signal.SIGKILL does not exist on Windows, so pyright failed the Windows Smoke job on kill_sync. Resolve it once as _SIGKILL (SIGTERM fallback) and use it in kill_sync and the group sweep; the sweep itself was already a runtime no-op off POSIX. - The Windows job also runs pytest: install the os.killpg test stub with raising=False (the attribute is absent there) and skip the process-group-teardown test class off POSIX, since the sweep it asserts is POSIX-only by design. - CodeQL py/mixed-returns on communicate(): the final except ends in _crash_turn, whose NoReturn CodeQL cannot see — add an explicit unreachable raise so no path looks like an implicit None return. - CodeQL py/ineffectual-statement on the cancellation test's bare 'await task': bind the (never-produced) value so the statement's effect is explicit.
The E2E job runs --tags smoke-pass on Bedrock runners that have neither the opencode CLI nor OpenRouter credentials, and pins the bucket at exactly 7 tasks; the new task's smoke-pass tag made it an 8th, un-runnable entry. Drop the tag (the task keeps smoke/opencode for local runs) — live opencode coverage needs its own credentialed job, the way Codex has one.
Collaborator
|
Let's wait for merging until we get some results from this code. |
`plugins:` is how a task ships the skills under test, but the OpenCode agent listed it among the fields it silently drops. A skill-injection run therefore looked entirely normal while measuring the bare model: the only loadable skill was OpenCode's built-in `customize-opencode`, and an attempt to load a real one returned an error. Map each local plugin root to OpenCode's `skills.paths`: - Read the `skills` field of `<root>/.claude-plugin/plugin.json` (string or list), the same field Claude Code reads, so one `plugins:` line means the same thing on both harnesses. Fall back to the convention default `<root>/skills`, or to the root itself when it is already a bare skills directory. - Hand the paths over via OPENCODE_CONFIG_CONTENT, which the CLI merges as a final local-scope layer. Chosen over writing `<sandbox>/.opencode/skills/`: it writes nothing into the sandbox that is later preserved as a run artifact and inspected by file criteria, and it does not depend on how the CLI resolves a project root from `--dir`. An inherited value is merged into, not clobbered; with no `plugins:` block the variable is untouched, so runs without one are byte-for-byte unchanged. - Never point at a plugin root that has a skills subdir. `skills.paths` is scanned recursively and a root can hold a self-referential symlink, which resolves skills through an arbitrary path and drops duplicate names. `--pure` skips external *plugins*, not configured skill paths, so the default `pure: true` is unaffected. Also make the engagement observable, without which the injection cannot be told from the old behavior: map OpenCode's lowercase `skill` tool to the canonical `Skill`, and read the skill name from `parameters["name"]` (OpenCode) as well as `parameters["skill"]` (Claude). Every way this can resolve to nothing — unset env var, missing directory, no SKILL.md under the root — is warned at `start()`, and the resolved paths are recorded per task under `environment_info.opencode_skill_paths`. Verified against the real CLI: 1 -> 27 loadable skills, and a live smoke task goes from an invented command at score 0.0 to `Skill` engagement plus the correct invocation at score 1.000.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new agent kind
opencode(registered via the existing plugin SPI, selectable withagent.type: opencodeor-D agent.type=opencode) that drivesopencode run --format jsonand reduces its nd-JSON stream into the standard event protocol —EventCollectorbuilds theTurnRecord, no hand-assembled telemetry. Since OpenCode is model-agnostic, this is the cheap path for evaluating open-weight models (DeepSeek, Kimi, GLM, …) through one harness.Highlights
Agentcontract: pending-turn crash/timeout semantics, cooperativeshould_stop, one terminal event per turn;max_turnscounts OpenCode's native steps (documented in the run-limit parity table).total; the reconciliation invariant holds exactly; tool names normalized to the canonical vocabulary.openrouter/model ids normalize to bare rate-card keys (mirrored in evalboard).[opencode]extra is deliberately empty — OpenCode is a Node CLI; a missing binary fails atstart()with the install command.Model: everything is standardized on DeepSeek V4 Pro (
openrouter/deepseek/deepseek-v4-pro) — the smoke task, docs examples, and tests — so the checked-in task runs out of the box. DeepSeek V4 Flash 0731 was dropped: every OpenRouter provider serving it is excluded by the account's data-policy settings (an account-level restriction, not a code issue), so no reference to it ships.Validation (all on DeepSeek V4 Pro via OpenRouter):
make verifygreen (4,149 tests, coverage gate met); live end-to-end runs confirm real telemetry with exact reconciliation (including cache-heavy traffic), zero warnings, zero leaked processes,--sessionresume across turns, and bothtasks/run_limits/parity fixtures behaving per contract (clean cap stop vs. timeout failure).Docs:
docs/agents/OPENCODE.md(wired into nav + generated indexes) + an OpenCode column indocs/agents/HARNESS_PARITY.md.Results — the same 174 tasks on two models
174 tasks from
skills/tests/tasks/across 10 skill directories, run through the OpenCode harness.coder_eval@fcb7dad,skills@1e16eedb,-e tests/experiments/default.yaml --type opencode -j 4— tempdir driver,max_turns: 200,task_timeout: 1200,turn_timeout: 900. Identical task list, experiment, harness and concurrency for both; only-mdiffers.global.profile,/v1/responses)SkillcallsGPT-5.6 Luna wins on every axis: +5.1 pp pass rate, 2.6× faster, and ~17× cheaper — $0.021 vs $0.388 per passing task. It also produced no timeouts at all, where DeepSeek lost 6 tasks to the 1200s task cap.
By skill directory
The two models agree on most of the suite: 133 tasks pass on both, 18 fail on both, 7 pass only on DeepSeek and 16 only on Luna.
uipath-testis the shared floor at 8/18 for both — a suite-side gap rather than a model gap. Luna's gains concentrate inuipath-human-in-the-loop(+19 pp) anduipath-tasks(+33 pp); the singleuipath-platformregression is a one-task directory.Tokens and cost basis
Neither model has a rate-card entry in
src/coder_eval/pricing.py, so both runs report$0.00. Costs above are computed by hand from the models.dev catalog entry for the provider each run actually used, so the two are priced from one source:azure): $1.74/M in, $3.48/M out. Azure publishes no cache-read rate; every provider at this same tier lists ~$0.145/M, used here as an estimate.amazon-bedrock / openai.gpt-5.6-luna): $0.22/M in, $1.32/M out, $0.022/M cache read, $0.275/M cache write (base tier; a higher tier applies above 272k context).Caveat on the cost gap: part of it is billing shape, not just the rate card. The Bedrock route cached the prompt prefix (5.7M cache writes, 48M cache reads, almost no uncached input), while the Azure route billed 27.3M tokens at full input rate — that single bucket is $47.58 of DeepSeek's $54.32.